Rで複数の文書をキーワードごとに分類する ～smdcパッケージの使い方～
目的
文書をキーワードごとに分類する。概略図を以下に示す。
文書ファイルとキーワードファイルの類似度を算出し、類似度に基づいて分類を行っている。
類似度の算出方法は以下のとおり。
①N-gramによる類似度算出
②形態素解析による類似度算出
③センチメント分析による類似度算出
手法
RのRMeCabとsmdcパッケージを用いる。
smdcパッケージをCRANにアップロードするまでの苦労話は以前の記事をご覧ください。
1. データの準備
分類対象ドキュメントと分類キーワード群を用意する。
それぞれを、以下のようなディレクトリ構成で保持する。
C:/LyricsWorkspace |--Documents #分類対象ドキュメント | |--旅立ちの唄.txt | |--常套句.txt | |--シーソーゲーム.txt |--KeyWords #分類キーワード | |--plus.txt | |--negative.txt | |--minus.txt | |--positive.txt |--Dictionary #simDicを使ってセンチメント分析等を行う場合のみ必要 | |--pn_ja.txt #例えば http://www.lr.pi.titech.ac.jp/~takamura/pubs/pn_ja.dic
2. 分類処理
Rで分類処理を行う。
# ライブラリの読込み install.packages('smdc') library(smdc) library(RMeCab) # ディレクトリとファイルを指定 dir1 <- "C:/LyricsWorkspace/KeyWords" dir2 <- "C:/LyricsWorkspace/Documents" dict <- "C:/LyricsWorkspace/Dictionary/pn_ja.txt" # N-gramによる類似度算出 docMatrix1 <- docNgram(dir1, type=0, N=2) docMatrix2 <- docNgram(dir2, type=0, N=2) sim1 <- simDoc(docMatrix1,docMatrix2,norm=TRUE,recursive=TRUE) # ver.0.0.2では sim1 <- simDoc(docMatrix1,docMatrix2,norm=TRUE) simSum(sim1) # 形態素解析による類似度算出 docMatrix3 <- docMatrix(dir1,weight="tf3*idf4*norm",pos=c("名詞")) docMatrix4 <- docMatrix(dir2,weight="tf3*idf4*norm",pos=c("名詞")) sim2 <- simDoc(docMatrix3,docMatrix4,norm=TRUE,recursive=TRUE) # ver.0.0.2では sim2 <- simDoc(docMatrix3,docMatrix4,norm=TRUE) simSum(sim2) # センチメント分析による類似度算出 docMatrix5 <- docMatrix(dir1,weight="tf3*idf4*norm",pos=c("名詞","動詞","形容詞")) docMatrix6 <- docMatrix(dir2,weight="tf3*idf4*norm",pos=c("名詞","動詞","形容詞")) sentiDic <- read.table(dict,sep=":",as.is=TRUE) #因子化を抑制しておく sentiDic <- sentiDic[order(sentiDic[,1]),] #ソートしておく sim3 <- simDic(docMatrix5,docMatrix6,sentiDic[,c(1,4)],breaks=seq(-1,1,length=21),norm=TRUE) simSum(sim3) # 類似度の統合と結果表示 sim <- simSyn(list(sim1,sim2,sim3),c(0.2,0.3,0.5)) summary <- simSum(sim) summary
結果
$minus.txt [1] "[Mr.Children]常套句.txt" "[Mr.Children]hypnosis.txt" [3] "[Mr.Children]旅人.txt" "[Mr.Children]youthful days.txt" [5] "[Mr.Children]【es】～Theme of es～.txt" "[Mr.Children]その向こうへ行こう.txt" [7] "[Mr.Children]Hallelujah.txt" "[Mr.Children]HANABI.txt" [9] "[Mr.Children]and I love you.txt" "[Mr.Children]HERO.txt" [11] "[Mr.Children]Round About ～孤独の肖像～.txt" "[Mr.Children]マーマレード・キッス.txt" [13] "[Mr.Children]my life.txt" "[Mr.Children]口笛.txt" [15] "[Mr.Children]CANDY.txt" "[Mr.Children]Surrender.txt" [17] "[Mr.Children]I'm talking about Lovin'.txt" "[Mr.Children]未来.txt" [19] "[Mr.Children]終末のコンフィデンスソング.txt" "[Mr.Children]東京.txt" [21] "[Mr.Children]365日.txt" "[Mr.Children]Wake me up!.txt" [23] "[Mr.Children]幸せのカテゴリー.txt" "[Mr.Children]終わりなき旅.txt" [25] "[Mr.Children]PADDLE.txt" "[Mr.Children]Replay.txt" [27] "[Mr.Children]ジェラシー.txt" "[Mr.Children]ロックンロールは生きている.txt" [29] "[Mr.Children]こんな風にひどく蒸し暑い日.txt" "[Mr.Children]LOVE はじめました.txt" [31] "[Mr.Children]君が好き.txt" "[Mr.Children]Drawing.txt" [33] "[Mr.Children]通り雨.txt" "[Mr.Children]GIFT.txt" [35] "[Mr.Children]ティーンエイジ・ドリーム（I～II）.txt" "[Mr.Children]掌.txt" [37] "[Mr.Children]十二月のセントラルパークブルース.txt" "[Mr.Children]Worlds end.txt" [39] "[Mr.Children]イミテーションの木.txt" "[Mr.Children]So Let's Get Truth.txt" [41] "[Mr.Children]I'LL BE.txt" "[Mr.Children]innocent world.txt" [43] "[Mr.Children]インマイタウン.txt" "[Mr.Children]星になれたら.txt" [45] "[Mr.Children]エソラ.txt" "[Mr.Children]天頂バス.txt" [47] "[Mr.Children]End of the day.txt" "[Mr.Children]Happy Song.txt" [49] "[Mr.Children]#2601.txt" "[Mr.Children]ポケット カスタネット.txt" [51] "[Mr.Children]グッバイ・マイ・グルーミーデイズ.txt" "[Mr.Children]Door.txt" $negative.txt [1] "[Mr.Children]NOT FOUND.txt" "[Mr.Children]It's a wonderful world.txt" [3] "[Mr.Children]Everything (It's you).txt" "[Mr.Children]いつでも微笑みを.txt" [5] "[Mr.Children]友とコーヒーと嘘と胃袋.txt" "[Mr.Children]やわらかい風.txt" [7] "[Mr.Children]UFO.txt" "[Mr.Children]Simple.txt" [9] "[Mr.Children]All by myself.txt" "[Mr.Children]Dear wonderful world.txt" [11] "[Mr.Children]CROSS ROAD.txt" "[Mr.Children]Over.txt" [13] "[Mr.Children]ランニングハイ.txt" "[Mr.Children]ほころび.txt" [15] "[Mr.Children]もっと.txt" "[Mr.Children]ニシエヒガシエ.txt" [17] "[Mr.Children]DISCOVERY.txt" "[Mr.Children]箒星.txt" [19] "[Mr.Children]祈り～涙の軌道.txt" "[Mr.Children]旅立ちの唄.txt" [21] "[Mr.Children]かぞえうた.txt" "[Mr.Children]逃亡者.txt" [23] "[Mr.Children]空風の帰り道.txt" "[Mr.Children]花 －Memento-Mori－.txt" [25] "[Mr.Children]水上バス.txt" "[Mr.Children]Pink～奇妙な夢.txt" [27] "[Mr.Children]and I close to you.txt" "[Mr.Children]手紙.txt" [29] "[Mr.Children]ALIVE.txt" "[Mr.Children]つよがり.txt" [31] "[Mr.Children]ラララ.txt" "[Mr.Children]クラスメイト.txt" [33] "[Mr.Children]Any.txt" "[Mr.Children]渇いたkiss.txt" [35] "[Mr.Children]ロード・アイ・ミス・ユー.txt" "[Mr.Children]声.txt" [37] "[Mr.Children]Heavenly kiss.txt" "[Mr.Children]one two three.txt" [39] "[Mr.Children]ためいきの日曜日.txt" "[Mr.Children]さよならは夢の中へ.txt" [41] "[Mr.Children]デルモ.txt" "[Mr.Children]車の中でかくれてキスをしよう.txt" [43] "[Mr.Children]Mirror.txt" "[Mr.Children]光の射す方へ.txt" [45] "[Mr.Children]花言葉.txt" "[Mr.Children]独り言.txt" [47] "[Mr.Children]いつの日にか二人で.txt" "[Mr.Children]Brandnew my lover.txt" [49] "[Mr.Children]CHILDREN'S WORLD.txt" "[Mr.Children]Another Mind.txt" [51] "[Mr.Children]風 ～The wind knows how I feel～.txt" "[Mr.Children]Dance Dance Dance.txt" [53] "[Mr.Children]スロースターター.txt" "[Mr.Children]Mr.Shining Moon.txt" [55] "[Mr.Children]妄想満月.txt" "[Mr.Children]I'm sorry.txt" $plus.txt [1] "[Mr.Children]羊、吠える.txt" "[Mr.Children]LOVE.txt" [3] "[Mr.Children]深海.txt" "[Mr.Children]夏が終わる ～夏の日のオマージュ～.txt" [5] "[Mr.Children]ファスナー.txt" "[Mr.Children]名もなき詩.txt" [7] "[Mr.Children]ロザリータ.txt" "[Mr.Children]蒼.txt" [9] "[Mr.Children]ラヴ コネクション.txt" "[Mr.Children]彩り.txt" [11] "[Mr.Children]ロックンロール.txt" "[Mr.Children]Bird Cage.txt" [13] "[Mr.Children]Love is Blindness.txt" "[Mr.Children]Asia(エイジア).txt" [15] "[Mr.Children]Forever.txt" "[Mr.Children]マシンガンをぶっ放せ.txt" [17] "[Mr.Children]擬態.txt" "[Mr.Children]タダダキアッテ.txt" [19] "[Mr.Children]口がすべって.txt" "[Mr.Children]抱きしめたい.txt" [21] "[Mr.Children]僕らの音.txt" "[Mr.Children]また会えるかな.txt" [23] "[Mr.Children]くるみ.txt" "[Mr.Children]I.txt" [25] "[Mr.Children]少年.txt" "[Mr.Children]あんまり覚えてないや.txt" [27] "[Mr.Children]pieces.txt" "[Mr.Children]タガタメ.txt" [29] "[Mr.Children]1999年、夏、沖縄.txt" "[Mr.Children]Prelude.txt" [31] "[Mr.Children]アンダーシャツ.txt" "[Mr.Children]蘇生.txt" [33] "[Mr.Children]HOWL.txt" "[Mr.Children]シーラカンス.txt" [35] "[Mr.Children]PIANO MAN.txt" "[Mr.Children]虹の彼方へ.txt" [37] "[Mr.Children]Marshmallow day.txt" "[Mr.Children]横断歩道を渡る人たち.txt" [39] "[Mr.Children]傘の下の君に告ぐ.txt" "[Mr.Children]しるし.txt" [41] "[Mr.Children]Image.txt" "[Mr.Children]思春期の夏 ～君との恋が今も牧場に～.txt" [43] "[Mr.Children]メインストリートに行こう.txt" "[Mr.Children]ボレロ.txt" [45] "[Mr.Children]Another Story.txt" $positive.txt [1] "[Mr.Children]過去と未来と交信する男.txt" [2] "[Mr.Children]SUNRISE.txt" [3] "[Mr.Children]僕が僕であるために.txt" [4] "[Mr.Children]風と星とメビウスの輪.txt" [5] "[Mr.Children]Everything is made from a dream.txt" [6] "[Mr.Children]安らげる場所.txt" [7] "[Mr.Children]my sweet heart.txt" [8] "[Mr.Children]Sign.txt" [9] "[Mr.Children]蜃気楼.txt" [10] "[Mr.Children]シーソーゲーム ～勇敢な恋の歌～.txt" [11] "[Mr.Children]ロードムービー.txt" [12] "[Mr.Children]BLUE.txt" [13] "[Mr.Children]ゆりかごのある丘から.txt" [14] "[Mr.Children]言わせてみてぇもんだ.txt" [15] "[Mr.Children]Distance.txt" [16] "[Mr.Children]Tomorrow never knows.txt" [17] "[Mr.Children]雨のち晴れ.txt" [18] "[Mr.Children]隔たり.txt" [19] "[Mr.Children]my confidence song.txt" [20] "[Mr.Children]ハル.txt" [21] "[Mr.Children]靴ひも.txt" [22] "[Mr.Children]fanfare.txt" [23] "[Mr.Children]タイムマシーンに乗って.txt" [24] "[Mr.Children]フェイク.txt" [25] "[Mr.Children]ひびき.txt" [26] "[Mr.Children]君がいた夏.txt" [27] "[Mr.Children]ありふれた Love Story ～男女問題はいつも面倒だ～.txt" [28] "[Mr.Children]跳べ.txt" [29] "[Mr.Children]Prism.txt" [30] "[Mr.Children]花の匂い.txt" [31] "[Mr.Children]Monster.txt" [32] "[Mr.Children]血の管.txt" [33] "[Mr.Children]優しい歌.txt" [34] "[Mr.Children]友達のままで.txt" [35] "[Mr.Children]CENTER OF UNIVERSE.txt" [36] "[Mr.Children]君の事以外は何も考えられない.txt" [37] "[Mr.Children]ヨーイドン.txt" [38] "[Mr.Children]虜.txt" [39] "[Mr.Children]everybody goes ～秩序のない現代にドロップキック～.txt" [40] "[Mr.Children]潜水.txt"
