http://yixuan.cos.name/cn/2011/03/text-mining-of-song-poems/
看了宋词频率统计的例子，想用php实现一下，php的split中文让我崩溃了。。。
php都5.3.8了，一个中文的问题还搞得这么烂。。。
-----------------------------------------------------
(add php源码)
终于能workaround搞定中文的php
<?php结果
ini_set("memory_limit","1024M");
echo "start\n";
$file_handle = fopen("tangshi.txt", "r");
$ci = array();
$ct = 0;
while (!feof($file_handle)) {
$line = fgets($file_handle);
if (preg_match('/\d+/',$line)) continue;
if (strlen($line) > 6*3 && strlen($line) < 500*3)
{
$ci[$ct++] = $line;
}
}
fclose($file_handle);
$groupcount = 3;
$sentence = array();
$ct = 0;
echo "start sentence\n";
foreach ($ci as $v) {
preg_match_all('/([\w]+)/u', $v, $matches);
$li = $matches[0];
foreach ($li as $val)
{
if (strlen($val) >= $groupcount*3)
{
$sentence[$ct++] = $val;
}
}
}
$word = array();
$ct = 0;
echo "start word\n";
foreach ($sentence as $v) {
if (strlen($v) > 500*3 || strlen($v) < $groupcount*3) continue;
for($i = 0; $i <= strlen($v)-$groupcount*3; $i+=3)
{
$word[$ct++] = substr($v,$i,$groupcount*3);
}
}
echo "start statistic\n";
$statistic = array();
foreach ($word as $v) {
if (array_key_exists($v, $statistic))
{
$statistic[$v] = $statistic[$v] + 1;
}
else
{
$statistic[$v] = 1;
}
}
echo "start output\n";
arsort($statistic);
while(list($key,$val) = each($statistic)) {
if ($val > 50)
{
echo $key . $val . "\r\n";
}
}
echo "done";
?>
倚阑干125 知何处107广寒宫94到如今89东风吹86留不住79人何处76有谁知74三十六70西风吹66云深处63人间世62不知何62人不见61人千里58君知否58与谁同57不如归55春归去54归去来53年今日53何处是52
东风1400 何处1224 人间1196 风流848 春风800 归去793 西风780 江南773 归来770 相思763 梅花744 千里675 明月659 回首648 多少646 如今636 阑干632 年年608 万里592 一笑586 黄昏550 天涯538 当年537 一枝528 芳草527 相逢524 尊前516 风雨506 故人505 流水475 风吹472 依旧470 木兰467 风月460 多情455 当时446 无人444 斜阳438 减字435 不知434 兰花428 不见423 深处422 字木412 时节401 平生399 春色396 凄凉390 功名380 一点378 匆匆377 无限377 天上374 今日369 西湖360 杨柳360 桃花355 消息350 扁舟349 芙蓉344 憔悴338 神仙335 何事334 桃李333 一片332 十分332 心事329 黄花329 人生328 一声325 佳人322 长安320 断肠316 东君315 鸳鸯314 为谁313 而今313 少年310 十年310 去年309 海棠308 无情306 不是306 昨夜304 富贵303 时候303 行人302 今夜301 江上300 蓬莱300 不似299 青山296 谁知296 寂寞296 几度295 天气295 何时293 惟有293 一曲292 肠断292 往事291 月明290 笙歌287 如何285 燕子282 清明281 人在281 悠悠280 无数280 明年279 落花278 千古270 一番269 十二266 精神266 梨花264 今年263 十里261 垂杨261 分付260 何人260 年时260 春归259 *258 有人257 帘幕257 思量256 几时256 如此255 如许252 秋风252 明朝249 夕阳248 记得247 清风244 风露243 不须243 只有243 相对240 不堪240 酒醒239 从今239 庭院239 又是238 今朝238 缥缈238 盈盈237 花开237 不如236 殷勤236 风光235 夜来233 无语232 分明232 只恐229 旧时229 倚阑229 秋千228 依然228 今宵228 行云227 春来224 自有224 携手223 金缕221 花前221
ps唐诗的结果
何处1662 不知1462 万里1445 千里1299 今日1159 不见1155 不可1144 春风1123 白云1105 不得943 人间890 明月886 无人875 风吹832 故人778 惆怅775 秋风745 悠悠737 相思733 长安721 白日689 如何684 十年675 青山664 何人659 少年631 相逢627 平生589 年年587 寂寞586 天子585 天地584 人不583 黄金580 何事575 江上555 流水547 可怜535 回首533 如此521 主人520 白发520 今朝515 月明512 从此505 日月505 行人501 不如495 三十494 将军494 归去493 落日493 日暮487 不能482 别离480 洛阳474 何时474 此时472 天下471 芳草470 归来469 无事468 相见466 夕阳459 江南458 当时454 杨柳452 风雨450 东风440 青云435 洞庭432 参差432 花落430 落花429 天涯429 芙蓉428 清风423 不是420 烟霞415 白头415 桃花412 不相411 唯有408 君不408 何如407 南山403 谁能398 千年392 如今392 天上389 十二387 花开384 与君383 桃李381 君王379 终日379 殷勤379 此地377 浮云377 二十377 苍苍374 门前374 凤凰374 神仙373 千万370 山中367 美人363 鸳鸯361 有时360 不敢360 自有359 无限356 有馀355 风起354 处处353 萧萧352 一声352 尽日352 三千351 风流349 山川349 君子348 春色343 故乡343 萧条341 何必339 裴回339 分明338 不堪336 几时333
君不见234 不知何127 行路难108 三千里108 不可见100 三十六94 在何处90 知何处90 二十年87 卷三百84 百六十82 卷八百81 卷五百81 卷二百81 卷七百81 卷四百81 卷一百81 卷六百80 百四十77 百二十76 三十年75 百七十75 百八十74 百九十74 百三十74 不相见73 无消息73 百一十73 何处去72 百五十72 无一事70 洛阳城69 千万里69 何处是68 水东流68 向人间66 归未得65 歌一曲62 一杯酒61 千里外61 明月夜58 归何处57 从此去56 东风吹56 归去来55 今何在55 人不知55 草萋萋55 春风吹54 无人知53 不知谁53 人不见53 不见人52 与君同52 不得意52 长安道52 复何如51 人间事51
-----------------------------------------------------
于是看看 yixuan 写的R语言代码，下了个R语言环境。
运行结果命令行里面显示的中文都是空的。。。
原来R语言的命令行不支持显示中文。。。。
改下输出到文件，这才看到结果
l = scan("Ci.txt", "character", sep = "\n");
l.len = nchar(l);
# 某些行是作者和标题，所以选取长度大于10的行；
# 另外这个文本文件不太规整，有些网址什么的，
# 所以也要排除那些长度太长的。
ci = l[l.len > 10 & l.len < 500];
# 句子用标点符号分割。
sentences = strsplit(ci, "，|。|！|？|、");
sentences = unlist(sentences);
sentences = sentences[sentences != ""];
s.len = nchar(sentences);
# 单句太长了说明有可能是错误的字符，去除掉。
sentences = sentences[s.len <= 10 & s.len >= 2];
s.len = nchar(sentences);
# 暴力挨个拆分，比如“犹解嫁东风”的所有二字组合为
# “犹解”“解嫁”“嫁东”“东风”，
# 无意义的词其频数自然就落在后面了。
splitwords = function(x, x.len) substring(x, 1:(x.len+1 - 2), 2:x.len);
words = mapply(splitwords, sentences, s.len, SIMPLIFY = TRUE, USE.NAMES = FALSE);
words = unlist(words);
words.freq = table(words);
words.freq = sort(words.freq, decreasing = TRUE);
df<-data.frame(Word = names(words.freq[1:100]), Freq = as.integer(words.freq[1:100]));
write.table(df, "1.txt");
修改绿色部分就可以获得3个字和4个字的结果
这回结果终于对了。
"Word" "Fre"
1,□□,1485
2,东风,1382
3,何处,1230
4,人间,1202
5,风流,857
6,归去,812
7,春风,802
8,西风,779
9,归来,771
10,江南,765
11,相思,753
12,梅花,732
13,千里,676
14,回首,656
15,明月,651
16,多少,648
17,如今,642
18,阑干,630
19,年年,613
20,万里,590
21,一笑,582
22,黄昏,550
23,当年,542
24,天涯,537
25,相逢,528
26,芳草,527
27,尊前,516
28,一枝,512
29,风雨,505
30,流水,472
31,依旧,472
32,风吹,471
33,风月,461
34,多情,457
35,故人,451
36,当时,450
37,无人,445
38,斜阳,438
39,不知,430
40,不见,429
41,深处,422
42,时节,403
43,平生,398
44,凄凉,398
45,春色,394
46,匆匆,383
47,功名,383
48,一点,378
49,无限,377
50,今日,369
51,天上,368
52,杨柳,362
53,西湖,356
54,桃花,354
55,扁舟,353
56,消息,351
57,憔悴,344
58,何事,339
59,芙蓉,338
60,神仙,334
61,一片,334
62,桃李,333
63,人生,332
64,十分,331
65,心事,329
66,黄花,328
67,一声,325
68,佳人,324
69,长安,321
70,东君,319
71,断肠,316
72,而今,315
73,鸳鸯,314
74,为谁,313
75,十年,310
76,去年,309
77,少年,308
78,海棠,307
79,寂寞,306
80,无情,306
81,不是,305
82,时候,304
83,肠断,303
84,富贵,303
85,蓬莱,303
86,昨夜,303
87,行人,302
88,今夜,301
89,谁知,300
90,不似,299
91,江上,298
92,悠悠,296
93,几度,295
94,青山,295
95,何时,294
96,天气,293
97,惟有,293
98,一曲,291
99,月明,291
100,往事,290
"少年" 308
"78" "海棠" 307
"79" "寂寞" 306
"80" "无情" 306
"81" "不是" 305
"82" "时候" 304
"83" "肠断" 303
"84" "富贵" 303
"85" "蓬莱" 303
"86" "昨夜" 303
"87" "行人" 302
"88" "今夜" 301
"89" "谁知" 300
"90" "不似" 299
"91" "江上" 298
"92" "悠悠" 296
"93" "几度" 295
"94" "青山" 295
"95" "何时" 294
"96" "天气" 293
"97" "惟有" 293
"98" "一曲" 291
"99" "月明" 291
"100" "往事" 290
三个字的
"Word" "Freq"
"1" "□□□" 893
"2" "倚阑干" 125
"3" "知何处" 107
"4" "广寒宫" 94
"5" "到如今" 89
"6" "东风吹" 85
"7" "留不住" 76
"8" "人何处" 76
"9" "有谁知" 74
"10" "三十六" 70
"11" "西风吹" 66
"12" "云深处" 63
"13" "不知何" 62
"14" "人间世" 61
"15" "人不见" 60
"16" "君知否" 58
"17" "人千里" 58
"18" "与谁同" 57
"19" "不如归" 54
"20" "春归去" 54
"21" "年今日" 53
"22" "何处是" 52
"23" "归去来" 51
"24" "二十四" 47
"25" "归何处" 47
"26" "花深处" 47
"27" "江南春" 46
"28" "向尊前" 46
"29" "花飞絮" 45
"30" "花流水" 45
"31" "记当年" 44
"32" "雨初晴" 43
"33" "长安道" 43
"34" "倚东风" 42
"35" "功名事" 41
"36" "海棠开" 41
"37" "谁知道" 41
"38" "知多少" 41
"39" "东风里" 40
"40" "海棠花" 40
"41" "老人星" 40
"42" "那堪更" 40
"43" "桥流水" 40
"44" "山无数" 40
"45" "在何处" 40
"46" "断人肠" 39
"47" "水晶宫" 39
"48" "月清风" 39
"49" "落花飞" 38
"50" "如归去" 38
"51" "无消息" 38
"52" "花无数" 37
"53" "人间天" 37
"54" "随流水" 37
"55" "TXT" 36
"56" "杜鹃啼" 36
"57" "歌金缕" 36
"58" "个人人" 36
"59" "故人相" 36
"60" "间天上" 36
"61" "明月清" 36
"62" "送春归" 36
"63" "影横斜" 36
"64" "春消息" 35
"65" "人间何" 35
"66" "一番新" 35
"67" "一年春" 35
"68" "一枝春" 35
"69" "二十年" 34
"70" "寄相思" 34
"71" "去来兮" 34
"72" "去年今" 34
"73" "人如玉" 34
"74" "是人间" 34
"75" "寿阳妆" 34
"76" "无觅处" 34
"77" "有个人" 34
"78" "此时情" 33
"79" "江南路" 33
"80" "留春住" 33
"81" "然一笑" 33
"82" "唱阳关" 32
"83" "分付与" 32
"84" "何处去" 32
"85" "江南岸" 32
"86" "人憔悴" 32
"87" "人去后" 32
"88" "十年前" 32
"89" "天如水" 32
"90" "须信道" 32
"91" "一枝斜" 32
"92" "又是一" 32
"93" "几时休" 31
"94" "尽人间" 31
"95" "天上人" 31
"96" "月黄昏" 31
"97" "在人间" 31
"98" "最好是" 31
"99" "暗香浮" 30
"100" "断肠声" 30
四个字
"Word" "Freq"
"1" "□□□□" 514
"2" "不知何处" 39
"3" "不如归去" 36
"4" "人间天上" 36
"5" "归去来兮" 34
"6" "明月清风" 34
"7" "广寒宫殿" 29
"8" "落花飞絮" 28
"9" "天上人间" 28
"10" "江南江北" 27
"11" "落花流水" 26
"12" "绿鬓朱颜" 26
"13" "一觞一咏" 26
"14" "去年今日" 25
"15" "人间何处" 25
"16" "朝朝暮暮" 24
"17" "三万六千" 24
"18" "疏影横斜" 24
"19" "清风明月" 23
"20" "三十六宫" 23
"21" "十洲三岛" 23
"22" "暗香浮动" 22
"23" "独倚阑干" 22
"24" "良辰美景" 22
"25" "功名富贵" 21
"26" "寒食清明" 21
"27" "岁岁年年" 21
"28" "五云深处" 21
"29" "嫣然一笑" 21
"30" "一轮明月" 21
"31" "朱颜绿鬓" 21
"32" "不堪回首" 20
"33" "二十四桥" 20
"34" "江北江南" 20
"35" "十二阑干" 20
"36" "冰肌玉骨" 19
"37" "满城风雨" 19
"38" "钱塘江上" 19
"39" "竹篱茅舍" 19
"40" "百花头上" 18
"41" "登山临水" 18
"42" "故人何处" 18
"43" "纶巾羽扇" 18
"44" "前度刘郎" 18
"45" "夜来风雨" 18
"46" "一钩新月" 18
"47" "有个人人" 18
"48" "玉皇香案" 18
"49" "葱葱佳气" 17
"50" "道骨仙风" 17
"51" "故人千里" 17
"52" "凌波微步" 17
"53" "茂林修竹" 17
"54" "赏心乐事" 17
"55" "数声啼鸟" 17
"56" "斜风细雨" 17
"57" "一年春事" 17
"58" "玉骨冰肌" 17
"59" "非烟非雾" 16
"60" "南北东西" 16
"61" "年年今日" 16
"62" "沈香亭北" 16
"63" "春归何处" 15
"64" "广寒宫阙" 15
"65" "记得年时" 15
"66" "桃花流水" 15
"67" "小春时候" 15
"68" "燕子归来" 15
"69" "又是一番" 15
"70" "与花为主" 15
"71" "整顿乾坤" 15
"72" "尊前一笑" 15
"73" "今夕何夕" 14
"74" "金风玉露" 14
"75" "绿窗朱户" 14
"76" "年年岁岁" 14
"77" "千岩万壑" 14
"78" "人生行乐" 14
"79" "如今憔悴" 14
"80" "天长地久" 14
"81" "仙风道骨" 14
"82" "一叶扁舟" 14
"83" "暗香疏影" 13
"84" "不比寻常" 13
"85" "潮生潮落" 13
"86" "对酒当歌" 13
"87" "富贵功名" 13
"88" "画桥流水" 13
"89" "*如画" 13
"90" "七十古来" 13
"91" "曲水流觞" 13
"92" "十古来稀" 13
"93" "天寒日暮" 13
"94" "天若有情" 13
"95" "文章太守" 13
"96" "细雨斜风" 13
"97" "一年一度" 13
"98" "倚遍阑干" 13