用WordNet实现中文情感分析

时间:2024-10-25 08:37:50

1.      分析

中文的情感分析可以用词林做,词林有一大类(G类)对应心理活动,但是相对于wordnet还是太简单了.因此使用nltk+wordnet的方案,如下:

1)       中文分词:结巴分词

2)       中英文翻译:wordnet汉语开放词网,可从以下网址下载:
/cow/

3)       情感分析:wordnet的sentiwordnet组件

4)       停用词:参考以下网页,另外加入常用标点符号
/u010533386/article/details/51458591

2.      代码

# encoding=utf-8
import jieba
import sys
import codecs

reload(sys)

import nltk
from  import wordnet as wn
from  import sentiwordnet as swn

('utf8')

def doSeg(filename) :
    f = open(filename, 'r+')
    file_list = ()
    ()

    seg_list = (file_list)

    stopwords = []  
    for word in open("./stop_words.txt", "r"):  
        (()) 

    ll = []
    for seg in seg_list :
        if (("utf-8") not in stopwords and seg != ' ' and seg != '' and seg != "\n" and seg != "\n\n"):
            (seg)
    return ll

def loadWordNet():
    f = ("./", "rb", "utf-8")
    known = set()
    for l in f:
        if ('#') or not ():
            continue
        row = ().split("\t")
        if len(row) == 3:
            (synset, lemma, status) = row 
        elif len(row) == 2:
            (synset, lemma) = row 
            status = 'Y'
        else:
            print "illformed line: ", ()
        if status in ['Y', 'O' ]:
            if not ((), ()) in known:
                (((), ()))
    return known

def findWordNet(known, key):
    ll = [];
    for kk in known:
        if (kk[1] == key):
             (kk[0])
    return ll

def id2ss(ID):
    return wn._synset_from_pos_and_offset(str(ID[-1:]), int(ID[:8]))

def getSenti(word):
    return swn.senti_synset(())

if __name__ == '__main__' :
    known = loadWordNet()
    words = doSeg([1])

    n = 0
    p = 0
    for word in words:
      ll = findWordNet(known, word)
      if (len(ll) != 0):
          n1 = 0.0
          p1 = 0.0
          for wid in ll:
              desc = id2ss(wid)
              swninfo = getSenti(desc)
              p1 = p1 + swninfo.pos_score()
              n1 = n1 + swninfo.neg_score()
          if (p1 != 0.0 or n1 != 0.0):
              print word, '-> n ', (n1 / len(ll)), ", p ", (p1 / len(ll))
          p = p + p1 / len(ll)
          n = n + n1 / len(ll)
    print "n", n, ", p", p

3.      待解决的问题

1)       结巴分词与wordnet chinese中的词不能一一对应
结巴分词虽然可以导入自定义的词典,但仍有些结巴分出的词,在wordnet找不到对应词义,比如"太后","童子",还有一些组合词如"很早已前","黄山"等等.大多是名词,需要进一步"学习".
临时的解决方案是:将其当作"专有名词"处理

2)       一词多义/一义多词
无论是情感分析,还是语义分析,中文或者英文,都需要解决词和义的对应问题.
临时的解决方案是:找到该词的所有语义,取其平均的情感值.另外,结巴也可判断出词性作为进一步参考.

3)       语义问题
语义问题是最根本的问题,一方面需要分析句子的结构,另外也和内容也有关,尤其是长文章,经常会使用"先抑后扬""对比分析",这样就比较难以判断感情色彩了.

4.      参考

1)       Learning lexical scales:WordNet and SentiWordNet
/

2)       SentiWordNet Interface
/howto/