文本处理之贝叶斯垃圾邮件分类

本文所讲解的是如何通过Python将文本读取,并且将每一个文本生成对应的词向量并返回. 文章的背景是将50封邮件(包含25封正常邮件,25封垃圾邮件)通过贝叶斯算法对其进行分类.

主要分为如下几个部分:
①读取所有邮件;
②建立词汇表;
③生成没封邮件对应的词向量(词集模型);
④用sklearn中的朴素贝叶斯算法进行分类;
⑤生成性能评估报告

1.函数介绍

下面先介绍需要用到的功能函数

1.1建立词汇表

思路:用所给的文本建立一个词汇表;就是将用所有出现的单词构成一个不重复的集合,即不含同一个单词.

def createVocabList(dataSet):
    vocabSet = set([])  #create empty set
for document in dataSet:
        vocabSet = vocabSet | set(document) #union of the two sets
return list(vocabSet)

postingList=[['my', 'dog', 'dog','has']]
print createVocabList(postingList)
>> ['has', 'my', 'dog']

1.2 将所有的大写字母转换成小写字母,并且去掉长度小于两个字符的单词

def textParse(bigString):    #input is big string, #output is word list
import re
    listOfTokens = re.split(r'\W*', bigString)
return [tok.lower() for tok in listOfTokens if len(tok) > 2]
# 去掉长度小于两个字符的单词,2可以自己调节

s = 'i Love YYUU'
print textParse(s)
>> ['love', 'yyuu']

1.3将每一个文本变成一个词向量

构建词向量有两种方式:第一种是用文本里面出现的单词,同词汇表向量进行对比,如果出现在词汇表中,则对应位置为1,反之为0.这种方式只管有无出现,不管出现次数,称为词集模型(set-of-words model);另外一种就是,同时也统计出现次数,称为词袋模型(bag-of-words model).

def setOfWords2Vec(vocabList, inputSet):
    returnVec = [0]*len(vocabList)
for word in inputSet:
if word in vocabList:
            returnVec[vocabList.index(word)] = 1
else: print "the word: %s is not in my Vocabulary!" % word
return returnVec

vocabulary = ['wo','do','like','what','go']
text = ['do','go','what','do']
print setOfWords2Vec(vocabulary,text)
>> [0, 1, 0, 1, 1]

def bagOfWords2Vec(vocabList, inputSet):
    returnVec = [0]*len(vocabList)
for word in inputSet:
if word in vocabList:
            returnVec[vocabList.index(word)] += 1
else: print "the word: %s is not in my Vocabulary!" % word
return returnVec

vocabulary = ['wo','do','like','what','go']
text = ['do','go','what','do']
print setOfWords2Vec(vocabulary,text)
>> [0, 2, 0, 1, 1]

2.整合函数

将上面三个函数写在一起;下面的操作方式只是针对本例,但是只要稍作修改同样能够适应其它地方.

def createVocabList(dataSet):# 建立词汇表
    vocabSet = set([])  #create empty set
for document in dataSet:
        vocabSet = vocabSet | set(document) #union of the two sets
return list(vocabSet)

def setOfWords2Vec(vocabList, inputSet):# 建立词向量
    returnVec = [0]*len(vocabList)
for word in inputSet:
if word in vocabList:
            returnVec[vocabList.index(word)] = 1
else: print "the word: %s is not in my Vocabulary!" % word
return returnVec

def textParse(bigString):    #input is big string, #output is word list
import re
    listOfTokens = re.split(r'\W*', bigString)
return [tok.lower() for tok in listOfTokens if len(tok) > 2]

def preProcessing():
    docList=[]; classList = []; fullText =[]
for i in range(1,26):
        wordList = textParse(open('email/spam/%d.txt' % i).read())
        docList.append(wordList)# 读取文本
        classList.append(1)# 读取每个文本的标签
        wordList = textParse(open('email/ham/%d.txt' % i).read())
        docList.append(wordList)
        classList.append(0)
    vocabList = createVocabList(docList)#create vocabulary# 生成词向表
    data = []
    target = classList
for docIndex in range(50):# 本例一共有50个文本
        data.append(setOfWords2Vec(vocabList,docList[docIndex]))生成词向量
return data,target#返回处理好的词向量和标签

3.训练并预测

import textProcess as tp
from sklearn.naive_bayes import MultinomialNB
from sklearn.cross_validation import train_test_split
from sklearn.metrics import classification_report

data,target= tp.preProcessing()


X_train,X_test,y_train,y_test = train_test_split(data,target,test_size=0.25)

mnb = MultinomialNB()
mnb.fit(X_train,y_train)
y_pre = mnb.predict((X_test))
print y_pre # 预测结果
print y_test # 实际结果
print 'The accuracy of Naive Bayes Classifier is',mnb.score(X_test,y_test)
print classification_report(y_test,y_pre)

源码及数据集

参考

机器学习实战
Python机器学习及实践

秒客网

文本处理之贝叶斯垃圾邮件分类

相关文章