基本思路：
每个评论取前200个单词。然后生成词汇表，利用词汇index标注评论（对 每条评论的前200个单词编号而已），然后使用LSTM做正负评论检测。
代码解读见【【【评论】】】！embedding层本质上是word2vec！！！在进行数据降维，但是不是所有的LSTM都需要这个，比如在图像检测mnist时候，就没有这层！
import tensorflow as tf
from tensorflow.contrib.learn.python import learn
from sklearn import metrics
from sklearn.model_selection import train_test_split
import numpy as np
from sklearn.naive_bayes import GaussianNB
import os
from sklearn.feature_extraction.text import CountVectorizer
from tensorflow.contrib.layers.python.layers import encoders
from sklearn import svm
import tflearn
from tflearn.data_utils import to_categorical, pad_sequences
from tflearn.datasets import imdb MAX_DOCUMENT_LENGTH = 200
EMBEDDING_SIZE = 50 n_words=0 def load_one_file(filename):
x=""
with open(filename) as f:
for line in f:
x+=line
return x def load_files(rootdir,label):
list = os.listdir(rootdir)
x=[]
y=[]
for i in range(0, len(list)):
path = os.path.join(rootdir, list[i])
if os.path.isfile(path):
print "Load file %s" % path
y.append(label)
x.append(load_one_file(path))
return x,y def load_data():
x=[]
y=[]
x1,y1=load_files("../data/movie-review-data/review_polarity/txt_sentoken/pos/",0)
x2,y2=load_files("../data/movie-review-data/review_polarity/txt_sentoken/neg/", 1)
x=x1+x2
y=y1+y2
return x,y def do_rnn(trainX, testX, trainY, testY):
global n_words
# Data preprocessing
# Sequence padding
print "GET n_words embedding %d" % n_words trainX = pad_sequences(trainX, maxlen=MAX_DOCUMENT_LENGTH, value=0.)
testX = pad_sequences(testX, maxlen=MAX_DOCUMENT_LENGTH, value=0.)
# Converting labels to binary vectors
trainY = to_categorical(trainY, nb_classes=2)
testY = to_categorical(testY, nb_classes=2)
print trainX[:10]
print testX[:10]
# Network building
net = tflearn.input_data([None, MAX_DOCUMENT_LENGTH])
net = tflearn.embedding(net, input_dim=n_words, output_dim=128)
net = tflearn.lstm(net, 128, dropout=0.8)
net = tflearn.fully_connected(net, 2, activation='softmax')
net = tflearn.regression(net, optimizer='adam', learning_rate=0.001,
loss='categorical_crossentropy') # Training model = tflearn.DNN(net, tensorboard_verbose=3)
model.fit(trainX, trainY, validation_set=(testX, testY), show_metric=True,
batch_size=32,run_id="maidou") def do_NB(x_train, x_test, y_train, y_test):
gnb = GaussianNB()
y_predict = gnb.fit(x_train, y_train).predict(x_test)
score = metrics.accuracy_score(y_test, y_predict)
print('NB Accuracy: {0:f}'.format(score)) def main(unused_argv):
global n_words x,y=load_data() x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.4, random_state=0) vp = learn.preprocessing.VocabularyProcessor(max_document_length=MAX_DOCUMENT_LENGTH, min_frequency=1)
vp.fit(x)
x_train = np.array(list(vp.transform(x_train)))
x_test = np.array(list(vp.transform(x_test)))
n_words=len(vp.vocabulary_)
print('Total words: %d' % n_words) do_NB(x_train, x_test, y_train, y_test)
do_rnn(x_train, x_test, y_train, y_test) if __name__ == '__main__':
tf.app.run()
负面的示例评论：
plot : two teen couples go to a church party , drink and then drive .
they get into an accident .
one of the guys dies , but his girlfriend continues to see him in her life , and has nightmares .
what's the deal ?
watch the movie and " sorta " find out . . .
critique : a mind-fuck movie for the teen generation that touches on a very cool idea , but presents it in a very bad package .
which is what makes this review an even harder one to write , since i generally applaud films which attempt to break the mold , mess with your head and such ( lost highway & memento ) , but there are good and bad ways of making all types of films , and these folks just didn't snag this one correctly .
they seem to have taken this pretty neat concept , but executed it terribly .
so what are the problems with the movie ?
well , its main problem is that it's simply too jumbled .
it starts off " normal " but then downshifts into this " fantasy " world in which you , as an audience member , have no idea what's going on .
there are dreams , there are characters coming back from the dead , there are others who look like the dead , there are strange apparitions , there are disappearances , there are a looooot of chase scenes , there are tons of weird things that happen , and most of it is simply not explained .
now i personally don't mind trying to unravel a film every now and then , but when all it does is give me the same clue over and over again , i get kind of fed up after a while , which is this film's biggest problem .
it's obviously got this big secret to hide , but it seems to want to hide it completely until its final five minutes .
and do they make things entertaining , thrilling or even engaging , in the meantime ?
not really .
the sad part is that the arrow and i both dig on flicks like this , so we actually figured most of it out by the half-way point , so all of the strangeness after that did start to make a little bit of sense , but it still didn't the make the film all that more entertaining .
i guess the bottom line with movies like this is that you should always make sure that the audience is " into it " even before they are given the secret password to enter your world of understanding .
i mean , showing melissa sagemiller running away from visions for about 20 minutes throughout the movie is just plain lazy ! !
okay , we get it . . . there
are people chasing her and we don't know who they are .
do we really need to see it over and over again ?
how about giving us different scenes offering further insight into all of the strangeness going down in the movie ?
apparently , the studio took this film away from its director and chopped it up themselves , and it shows .
there might've been a pretty decent teen mind-fuck movie in here somewhere , but i guess " the suits " decided that turning it into a music video with little edge , would make more sense .
the actors are pretty good for the most part , although wes bentley just seemed to be playing the exact same character that he did in american beauty , only in a new neighborhood .
but my biggest kudos go out to sagemiller , who holds her own throughout the entire film , and actually has you feeling her character's unraveling .
overall , the film doesn't stick because it doesn't entertain , it's confusing , it rarely excites and it feels pretty redundant for most of its runtime , despite a pretty cool ending and explanation to all of the craziness that came before it .
oh , and by the way , this is not a horror or teen slasher flick . . . it's
just packaged to look that way because someone is apparently assuming that the genre is still hot with the kids .
it also wrapped production two years ago and has been sitting on the shelves ever since .
whatever . . . skip
it !
正面的：
films adapted from comic books have had plenty of success , whether they're about superheroes ( batman , superman , spawn ) , or geared toward kids ( casper ) or the arthouse crowd ( ghost world ) , but there's never really been a comic book like from hell before .
for starters , it was created by alan moore ( and eddie campbell ) , who brought the medium to a whole new level in the mid '80s with a 12-part series called the watchmen .
to say moore and campbell thoroughly researched the subject of jack the ripper would be like saying michael jackson is starting to look a little odd .
the book ( or " graphic novel , " if you will ) is over 500 pages long and includes nearly 30 more that consist of nothing but footnotes .
in other words , don't dismiss this film because of its source .
if you can get past the whole comic book thing , you might find another stumbling block in from hell's directors , albert and allen hughes .
getting the hughes brothers to direct this seems almost as ludicrous as casting carrot top in , well , anything , but riddle me this : who better to direct a film that's set in the ghetto and features really violent street crime than the mad geniuses behind menace ii society ?
the ghetto in question is , of course , whitechapel in 1888 london's east end .
it's a filthy , sooty place where the whores ( called " unfortunates " ) are starting to get a little nervous about this mysterious psychopath who has been carving through their profession with surgical precision .
when the first stiff turns up , copper peter godley ( robbie coltrane , the world is not enough ) calls in inspector frederick abberline ( johnny depp , blow ) to crack the case .
abberline , a widower , has prophetic dreams he unsuccessfully tries to quell with copious amounts of absinthe and opium .
upon arriving in whitechapel , he befriends an unfortunate named mary kelly ( heather graham , say it isn't so ) and proceeds to investigate the horribly gruesome crimes that even the police surgeon can't stomach .
i don't think anyone needs to be briefed on jack the ripper , so i won't go into the particulars here , other than to say moore and campbell have a unique and interesting theory about both the identity of the killer and the reasons he chooses to slay .
in the comic , they don't bother cloaking the identity of the ripper , but screenwriters terry hayes ( vertical limit ) and rafael yglesias ( les mis ? rables ) do a good job of keeping him hidden from viewers until the very end .
it's funny to watch the locals blindly point the finger of blame at jews and indians because , after all , an englishman could never be capable of committing such ghastly acts .
and from hell's ending had me whistling the stonecutters song from the simpsons for days ( " who holds back the electric car/who made steve guttenberg a star ? " ) .
don't worry - it'll all make sense when you see it .
now onto from hell's appearance : it's certainly dark and bleak enough , and it's surprising to see how much more it looks like a tim burton film than planet of the apes did ( at times , it seems like sleepy hollow 2 ) .
the print i saw wasn't completely finished ( both color and music had not been finalized , so no comments about marilyn manson ) , but cinematographer peter deming ( don't say a word ) ably captures the dreariness of victorian-era london and helped make the flashy killing scenes remind me of the crazy flashbacks in twin peaks , even though the violence in the film pales in comparison to that in the black-and-white comic .
oscar winner martin childs' ( shakespeare in love ) production design turns the original prague surroundings into one creepy place .
even the acting in from hell is solid , with the dreamy depp turning in a typically strong performance and deftly handling a british accent .
ians holm ( joe gould's secret ) and richardson ( 102 dalmatians ) log in great supporting roles , but the big surprise here is graham .
i cringed the first time she opened her mouth , imagining her attempt at an irish accent , but it actually wasn't half bad .
the film , however , is all good .
2 : 00 - r for strong violence/gore , sexuality , language and drug content
pad后和category后的数据示例：
padded and cated data:
trainX 3条:
[[ 1299 6 1 1596 26 354 155 1 62 101 537 252
5 22 2048 516 4 140 252 119 19 1 147 226
16 56 19 2 435 37 2 77 648 15 1 164
222 22 389 12 93 39 19392 235 16 189 83 1299
6 2 1426 453 7 976 1375 97 1 67 6 2928
42 2489 58 1 251 225 3 36 4 133 120 305
138 8 4730 244 1274 70 1018 4 49 14539 2290 947
3881 22772 1594 1296 67 1812 11 2663 9397 7513 3133 2
1619 8232 307 16958 4 2015 1329 1 16813 3571 4 869
3376 5 1019 41 7 518 33 598 7 1 1600 4
15406 1473 29 2 77 199 812 15956 21 33 1841 315
1852 371 5280 27 468 2663 343 2 334 11397 1619 5
1562 47 19 0 3 4239 11 100 10 234 219 10
0 0 8 30 4 220 144 1 414 4 3226 11120
3161 92 299 366 725 1010 27520 5 3343 76 7 1
1205 12 12549 1121 4 44 1 2195 9938 6 23 0
12 2663 6858 5 1425 19 2 1378]
[ 1361 1 1647 4 1 4974 130 26 11041 1126 130 1232
1 57 26 7 269 641 5 205 3325 1053 3 5152
6318 622 2 5999 4 911 223 14 3772 5166 15739 6635
2036 633 1 2146 778 2697 327 9589 8311 3 3031 19
36 1 4974 8164 28 1 3103 4276 6344 27 618 2
4266 5 1 4203 1427 1199 1083 7 150 192 1 2294
3 15520 185 52 6 2 3689 572 4 6431 15520 6635
6 130 1232 2 5020 778 503 12 36 2805 4 1538
9333 4795 1518 4 25 405 1539 17927 6489 1427 6646 34
17491 13 3501 99 1232 5309 17 90 2 4074 1232 32
68 13660 162 5 2 7412 258 83 4 405 460 11
8238 12857 18618 3890 922 3915 3 146 32 5 488 10
2125 9736 5 2 2217 16298 3915 81 2529 48 1232 996
4 54 1053 522 18 157 9 410 24 25 4 23045
348 24 1535 35 1689 1 5410 1232 23995 3 4 220
9 340 41 1053 6 1391 18618 9608 16865 1232 24 272
6 681 7 0 100 20 109 642]
[ 83 59 25 11208 9 371 3442 7 2 546 181 29
176 158 13 546 25133 3 13 1554 4819 20 25356 12
36 46 5311 1 1075 4 3442 169 31 134 5 75
11 1 98 44 104 22 6759 12 2 13377 235 1397
4 1 1948 826 26697 371 3442 1605 13 260 1364 12771
4462 7 2 429 1340 29 1 164 63 1142 7782 4587
1599 6 7 1 1758 1217 12 2 541 8661 1142 168
10363 541 3 9 588 33 5 826 37 1 546 4553
4 36 140 300 93 97 361 168 2 8661 28 2
1988 508 3 102 6 2524 5 7651 100 516 1180 20
4837 11 13791 5 1 67 8 115 245 529 391 109
2 821 78 578 198 715 5 103 1218 95 5 1
415 662 337 415 1605 337 1415 3 10 2 571 6
2311 2812 3 10809 3442 3 1599 245 2349 5 4402 87
4339 3 18 1422 6642 12 2 11316 8790 5 819 46
116 266 193 1599 32 1585 5 141 85 7 546 487
144 18 1537 3442 124 41 4 13]]
trainY 3条:
[[ 1. 0.] [ 0. 1.] [ 0. 1.]]
其中，MAX_DOCUMENT_LENGTH = 200，由于每个文档都进行了剪切。超过200的就直接截断文本，不再计算了！！！因为：
tf.contrib.learn.preprocessing.VocabularyProcessor (max_document_length, min_frequency=0, vocabulary=None, tokenizer_fn=None)参数：
max_document_length: 文档的最大长度。如果文本的长度大于最大长度，那么它会被剪切，反之则用0填充。 min_frequency: 词频的最小值，出现次数小于最小词频则不会被收录到词表中。 vocabulary: CategoricalVocabulary 对象。 tokenizer_fn：分词函数
代码：
from tensorflow.contrib import learn
import numpy as np
max_document_length = 4
x_text =[
'i love you',
'me too'
]
vocab_processor = learn.preprocessing.VocabularyProcessor(max_document_length)
vocab_processor.fit(x_text)
print next(vocab_processor.transform(['i me too'])).tolist()
x = np.array(list(vocab_processor.fit_transform(x_text)))
print x
[1, 4, 5, 0]
[[1 2 3 0]
[4 5 0 0]]