Python3自然语言（NLTK）——语言大数据

NLTK

这是一个处理文本的python库，我们知道文字性的知识可是拥有非常庞大的数据量，故而这属于大数据系列。

本文只是浅尝辄止，目前本人并未涉及这块知识，只是偶尔好奇，才写本文。

从NLTK中的book模块中，载入所有条目

• book 模块包含所有数据

from nltk.book import *

*** Introductory Examples for the NLTK Book ***
Loading text1, ..., text9 and sent1, ..., sent9
Type the name of the text or sentence to view it.
Type: 'texts()' or 'sents()' to list the materials.
text1: Moby Dick by Herman Melville 1851
text2: Sense and Sensibility by Jane Austen 1811
text3: The Book of Genesis
text4: Inaugural Address Corpus
text5: Chat Corpus
text6: Monty Python and the Holy Grail
text7: Wall Street Journal
text8: Personals Corpus
text9: The Man Who Was Thursday by G . K . Chesterton 1908

text1

<Text: Moby Dick by Herman Melville 1851>

text2

<Text: Sense and Sensibility by Jane Austen 1811>

搜索文本或主题

1. concordance允许在课文中查找单词，并打印出来
2. similar 用来识别文章中和搜索词相似的词语，可以用在搜索引擎中的相关度识别功能中。
3. common_contexts 用来识别2个关键词相似的词语。
4. dispersion_plot 绘制单词的离散图

text1.concordance('monstrous') # 在text1中查阅词汇'monstrous'
# concordance
# 英 [kən'kɔːd(ə)ns]  美 [kən'kɔrdns]
# n. 调和，一致；用语索引；著作或作家全集的重要用字索引

Displaying 11 of 11 matches:
ong the former , one was of a most monstrous size . ... This came towards us ,
ON OF THE PSALMS . " Touching that monstrous bulk of the whale or ork we have r
ll over with a heathenish array of monstrous clubs and spears . Some were thick
d as you gazed , and wondered what monstrous cannibal and savage could ever hav
that has survived the flood ; most monstrous and most mountainous ! That Himmal
they might scout at Moby Dick as a monstrous fable , or still worse and more de
th of Radney .'" CHAPTER 55 Of the Monstrous Pictures of Whales . I shall ere l
ing Scenes . In connexion with the monstrous pictures of whales , I am strongly
ere to enter upon those still more monstrous stories of them which are to be fo
ght have been rummaged out of this monstrous cabinet there is no telling . But
of Whale - Bones ; for Whales of a monstrous size are oftentimes cast up dead u

text2.concordance('affection')

Displaying 25 of 79 matches:
, however , and , as a mark of his affection for the three girls , he left them
t . It was very well known that no affection was ever supposed to exist between
deration of politeness or maternal affection on the side of the former , the tw
d the suspicion -- the hope of his affection for me may warrant , without impru
hich forbade the indulgence of his affection . She knew that his mother neither
rd she gave one with still greater affection . Though her late conversation wit
 can never hope to feel or inspire affection again , and if her home be uncomfo
m of the sense , elegance , mutual affection , and domestic comfort of the fami
, and which recommended him to her affection beyond every thing else . His soci
ween the parties might forward the affection of Mr . Willoughby , an equally st
 the most pointed assurance of her affection . Elinor could not be surprised at
he natural consequence of a strong affection in a young and ardent mind . This
 opinion . But by an appeal to her affection for her mother , by representing t
 every alteration of a place which affection had established as perfect with hi
e will always have one claim of my affection , which no other can possibly shar
f the evening declared at once his affection and happiness . " Shall we see you
ause he took leave of us with less affection than his usual behaviour has shewn
ness ." " I want no proof of their affection ," said Elinor ; " but of their en
onths , without telling her of his affection ;-- that they should part without
ould be the natural result of your affection for her . She used to be all unres
distinguished Elinor by no mark of affection . Marianne saw and listened with i
th no inclination for expense , no affection for strangers , no profession , an
till distinguished her by the same affection which once she had felt no doubt o
al of her confidence in Edward ' s affection , to the remembrance of every mark
 was made ? Had he never owned his affection to yourself ?" " Oh , no ; but if

text1.similar('monstrous')

true contemptible christian abundant few part mean careful puzzled
mystifying passing curious loving wise doleful gamesome singular
delightfully perilous fearless

text2.similar('monstrous')

very so exceedingly heartily a as good great extremely remarkably
sweet vast amazingly

text2.common_contexts(['monstrous','very'])

a_pretty am_glad a_lucky is_pretty be_glad

# 从文本中检查一个单词的位置，从该单词出现开始出现了多少次。
# Each stripe represents an instance of a word,
# and each row represents the entire text.
text4.dispersion_plot(['citizens','democracy','freedon','duties','America','liberty'])
# dispersion
# 英 [dɪ'spɜːʃ(ə)n]  美 [dɪ'spɝʒn]
# n. 散布；[统计][数] 离差；驱散

print(text3.generate('monstrous'))

None

统计词汇

len(text3)

44764

sorted(set(text3))

['!',
 "'",
 '(',
 ')',
 ',',
 ',)',
 '.',
 '.)',
 ':',
 ';',
 ';)',
 '?',
 '?)',
 'A',
 'Abel',
 'Abelmizraim',
 'Abidah',
 'Abide',
 'Abimael',
 'Abimelech',
 'Abr',
 'Abrah',
 'Abraham',
 'Abram',
 'Accad',
 'Achbor',
 'Adah',
 'Adam',
 'Adbeel',
 'Admah',
 'Adullamite',
 'After',
 'Aholibamah',
 'Ahuzzath',
 'Ajah',
 'Akan',
 'All',
 'Allonbachuth',
 'Almighty',
 'Almodad',
 'Also',
 'Alvah',
 'Alvan',
 'Am',
 'Amal',
 'Amalek',
 'Amalekites',
 'Ammon',
 'Amorite',
 'Amorites',
 'Amraphel',
 'An',
 'Anah',
 'Anamim',
 'And',
 'Aner',
 'Angel',
 'Appoint',
 'Aram',
 'Aran',
 'Ararat',
 'Arbah',
 'Ard',
 'Are',
 'Areli',
 'Arioch',
 'Arise',
 'Arkite',
 'Arodi',
 'Arphaxad',
 'Art',
 'Arvadite',
 'As',
 'Asenath',
 'Ashbel',
 'Asher',
 'Ashkenaz',
 'Ashteroth',
 'Ask',
 'Asshur',
 'Asshurim',
 'Assyr',
 'Assyria',
 'At',
 'Atad',
 'Avith',
 'Baalhanan',
 'Babel',
 'Bashemath',
 'Be',
 'Because',
 'Becher',
 'Bedad',
 'Beeri',
 'Beerlahairoi',
 'Beersheba',
 'Behold',
 'Bela',
 'Belah',
 'Benam',
 'Benjamin',
 'Beno',
 'Beor',
 'Bera',
 'Bered',
 'Beriah',
 'Bethel',
 'Bethlehem',
 'Bethuel',
 'Beware',
 'Bilhah',
 'Bilhan',
 'Binding',
 'Birsha',
 'Bless',
 'Blessed',
 'Both',
 'Bow',
 'Bozrah',
 'Bring',
 'But',
 'Buz',
 'By',
 'Cain',
 'Cainan',
 'Calah',
 'Calneh',
 'Can',
 'Cana',
 'Canaan',
 'Canaanite',
 'Canaanites',
 'Canaanitish',
 'Caphtorim',
 'Carmi',
 'Casluhim',
 'Cast',
 'Cause',
 'Chaldees',
 'Chedorlaomer',
 'Cheran',
 'Cherubims',
 'Chesed',
 'Chezib',
 'Come',
 'Cursed',
 'Cush',
 'Damascus',
 'Dan',
 'Day',
 'Deborah',
 'Dedan',
 'Deliver',
 'Diklah',
 'Din',
 'Dinah',
 'Dinhabah',
 'Discern',
 'Dishan',
 'Dishon',
 'Do',
 'Dodanim',
 'Dothan',
 'Drink',
 'Duke',
 'Dumah',
 'Earth',
 'Ebal',
 'Eber',
 'Edar',
 'Eden',
 'Edom',
 'Edomites',
 'Egy',
 'Egypt',
 'Egyptia',
 'Egyptian',
 'Egyptians',
 'Ehi',
 'Elah',
 'Elam',
 'Elbethel',
 'Eldaah',
 'EleloheIsrael',
 'Eliezer',
 'Eliphaz',
 'Elishah',
 'Ellasar',
 'Elon',
 'Elparan',
 'Emins',
 'En',
 'Enmishpat',
 'Eno',
 'Enoch',
 'Enos',
 'Ephah',
 'Epher',
 'Ephra',
 'Ephraim',
 'Ephrath',
 'Ephron',
 'Er',
 'Erech',
 'Eri',
 'Es',
 'Esau',
 'Escape',
 'Esek',
 'Eshban',
 'Eshcol',
 'Ethiopia',
 'Euphrat',
 'Euphrates',
 'Eve',
 'Even',
 'Every',
 'Except',
 'Ezbon',
 'Ezer',
 'Fear',
 'Feed',
 'Fifteen',
 'Fill',
 'For',
 'Forasmuch',
 'Forgive',
 'From',
 'Fulfil',
 'G',
 'Gad',
 'Gaham',
 'Galeed',
 'Gatam',
 'Gather',
 'Gaza',
 'Gentiles',
 'Gera',
 'Gerar',
 'Gershon',
 'Get',
 'Gether',
 'Gihon',
 'Gilead',
 'Girgashites',
 'Girgasite',
 'Give',
 'Go',
 'God',
 'Gomer',
 'Gomorrah',
 'Goshen',
 'Guni',
 'Hadad',
 'Hadar',
 'Hadoram',
 'Hagar',
 'Haggi',
 'Hai',
 'Ham',
 'Hamathite',
 'Hamor',
 'Hamul',
 'Hanoch',
 'Happy',
 'Haran',
 'Hast',
 'Haste',
 'Have',
 'Havilah',
 'Hazarmaveth',
 'Hazezontamar',
 'Hazo',
 'He',
 'Hear',
 'Heaven',
 'Heber',
 'Hebrew',
 'Hebrews',
 'Hebron',
 'Hemam',
 'Hemdan',
 'Here',
 'Hereby',
 'Heth',
 'Hezron',
 'Hiddekel',
 'Hinder',
 'Hirah',
 'His',
 'Hitti',
 'Hittite',
 'Hittites',
 'Hivite',
 'Hobah',
 'Hori',
 'Horite',
 'Horites',
 'How',
 'Hul',
 'Huppim',
 'Husham',
 'Hushim',
 'Huz',
 'I',
 'If',
 'In',
 'Irad',
 'Iram',
 'Is',
 'Isa',
 'Isaac',
 'Iscah',
 'Ishbak',
 'Ishmael',
 'Ishmeelites',
 'Ishuah',
 'Isra',
 'Israel',
 'Issachar',
 'Isui',
 'It',
 'Ithran',
 'Jaalam',
 'Jabal',
 'Jabbok',
 'Jac',
 'Jachin',
 'Jacob',
 'Jahleel',
 'Jahzeel',
 'Jamin',
 'Japhe',
 'Japheth',
 'Jared',
 'Javan',
 'Jebusite',
 'Jebusites',
 'Jegarsahadutha',
 'Jehovahjireh',
 'Jemuel',
 'Jerah',
 'Jetheth',
 'Jetur',
 'Jeush',
 'Jezer',
 'Jidlaph',
 'Jimnah',
 'Job',
 'Jobab',
 'Jokshan',
 'Joktan',
 'Jordan',
 'Joseph',
 'Jubal',
 'Judah',
 'Judge',
 'Judith',
 'Kadesh',
 'Kadmonites',
 'Karnaim',
 'Kedar',
 'Kedemah',
 'Kemuel',
 'Kenaz',
 'Kenites',
 'Kenizzites',
 'Keturah',
 'Kiriathaim',
 'Kirjatharba',
 'Kittim',
 'Know',
 'Kohath',
 'Kor',
 'Korah',
 'LO',
 'LORD',
 'Laban',
 'Lahairoi',
 'Lamech',
 'Lasha',
 'Lay',
 'Leah',
 'Lehabim',
 'Lest',
 'Let',
 'Letushim',
 'Leummim',
 'Levi',
 'Lie',
 'Lift',
 'Lo',
 'Look',
 'Lot',
 'Lotan',
 'Lud',
 'Ludim',
 'Luz',
 'Maachah',
 'Machir',
 'Machpelah',
 'Madai',
 'Magdiel',
 'Magog',
 'Mahalaleel',
 'Mahalath',
 'Mahanaim',
 'Make',
 'Malchiel',
 'Male',
 'Mam',
 'Mamre',
 'Man',
 'Manahath',
 'Manass',
 'Manasseh',
 'Mash',
 'Masrekah',
 'Massa',
 'Matred',
 'Me',
 'Medan',
 'Mehetabel',
 'Mehujael',
 'Melchizedek',
 'Merari',
 'Mesha',
 'Meshech',
 'Mesopotamia',
 'Methusa',
 'Methusael',
 'Methuselah',
 'Mezahab',
 'Mibsam',
 'Mibzar',
 'Midian',
 'Midianites',
 'Milcah',
 'Mishma',
 'Mizpah',
 'Mizraim',
 'Mizz',
 'Moab',
 'Moabites',
 'Moreh',
 'Moreover',
 'Moriah',
 'Muppim',
 'My',
 'Naamah',
 'Naaman',
 'Nahath',
 'Nahor',
 'Naphish',
 'Naphtali',
 'Naphtuhim',
 'Nay',
 'Nebajoth',
 'Neither',
 'Night',
 'Nimrod',
 'Nineveh',
 'Noah',
 'Nod',
 'Not',
 'Now',
 'O',
 'Obal',
 'Of',
 'Oh',
 'Ohad',
 'Omar',
 'On',
 'Onam',
 'Onan',
 'Only',
 'Ophir',
 'Our',
 'Out',
 'Padan',
 'Padanaram',
 'Paran',
 'Pass',
 'Pathrusim',
 'Pau',
 'Peace',
 'Peleg',
 'Peniel',
 'Penuel',
 'Peradventure',
 'Perizzit',
 'Perizzite',
 'Perizzites',
 'Phallu',
 'Phara',
 'Pharaoh',
 'Pharez',
 'Phichol',
 'Philistim',
 'Philistines',
 'Phut',
 'Phuvah',
 'Pildash',
 'Pinon',
 'Pison',
 'Potiphar',
 'Potipherah',
 'Put',
 'Raamah',
 'Rachel',
 'Rameses',
 'Rebek',
 'Rebekah',
 'Rehoboth',
 'Remain',
 'Rephaims',
 'Resen',
 'Return',
 'Reu',
 'Reub',
 'Reuben',
 'Reuel',
 'Reumah',
 'Riphath',
 'Rosh',
 'Sabtah',
 'Sabtech',
 'Said',
 'Salah',
 'Salem',
 'Samlah',
 'Sarah',
 'Sarai',
 'Saul',
 'Save',
 'Say',
 'Se',
 'Seba',
 'See',
 'Seeing',
 'Seir',
 'Sell',
 'Send',
 'Sephar',
 'Serah',
 'Sered',
 'Serug',
 'Set',
 'Seth',
 'Shalem',
 'Shall',
 'Shalt',
 'Shammah',
 'Shaul',
 'Shaveh',
 'She',
 'Sheba',
 'Shebah',
 'Shechem',
 'Shed',
 'Shel',
 'Shelah',
 'Sheleph',
 'Shem',
 'Shemeber',
 'Shepho',
 'Shillem',
 'Shiloh',
 'Shimron',
 'Shinab',
 'Shinar',
 'Shobal',
 'Should',
 'Shuah',
 'Shuni',
 'Shur',
 'Sichem',
 'Siddim',
 'Sidon',
 'Simeon',
 'Sinite',
 'Sitnah',
 'Slay',
 'So',
 'Sod',
 'Sodom',
 'Sojourn',
 'Some',
 'Spake',
 'Speak',
 'Spirit',
 'Stand',
 'Succoth',
 'Surely',
 'Swear',
 'Syrian',
 'Take',
 'Tamar',
 'Tarshish',
 'Tebah',
 'Tell',
 'Tema',
 'Teman',
 'Temani',
 'Terah',
 'Thahash',
 'That',
 'The',
 'Then',
 'There',
 'Therefore',
 'These',
 'They',
 'Thirty',
 'This',
 'Thorns',
 'Thou',
 'Thus',
 'Thy',
 'Tidal',
 'Timna',
 'Timnah',
 'Timnath',
 'Tiras',
 'To',
 'Togarmah',
 'Tola',
 'Tubal',
 'Tubalcain',
 'Twelve',
 'Two',
 'Unstable',
 'Until',
 'Unto',
 'Up',
 'Upon',
 'Ur',
 'Uz',
 'Uzal',
 'We',
 'What',
 'When',
 'Whence',
 'Where',
 'Whereas',
 'Wherefore',
 'Which',
 'While',
 'Who',
 'Whose',
 'Whoso',
 'Why',
 'Wilt',
 'With',
 'Woman',
 'Ye',
 'Yea',
 'Yet',
 'Zaavan',
 'Zaphnathpaaneah',
 'Zar',
 'Zarah',
 'Zeboiim',
 'Zeboim',
 'Zebul',
 'Zebulun',
 'Zemarite',
 'Zepho',
 'Zerah',
 'Zibeon',
 'Zidon',
 'Zillah',
 'Zilpah',
 'Zimran',
 'Ziphion',
 'Zo',
 'Zoar',
 'Zohar',
 'Zuzims',
 'a',
 'abated',
 'abide',
 'able',
 'abode',
 'abomination',
 'about',
 'above',
 'abroad',
 'absent',
 'abundantly',
 'accept',
 'accepted',
 'according',
 'acknowledged',
 'activity',
 'add',
 'adder',
 'afar',
 'afflict',
 'affliction',
 'afraid',
 'after',
 'afterward',
 'afterwards',
 'aga',
 'again',
 'against',
 'age',
 'aileth',
 'air',
 'al',
 'alive',
 'all',
 'almon',
 'alo',
 'alone',
 'aloud',
 'also',
 'altar',
 'altogether',
 'always',
 'am',
 'among',
 'amongst',
 'an',
 'and',
 'angel',
 'angels',
 'anger',
 'angry',
 'anguish',
 'anointedst',
 'anoth',
 'another',
 'answer',
 'answered',
 'any',
 'anything',
 'appe',
 'appear',
 'appeared',
 'appease',
 'appoint',
 'appointed',
 'aprons',
 'archer',
 'archers',
 'are',
 'arise',
 'ark',
 'armed',
 'arms',
 'army',
 'arose',
 'arrayed',
 'art',
 'artificer',
 'as',
 'ascending',
 'ash',
 'ashamed',
 'ask',
 'asked',
 'asketh',
 'ass',
 'assembly',
 'asses',
 'assigned',
 'asswaged',
 'at',
 'attained',
 'audience',
 'avenged',
 'aw',
 'awaked',
 'away',
 'awoke',
 'back',
 'backward',
 'bad',
 'bade',
 'badest',
 'badne',
 'bak',
 'bake',
 'bakemeats',
 'baker',
 'bakers',
 'balm',
 'bands',
 'bank',
 'bare',
 'barr',
 'barren',
 'basket',
 'baskets',
 'battle',
 'bdellium',
 'be',
 'bear',
 'beari',
 'bearing',
 'beast',
 'beasts',
 'beautiful',
 'became',
 'because',
 'become',
 'bed',
 'been',
 'befall',
 'befell',
 'before',
 'began',
 'begat',
 'beget',
 'begettest',
 'begin',
 'beginning',
 'begotten',
 'beguiled',
 'beheld',
 'behind',
 'behold',
 'being',
 'believed',
 'belly',
 'belong',
 'beneath',
 'bereaved',
 'beside',
 'besides',
 'besought',
 'best',
 'betimes',
 'better',
 'between',
 'betwixt',
 'beyond',
 'binding',
 'bird',
 'birds',
 'birthday',
 'birthright',
 'biteth',
 'bitter',
 'blame',
 'blameless',
 'blasted',
 'bless',
 'blessed',
 'blesseth',
 'blessi',
 'blessing',
 'blessings',
 'blindness',
 'blood',
 'blossoms',
 'bodies',
 'boldly',
 'bondman',
 'bondmen',
 'bondwoman',
 'bone',
 'bones',
 'book',
 'booths',
 'border',
 'borders',
 'born',
 'bosom',
 'both',
 'bottle',
 'bou',
 'boug',
 'bough',
 'bought',
 'bound',
 'bow',
 'bowed',
 'bowels',
 'bowing',
 'boys',
 'bracelets',
 'branches',
 'brass',
 'bre',
 'breach',
 'bread',
 'breadth',
 'break',
 'breaketh',
 'breaking',
 'breasts',
 'breath',
 'breathed',
 'breed',
 'brethren',
 'brick',
 'brimstone',
 'bring',
 'brink',
 'broken',
 '*',
 'broth',
 'brother',
 'brought',
 'brown',
 'bruise',
 'budded',
 'build',
 'builded',
 'built',
 'bulls',
 'bundle',
 'bundles',
 'burdens',
 'buried',
 'burn',
 'burning',
 'burnt',
 'bury',
 'buryingplace',
 'business',
 'but',
 'butler',
 'butlers',
 'butlership',
 'butter',
 'buy',
 'by',
 'cakes',
 'calf',
 'call',
 'called',
 'came',
 'camel',
 'camels',
 'camest',
 'can',
 'cannot',
 'canst',
 'captain',
 'captive',
 'captives',
 'carcases',
 'carried',
 'carry',
 'cast',
 'castles',
 'catt',
 'cattle',
 'caught',
 'cause',
 'caused',
 'cave',
 'cease',
 'ceased',
 'certain',
 'certainly',
 'chain',
 'chamber',
 'change',
 'changed',
 'changes',
 'charge',
 'charged',
 'chariot',
 'chariots',
 'chesnut',
 'chi',
 'chief',
 'child',
 'childless',
 'childr',
 'children',
 'chode',
 'choice',
 'chose',
 'circumcis',
 'circumcise',
 'circumcised',
 'citi',
 'cities',
 'city',
 'clave',
 'clean',
 'clear',
 'cleave',
 'clo',
 'closed',
 'clothed',
 'clothes',
 'cloud',
 'clusters',
 'co',
 'coat',
 'coats',
 'coffin',
 'cold',
 ...]

len(set(text3))

2789

len(text3)/len(set(text3))

16.050197203298673

text3.count('smote')

5

100*text4.count('a')/len(text4)

1.4643016433938312

def lexical_diversity(text):
    # lexical英['leksɪk(ə)l] 美 ['lɛksɪkl]
    # adj.词汇的；[语] 词典的；词典编纂的
    # diversity英[daɪ'vɜːsɪtɪ; dɪ-]美 [dɪˈvəsɪti]
    # n.多样性；差异
    return len(text)/len(set(text))
def percentage(count, total):
    return 100*count/total

print('text3中词汇多样性指标：{}'.format(lexical_diversity(text3)))
print('text4中单词a占全文的百分比：{}'.format(percentage(text4.count('a'),len(text4))))

text3中词汇多样性指标：16.050197203298673
text4中单词a占全文的百分比：1.4643016433938312

列表 = Lists

sent1 = ['Call', 'me','Ishmael','.']
print('打印sent1中的内容：{}'.format(sent1))
print('打印sent1中内容的长度：{}'.format(len(sent1)))
print('sent1中词汇多样性指标：{}'.format(lexical_diversity(sent1)))

打印sent1中的内容：['Call', 'me', 'Ishmael', '.']
打印sent1中内容的长度：4
sent1中词汇多样性指标：1.0

sent1,sent2,sent3,sent4 # 这是内部定义好的列表

(['Call', 'me', 'Ishmael', '.'],
 ['The',
  'family',
  'of',
  'Dashwood',
  'had',
  'long',
  'been',
  'settled',
  'in',
  'Sussex',
  '.'],
 ['In',
  'the',
  'beginning',
  'God',
  'created',
  'the',
  'heaven',
  'and',
  'the',
  'earth',
  '.'],
 ['Fellow',
  '-',
  'Citizens',
  'of',
  'the',
  'Senate',
  'and',
  'of',
  'the',
  'House',
  'of',
  'Representatives',
  ':'])

sent4+sent1

['Fellow',
 '-',
 'Citizens',
 'of',
 'the',
 'Senate',
 'and',
 'of',
 'the',
 'House',
 'of',
 'Representatives',
 ':',
 'Call',
 'me',
 'Ishmael',
 '.']

sent1.append('Some')

['Call', 'me', 'Ishmael', '.', 'Some', 'Some', 'Some', 'Some']

列表索引

type(text4)

nltk.text.Text

text4[173]

'awaken'

text4.index('awaken')

173

text5[16715:16735]

['U86',
 'thats',
 'why',
 'something',
 'like',
 'gamefly',
 'is',
 'so',
 'good',
 'because',
 'you',
 'can',
 'actually',
 'play',
 'a',
 'full',
 'game',
 'without',
 'buying',
 'it']

text6[1600:1625]

['We',
 "'",
 're',
 'an',
 'anarcho',
 '-',
 'syndicalist',
 'commune',
 '.',
 'We',
 'take',
 'it',
 'in',
 'turns',
 'to',
 'act',
 'as',
 'a',
 'sort',
 'of',
 'executive',
 'officer',
 'for',
 'the',
 'week']

变量

sent1 = ['Call','me','Ishmael','.']
my_sent = ['Bravely','bold','Sir','Robin',',','rode','forth','from','Camelot','.']
noun_phrase = my_sent[1:4]
print('打印切片后的列表：noun_phrase-》{}'.format(noun_phrase))
wOrDs = sorted(noun_phrase)
print('打印排序后的列表：wOrDs-》{}'.format(wOrDs))

打印切片后的列表：noun_phrase-》['bold', 'Sir', 'Robin']
打印排序后的列表：wOrDs-》['Robin', 'Sir', 'bold']

字符串

name = 'bright'
print('打印name中的第一个字母：{}'.format(name[0]))
print(name[:4])
print(name*2)
print(name + '!')

打印name中的第一个字母：b
brig
brightbright
bright!

' '.join(['Monty', 'Python'])

'Monty Python'

'Monty Python'.split()

['Monty', 'Python']

saying = ['After','all','is','said','and','done','more','is','said','than','done']
tokens = set(saying)
tokens = sorted(tokens)
tokens[-2:]

['said', 'than']

fdist1 = FreqDist(text1)
vocabulary1 = fdist1.keys()
type(vocabulary1)

dict_keys

fdist1.plot(50, cumulative=True)
#Cumulative frequency plot for the 50 most frequently used words in Moby Dick, which
#account for nearly half of the tokens.

fdist1.hapaxes() #the words that occur once only

['Herman',
 'Melville',
 ']',
 'ETYMOLOGY',
 'Late',
 'Consumptive',
 'School',
 'threadbare',
 'lexicons',
 'mockingly',
 'flags',
 'mortality',
 'signification',
 'HACKLUYT',
 'Sw',
 'HVAL',
 'roundness',
 'Dut',
 'Ger',
 'WALLEN',
 'WALW',
 'IAN',
 'RICHARDSON',
 'KETOS',
 'GREEK',
 'CETUS',
 'LATIN',
 'WHOEL',
 'ANGLO',
 'SAXON',
 'WAL',
 'HWAL',
 'SWEDISH',
 'ICELANDIC',
 'BALEINE',
 'BALLENA',
 'FEGEE',
 'ERROMANGOAN',
 'Librarian',
 'painstaking',
 'burrower',
 'grub',
 'Vaticans',
 'stalls',
 'higgledy',
 'piggledy',
 'gospel',
 'promiscuously',
 'commentator',
 'belongest',
 'sallow',
 'Pale',
 'Sherry',
 'loves',
 'bluntly',
 'Subs',
 'thankless',
 'Hampton',
 'Court',
 'hie',
 'refugees',
 'pampered',
 'Michael',
 'Raphael',
 'unsplinterable',
 'GENESIS',
 'JOB',
 'JONAH',
 'punish',
 'ISAIAH',
 'soever',
 'cometh',
 'incontinently',
 'perisheth',
 'PLUTARCH',
 'MORALS',
 'breedeth',
 'Whirlpooles',
 'Balaene',
 'arpens',
 'PLINY',
 'Scarcely',
 'TOOKE',
 'LUCIAN',
 'TRUE',
 'catched',
 'OCTHER',
 'VERBAL',
 'TAKEN',
 'MOUTH',
 'ALFRED',
 '890',
 'gudgeon',
 'retires',
 'MONTAIGNE',
 'APOLOGY',
 'RAIMOND',
 'SEBOND',
 'Nick',
 'RABELAIS',
 'cartloads',
 'STOWE',
 'ANNALS',
 'LORD',
 'BACON',
 'Touching',
 'ork',
 'DEATH',
 'sovereignest',
 'bruise',
 'HAMLET',
 'leach',
 'Mote',
 'availle',
 'returne',
 'againe',
 'worker',
 'Dinting',
 'paine',
 'thro',
 'maine',
 'FAERIE',
 'Immense',
 'til',
 'DAVENANT',
 'PREFACE',
 'GONDIBERT',
 'spermacetti',
 'Hosmannus',
 'Nescio',
 'VIDE',
 'Spencer',
 'Talus',
 'flail',
 'threatens',
 'jav',
 'lins',
 'WALLER',
 'SUMMER',
 'ISLANDS',
 'Commonwealth',
 'Civitas',
 'OPENING',
 'SENTENCE',
 'HOBBES',
 'LEVIATHAN',
 'Silly',
 'Mansoul',
 'chewing',
 'sprat',
 'PILGRIM',
 'PROGRESS',
 'Created',
 'PARADISE',
 'LOST',
 '---"',
 'Hugest',
 'Stretched',
 'Draws',
 'FULLLER',
 'PROFANE',
 'HOLY',
 'STATE',
 'DRYDEN',
 'ANNUS',
 'MIRABILIS',
 'aground',
 'EDGE',
 'TEN',
 'SPITZBERGEN',
 'PURCHAS',
 'wantonness',
 'fuzzing',
 'vents',
 'HERBERT',
 'INTO',
 'ASIA',
 'AFRICA',
 'SCHOUTEN',
 'SIXTH',
 'CIRCUMNAVIGATION',
 'Elbe',
 'ducat',
 'herrings',
 'GREENLAND',
 'Several',
 'Fife',
 'Anno',
 '1652',
 'Pitferren',
 'SIBBALD',
 'FIFE',
 'KINROSS',
 'Myself',
 'Sperma',
 'ceti',
 'fierceness',
 'RICHARD',
 'STRAFFORD',
 'LETTER',
 'BERMUDAS',
 'PHIL',
 'TRANS',
 '1668',
 'PRIMER',
 'COWLEY',
 '1729',
 '"...',
 'frequendy',
 'insupportable',
 'disorder',
 'ULLOA',
 'SOUTH',
 'AMERICA',
 'sylphs',
 'petticoat',
 'Oft',
 'Tho',
 'RAPE',
 'LOCK',
 'NAT',
 'wales',
 'JOHNSON',
 'COOK',
 'dung',
 'lime',
 'juniper',
 'UNO',
 'VON',
 'TROIL',
 'LETTERS',
 'BANKS',
 'SOLANDER',
 '1772',
 'Nantuckois',
 'JEFFERSON',
 'MEMORIAL',
 'MINISTER',
 'REFERENCE',
 'PARLIAMENT',
 'SOMEWHERE',
 'guarding',
 'protecting',
 'robbers',
 'BLACKSTONE',
 'Rodmond',
 'suspends',
 'attends',
 'FALCONER',
 'Bright',
 'roofs',
 'domes',
 'rockets',
 'Around',
 'unwieldy',
 'COWPER',
 'VISIT',
 'LONDON',
 'HUNTER',
 'DISSECTION',
 'SMALL',
 'SIZED',
 'aorta',
 'gushing',
 'PALEY',
 'THEOLOGY',
 'mammiferous',
 'hind',
 'BARON',
 'CUVIER',
 'COLNETT',
 'PURPOSE',
 'EXTENDING',
 'SPERMACETI',
 'Floundered',
 'chace',
 'peopling',
 'Gather',
 'Led',
 'instincts',
 'trackless',
 'Assaulted',
 'voracious',
 'spiral',
 'MONTGOMERY',
 'WORLD',
 'FLOOD',
 'Paean',
 'fatter',
 'Flounders',
 'CHARLES',
 'LAMB',
 'TRIUMPH',
 '1690',
 'OBED',
 'Susan',
 'HAWTHORNE',
 'TWICE',
 'bespeak',
 'raal',
 'COOPER',
 'PILOT',
 'Berlin',
 'Gazette',
 'ECKERMANN',
 'CONVERSATIONS',
 'GOETHE',
 'ESSEX',
 'WAS',
 'ATTACKED',
 'FINALLY',
 'DESTROYED',
 'OWEN',
 'CHACE',
 'FIRST',
 'SAID',
 'VESSEL',
 'YORK',
 '1821',
 'piping',
 'dimmed',
 'phospher',
 'ELIZABETH',
 'OAKES',
 'SMITH',
 'amounted',
 '440',
 'SCORESBY',
 'Mad',
 'agonies',
 'endures',
 'infuriated',
 'rears',
 'snaps',
 'propelled',
 'observers',
 'opportunities',
 'habitudes',
 'BEALE',
 'offensively',
 'artful',
 'mischievous',
 'FREDERICK',
 'DEBELL',
 '1840',
 'October',
 'Raise',
 'ay',
 'THAR',
 'bowes',
 'os',
 'ROSS',
 'ETCHINGS',
 'CRUIZE',
 '1846',
 'Globe',
 'transactions',
 'relate',
 'HUSSEY',
 'SURVIVORS',
 'parried',
 'MISSIONARY',
 'JOURNAL',
 'TYERMAN',
 'boldest',
 'persevering',
 'REPORT',
 'DANIEL',
 'SPEECH',
 'SENATE',
 'APPLICATION',
 'ERECTION',
 'BREAKWATER',
 'CAPTORS',
 'WHALEMAN',
 'ADVENTURES',
 'BIOGRAPHY',
 'GATHERED',
 'HOMEWARD',
 'COMMODORE',
 'PREBLE',
 'REV',
 'CHEEVER',
 'MUTINEER',
 'BROTHER',
 'ANOTHER',
 'MCCULLOCH',
 'COMMERCIAL',
 'reciprocal',
 'clews',
 'SOMETHING',
 'UNPUBLISHED',
 'CURRENTS',
 'Pedestrians',
 'recollect',
 'gateways',
 'VOYAGER',
 'ARCTIC',
 'NEWSPAPER',
 'TAKING',
 'RETAKING',
 'HOBOMACK',
 'MIRIAM',
 'FISHERMAN',
 'appliance',
 'RIBS',
 'TRUCKS',
 'Terra',
 'Del',
 'Fuego',
 'DARWIN',
 'NATURALIST',
 ";--'",
 '!\'"',
 'WHARTON',
 'Loomings',
 'spleen',
 'regulating',
 'circulation',
 'Whenever',
 'drizzly',
 'hypos',
 'philosophical',
 'Cato',
 'Manhattoes',
 'reefs',
 'downtown',
 'gazers',
 'Circumambulate',
 'Corlears',
 'Coenties',
 'Slip',
 'Whitehall',
 'Posted',
 'sentinels',
 'spiles',
 'pier',
 'lath',
 'counters',
 'desks',
 'loitering',
 'shady',
 'Inlanders',
 'lanes',
 'alleys',
 'attract',
 'dale',
 'dreamiest',
 'shadiest',
 'quietest',
 'enchanting',
 'Saco',
 'crucifix',
 'Deep',
 'mazy',
 'Tiger',
 'Tennessee',
 'Rockaway',
 'Persians',
 'deity',
 'Narcissus',
 'ungraspable',
 'hazy',
 'quarrelsome',
 'offices',
 'abominate',
 'toils',
 'trials',
 'barques',
 'schooners',
 'broiling',
 'buttered',
 'judgmatically',
 'peppered',
 'reverentially',
 'idolatrous',
 'dotings',
 'ibis',
 'roasted',
 'bake',
 'plumb',
 'Van',
 'Rensselaers',
 'Randolphs',
 'Hardicanutes',
 'lording',
 'tallest',
 'decoction',
 'Seneca',
 'Stoics',
 'Testament',
 'promptly',
 'rub',
 'infliction',
 'BEING',
 'PAID',
 'urbane',
 'ills',
 'monied',
 'consign',
 'prevalent',
 'violate',
 'Pythagorean',
 'commonalty',
 'police',
 'surveillance',
 'programme',
 'solo',
 'CONTESTED',
 'ELECTION',
 'PRESIDENCY',
 'UNITED',
 'STATES',
 'ISHMAEL',
 'BLOODY',
 'AFFGHANISTAN',
 'managers',
 'genteel',
 'comedies',
 'farces',
 'cunningly',
 'disguises',
 'cajoling',
 'unbiased',
 'freewill',
 'discriminating',
 'overwhelming',
 'undeliverable',
 'itch',
 'forbidden',
 'ignoring',
 'lodges',
 'Carpet',
 'Bag',
 'Manhatto',
 'candidates',
 'penalties',
 'Tyre',
 'Carthage',
 'imported',
 'cobblestones',
 'bitingly',
 'shouldering',
 'price',
 'fervent',
 'asphaltic',
 'pavement',
 'flinty',
 'projections',
 'soles',
 'Too',
 'cheapest',
 'cheeriest',
 'invitingly',
 'particles',
 'peer',
 'Angel',
 'Doom',
 'wailing',
 'gnashing',
 'Wretched',
 'entertainment',
 'Moving',
 'emigrant',
 'poverty',
 'creak',
 'lodgings',
 'zephyr',
 'hob',
 'toasting',
 'observest',
 'sashless',
 'glazier',
 'reasonest',
 'chinks',
 'crannies',
 'lint',
 'chattering',
 'shiverings',
 'cob',
 'redder',
 'Orion',
 'glitters',
 'conservatories',
 'president',
 'temperance',
 'blubbering',
 'straggling',
 'wainscots',
 'reminding',
 'oilpainting',
 'besmoked',
 'defaced',
 'unequal',
 'crosslights',
 'hags',
 'delineate',
 'bewitched',
 'ponderings',
 'boggy',
 'soggy',
 'squitchy',
 'froze',
 'heath',
 'icebound',
 'represents',
 'Horner',
 'foundered',
 'clubs',
 'harvesting',
 'hacking',
 'horrifying',
 'Mixed',
 'Nathan',
 'Swain',
 'corkscrew',
 'Blanco',
 'sojourning',
 'fireplaces',
 'duskier',
 'cockpits',
 'rarities',
 'Projecting',
 'Within',
 'shelves',
 'flasks',
 'bustles',
 'deliriums',
 'Abominable',
 'tumblers',
 'cylinders',
 'goggling',
 'deceitfully',
 'tapered',
 'Parallel',
 'pecked',
 'footpads',
 'Fill',
 'shilling',
 'examining',
 'SKRIMSHANDER',
 'accommodated',
 'unoccupied',
 'haint',
 'pose',
 'whalin',
 'decidedly',
 'objectionable',
 'wander',
 'Battery',
 'ruminating',
 'adorning',
 'potatoes',
 'sartainty',
 'diabolically',
 'steaks',
 'undress',
 'looker',
 'rioting',
 'Grampus',
 'seed',
 'Feegees',
 'tramping',
 'Enveloped',
 'bedarned',
 'eruption',
 'officiating',
 'brimmers',
 'complained',
 'potion',
 'colds',
 'catarrhs',
 'liquor',
 'arrantest',
 'topers',
 'obstreperously',
 'aloof',
 'desirous',
 'hilarity',
 'coffer',
 'Southerner',
 'mountaineers',
 'Alleghanian',
 'missed',
 'supernaturally',
 'congratulate',
 'multiply',
 'bachelor',
 'abominated',
 'tidiest',
 'bedwards',
 'shan',
 'tablecloth',
 'Skrimshander',
 'bump',
 'spraining',
 'eider',
 'yoking',
 'rickety',
 'whirlwinds',
 'knockings',
 'dismissed',
 'popped',
 'cherishing',
 'chuckled',
 'chuckle',
 'mightily',
 'catches',
 'bamboozingly',
 'overstocked',
 'toothpick',
 'rayther',
 'BROWN',
 'slanderin',
 'farrago',
 'BROKE',
 'Sartain',
 'Mt',
 'Hecla',
 'persist',
 'mystifying',
 'unsay',
 'criminal',
 'Wall',
 'purty',
 'sarmon',
 'rips',
 'tellin',
 'bought',
 'balmed',
 'curios',
 'sellin',
 'inions',
 'fooling',
 'idolators',
 'Depend',
 'reg',
 'lar',
 'spliced',
 'Johnny',
 'sprawling',
 'Arter',
 'glim',
 'jiffy',
 'irresolute',
 'vum',
 'WON',
 'Folding',
 'scrutiny',
 'porcupine',
 'moccasin',
 'ponchos',
 'parade',
 'rainy',
 'remembering',
 'commended',
 'cobs',
 'Nod',
 'footfall',
 'unlacing',
 'blackish',
 'plasters',
 'inkling',
 'Placing',
 'crammed',
 'scalp',
 'mildewed',
 'Ignorance',
 'parent',
 'nonplussed',
 'undressing',
 'checkered',
 'Thirty',
 'frogs',
 'quaked',
 'wrapall',
 'dreadnaught',
 'fumbled',
 'Remembering',
 'manikin',
 'tenpin',
 'andirons',
 'jambs',
 'bricks',
 'appropriate',
 'applying',
 'hastier',
 'withdrawals',
 'antics',
 'devotee',
 'extinguishing',
 'unceremoniously',
 'bagged',
 'sportsman',
 'woodcock',
 'uncomfortableness',
 'deliberating',
 'puffed',
 'sang',
 'Stammering',
 'conjured',
 'responses',
 'debel',
 'flourishing',
 'Angels',
 'flourishings',
 'peddlin',
 'sleepe',
 'grunted',
 'gettee',
 'motioning',
 'comely',
 'insured',
 'Counterpane',
 'parti',
 'triangles',
 'interminable',
 'caper',
 'supperless',
 '21st',
 'hemisphere',
 'sigh',
 'Sixteen',
 'ached',
 'coaches',
 'stockinged',
 'slippering',
 'misbehaviour',
 'unendurable',
 'stepmothers',
 'misfortunes',
 'steeped',
 'shudderingly',
 'confounding',
 'soberly',
 'recurred',
 'predicament',
 'unlock',
 'bridegroom',
 'clasp',
 'hugged',
 'rouse',
 'snore',
 'scratch',
 'Throwing',
 'expostulations',
 'unbecomingness',
 'matrimonial',
 'dawning',
 'overture',
 'innate',
 'compliment',
 'civility',
 'rudeness',
 'toilette',
 'dressing',
 'donning',
 'gaspings',
 'booting',
 'caterpillar',
 'outlandishness',
 'manners',
 'education',
 'undergraduate',
 'dreamt',
 'cowhide',
 'pinched',
 'curtains',
 'indecorous',
 'contented',
 'restricting',
 'donned',
 'lathering',
 'unsheathes',
 'whets',
 'Rogers',
 'cutlery',
 'Afterwards',
 'baton',
 'Breakfast',
 'pleasantly',
 'bountifully',
 'laughable',
 'bosky',
 'unshorn',
 'gowns',
 'toasted',
 'lingers',
 'tarried',
 'barred',
 'Grub',
 'Park',
 'assurance',
 'polish',
 'occasioned',
 'embarrassed',
 'bashfulness',
 'duelled',
 'winking',
 'tastes',
 'sheepishly',
 'bashful',
 'icicle',
 'admirer',
 'cordially',
 'grappling',
 'genteelly',
 'eschewed',
 'undivided',
 '6',
 'circulating',
 'nondescripts',
 'Chestnut',
 'jostle',
 'Regent',
 'Lascars',
 'Bombay',
 'Apollo',
 'Feegeeans',
 'Tongatobooarrs',
 'Erromanggoans',
 'Pannangians',
 'Brighggians',
 'weekly',
 'Vermonters',
 'stalwart',
 'frames',
 'felled',
 'strutting',
 'wester',
 'bombazine',
 'cloak',
 'mow',
 'gloves',
 'joins',
 'outfit',
 'waistcoats',
 'Hay',
 'Seed',
 'tract',
 'dearest',
 'pave',
 'eggs',
 'patrician',
 'parks',
 'scraggy',
 'scoria',
 'Herr',
 'dowers',
 'nieces',
 'reservoirs',
 'maples',
 'bountiful',
 'proffer',
 'passer',
 'cones',
 'blossoms',
 'superinduced',
 'carnation',
 'Salem',
 'sweethearts',
 'Puritanic',
 'Whaleman',
 'Wrapping',
 'Each',
 'quote',
 'TALBOT',
 'Near',
 'Desolation',
 '1st',
 'SISTER',
 'ROBERT',
 'WILLIS',
 'ELLERY',
 'NATHAN',
 'COLEMAN',
 'WALTER',
 'CANNY',
 'SETH',
 'GLEIG',
 'Forming',
 'ELIZA',
 '31st',
 'MARBLE',
 'SHIPMATES',
 'EZEKIEL',
 'HARDY',
 'AUGUST',
 '3d',
 '1833',
 'WIDOW',
 'Shaking',
 'glazed',
 'Affected',
 'relatives',
 'unhealing',
 'sympathetically',
 'wounds',
 'bleed',
 'blanks',
 ...]

单词的精细选择

1. the set of all w such that w is an element of V (the vocabulary) and w has property P
   
   {w|w \(\in\) V and P(w)}
2. The corresponding Python expression is given:
   
   [w for w in V if p(w)]

V = set(text1)
long_words = [w for w in V if len(w)>15]
sorted(long_words)

['CIRCUMNAVIGATION',
 'Physiognomically',
 'apprehensiveness',
 'cannibalistically',
 'characteristically',
 'circumnavigating',
 'circumnavigation',
 'circumnavigations',
 'comprehensiveness',
 'hermaphroditical',
 'indiscriminately',
 'indispensableness',
 'irresistibleness',
 'physiognomically',
 'preternaturalness',
 'responsibilities',
 'simultaneousness',
 'subterraneousness',
 'supernaturalness',
 'superstitiousness',
 'uncomfortableness',
 'uncompromisedness',
 'undiscriminating',
 'uninterpenetratingly']

本文选自《Natural Language Processing with Python》