毕业设计-基于深度神经网络的语音关键词检出系统-使用python脚本作词频统计-Librispeech

<span style="font-family: Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255);"></span><span style="font-family: Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255);">TIMIT之后，这次来分析Librispeech的词频，文件组织结构如图所示：</span>

librispeech文件夹下的dev-clean中含有多个多级子文件夹，每个末节文件夹下含有一个txt含有抄本外加多个音频文件是该抄本的朗读：

脚本任务是对所有txt抄本读取其中的词数并作统计，抄本内容样例如下

其中全部是大写单词，我们的操作分为两步：

1.利用os.walk()遍历所有文件，记下所有txt文件路径写入一个path.txt文本中

2.从path.txt读取文本路径打开，用python字典记录词与频率，写入keyword.txt作词频统计

代码如下：

import os
import os.path

rootDir = "dev-clean"



#functions used
"""This functions get in the file path and dictionary of keyword to count keywords"""
def keywordCounter(fileDir, keywordContainer):
    f = open(fileDir)
    row = f.readline()
    while row != '':
        wordPart = row[row.index(" ")+1:-1]
        words = wordPart.split(" ")
        for word in words:
            if keywordContainer.has_key(word):
                keywordContainer[word] += 1
            else:
                keywordContainer[word] = 1
        row = f.readline()


#1
print "Step 1: Get absolute directory of all the transcript file inside this folder"
"""I got the script from cnblog without fully understanding, just use it as black box"""
transPathDoc = open("dreaminghzAnalysedData\pathDoc.txt","w+")
for parent, dirnames, filenames in os.walk(rootDir):
    #for dirname in dirnames:
        #print dirname

    for filename in filenames:
        if filename[-4:] == ".txt":
            #print os.path.join(parent,filename)
            transPathDoc.write(os.path.join(parent,filename)+"\n")
transPathDoc.close() 

print "Step 1 finished"

#2
print "Step 2: Read in all the transcript file and do the word counting"
keywordContainer = {}
pathes = open("dreaminghzAnalysedData\pathDoc.txt")
pathTmp = pathes.readline()
#readin path and call the function keywordCounter to process it
while pathTmp != '':
    keywordCounter(pathTmp[:-1], keywordContainer)
    pathTmp = pathes.readline()
pathes.close()
print "Step 2 finished"
#claim
print "There's totally " + str(len(keywordContainer)) + " keywords"

#3
print "Step 3: Save keyword into dreaminghzAnalysedData\keyWords.txt"
outfile = open("dreaminghzAnalysedData\keyWords.txt","w+")
for ks in keywordContainer.keys():
    outfile.write(ks + " " + str(keywordContainer[ks]) + "\n")
outfile.close()

同时，再次读取keyword.txt文件可以给出词频在所要求词频及以上的单词，便于关键词选定与以后的实验：

"""This script is used for finding high frequency keywords that appears more than given time number"""
#use the file generated by LibriWordCounter.py
kw = open("keyWords.txt")
qualification = False
while not qualification:
    try:
        num = int(raw_input("Input the lower bound of frequency as positive integer pls:"))
        if num <= 0:
            qualification = False
        else:
            qualification = True
    except:
        qualification = False

highFFilename = "keywords-noless-" + str(num) + "-times.txt"
highFkw = open(highFFilename,"w+")

#read in and write down qualified keywords
row = kw.readline()
ctr = 0
while row != '':
    unit = row[:-1].split(' ')
    key, frequency = unit[0], int(unit[1])
    if frequency >= num:
        highFkw.write(row)
        ctr += 1
    row = kw.readline()

kw.close()
highFkw.close()
print("Finished, there's totally " + str(ctr) + " records written into the file")