数据平滑 自然语言处理

问题提出：为了解决训练语料中的零概率问题。平滑处理的基本思想是一种“劫富济贫”，即提高低（零）概率，降低高概率，尽量使得概率分布趋于均匀。

加一平滑方法

假设每个二元语法出现的次数比实际出现的次数多一次，不妨将该处理方法称为加1法。

p(wi|wI−1)=1+c(wi−1,wi)∑wi[1+c(wi−1,wi)]=1+c(wi−1,wi)|V|+∑wi[1+c(wi−1,wi)] $p(w_i|w_{I-1})=\frac{1+c(w_{i-1},w_i)}{\underset{w_i}{\sum }[1+c(w_{i-1},w_i)]}=\frac{1+c(w_{i-1},w_i)}{|V|+\underset{w_i}{\sum }[1+c(w_{i-1},w_i)]}$

加法平滑方法

p(wi|wI−1)=δ+c(wi−1,wi)δ|V|+∑wi[1+c(wi−1,wi)] $p(w_i|w_{I-1})=\frac{\delta +c(w_{i-1},w_i)}{\delta |V|+\underset{w_i}{\sum }[1+c(w_{i-1},w_i)]}$

其中 0≤δ≤1 $0\le \delta \le 1$

Good-Turing估计法

nr是训练语料中恰好出现r次的n元语法的数目，假设它出现了r∗次。 $n_r是训练语料中恰好出现r次的n元语法的数目，假设它出现了r^{\ast }次。$

r∗=(r+1)nr+1nr $r^{\ast }=(r+1)\frac{n_{r+1}}{n_r}$

对于统计数为r的n元语法，其概率为 pr=r∗∑r=1∞nrr $p_r=\frac{r^{\ast }}{\underset{r=1}{\overset{\mathrm{\infty }}{\sum }}{n}_{r}r}$
该方法不能直接用于估计 nr=0 $n_r=0$的n-gram概率。也不能实现高阶模型与低阶模型的结合。

Katz平滑方法

pkatz(wiI−1)=dr⋅c(wii−1)c(wi−1)，r>0 $p_{katz}(w_{I-1}^i)=d_r\cdot \frac{c(w_{i-1}^i)}{c(w_{i-1})}，r>0$
dr=r∗r−(k+1)nk+1n11−(k+1)nk+1n1 $d_r=\frac{\frac{r^{\ast }}{r}-\frac{(k+1)n_{k+1}}{n_1}}{1-\frac{(k+1)n_{k+1}}{n_1}}$
pkatz(wiI−1)=α(wi−1⋅pML(wi))，r=0 $p_{katz}(w_{I-1}^i)=\alpha (w_{i-1}\cdot p_{ML}(w_i))，r=0$
α(wi−1)=1−∑wj:c(wii−1)>0pkatz(wi|wi−1)1−∑wj:c(wii−1)>0pML(wi) $\alpha (w_{i-1})=\frac{1-\underset{w_j:c(w_{i-1}^i)>0}{\sum }{p}_{katz}(w_i|w_{i-1})}{1-\underset{w_j:c(w_{i-1}^i)>0}{\sum }{p}_{ML}(w_i)}$
pkatz(wi|wi−1)=ckatz(wii−1)∑wickatz(wii−1) $p_{katz}(w_i|w_{i-1})=\frac{c_{katz}(w_{i-1}^i)}{\underset{w_i}{\sum }{c}_{katz}(w_{i-1}^i)}$

Jelinek-Mercer平滑方法

pinterp(wi|wI−1I−n+1)=λwI−1I−n+1pML(wi|wI−1I−n+1)+(1−λwI−1I−n+1)pinterp(wi|wi−1I−n+2) $p_{interp}(w_i|w_{I-n+1}^{I-1})={\lambda }_{w_{I-n+1}^{I-1}}{p}_{ML}(w_i|w_{I-n+1}^{I-1})+(1-{\lambda }_{w_{I-n+1}^{I-1}})p_{interp}(w_i|w_{I-n+2}^{i-1})$

Witten-Bell平滑方法

pWB(wi|wI−1I−n+1)=c(wiI−n+1)+|wi:c(wi−1i−n+1⋅wi)>0|⋅pWB(wi|wI−1I−n+2)∑wic(wii−n+1)+|wi:c(wi−1i−n+1⋅wi)>0| $p_{WB}(w_i|w_{I-n+1}^{I-1})=\frac{c(w_{I-n+1}^i)+|w_i:c(w_{i-n+1\cdot w_i}^{i-1})>0|\cdot p_{WB}(w_i|w_{I-n+2}^{I-1})}{\underset{w_i}{\sum }c(w_{i-n+1}^i)+|w_i:c(w_{i-n+1\cdot w_i}^{i-1})>0|}$

绝对值减法

pabs(wi|wI−1I−n+1)=max{(wii−n+1−D,0)}∑wic(wii−n+1) $p_{abs}(w_i|w_{I-n+1}^{I-1})=\frac{max\{(w_{i-n+1}^i-D,0)\}}{\underset{w_i}{\sum }c(w_{i-n+1}^i)}$

Knerser-Ney平滑方法

pKN(wi)=N1+(⋅wi)N1+(⋅⋅) $p_{KN}(w_i)=\frac{N_{1+}(\cdot w_i)}{N_{1+}(\cdot \cdot )}$
pKN(wi|wi−1i−n+2)=N1+(⋅wi−1i−n+2)N1+(⋅wi−1i−n+2⋅) $p_{KN}(w_i|w_{i-n+2}^{i-1})=\frac{N_{1+}(\cdot w_{i-n+2}^{i-1})}{N_{1+}(\cdot w_{i-n+2}^{i-1}\cdot )}$

修正的Knerser-Ney平滑方法

修正的KN平滑方法修改 pKN(wi|wi−1i−n+1) $p_{KN}(w_i|w_{i-n+1}^{i-1})$的表示如下：

pKN(wi|wi−1i−n+1)=c(wii−n+1)−D(c(wii−n+1))∑wic(wii−n+1)+γ(wi−1i−n+1)pKN(wi|wi−1i−n+2) $$p_{KN}(w_i|w_{i-n+1}^{i-1})=\frac{c(w_{i-n+1}^i)-D(c(w_{i-n+1}^i))}{\underset{w_i}{\sum }c(w_{i-n+1}^i)}+\gamma (w_{i-n+1}^{i-1})p_{KN}(w_i|w_{i-n+2}^{i-1})$$

γ(wi−1i−n+1)=D1N1(wi−1i−n+1⋅)+D2N2(wi−1i−n+1⋅)+D3+N3+(wi−1i−n+1⋅)∑wic(wii−n+1) $\gamma (w_{i-n+1}^{i-1})=\frac{D_1{N}_1(w_{i-n+1}^{i-1}\cdot )+D_2{N}_2(w_{i-n+1}^{i-1}\cdot )+D_{3+}{N}_{3+}(w_{i-n+1}^{i-1}\cdot )}{\underset{w_i}{\sum }c(w_{i-n+1}^i)}$

D(c)=0,c=0=D1,c=1=D2,c=2=D3+,c≥3 $$\begin{gathered} D(c) \\ =0,c=0 \\ =D_1,c=1 \\ =D_2,c=2 \\ =D_{3+},c\ge 3 \end{gathered}$$

Y=n1n1+2n2D1=1−2Yn2n1D2=2−3Yn3n2D3+=3−4Yn4n3 $$\begin{gathered} Y=\frac{n_1}{n_1+2n_2} \\ {D}_1=1-2Y\frac{n_2}{n_1} \\ {D}_2=2-3Y\frac{n_3}{n_2} \\ {D}_{3+}=3-4Y\frac{n_4}{n_3} \end{gathered}$$

Reference：
《统计自然语言处理》