self-attention为什么要除以根号d_k

时间:2024-12-10 07:15:09
公式分析,首先假设q和k都是服从期望为0,方差为1的独立的随机变量。
Assume: X = q i X=q_{i} X=qi Y = k i Y=k_{i} Y=ki,那么:
1、 E ( X Y ) = E ( X ) E ( Y ) = 0 ∗ 0 = 0 E(XY)=E(X)E(Y)=0*0=0 E(XY)=E(X)E(Y)=00=0
2、 D ( X Y ) = E ( X 2 Y 2 ) − [ E ( X Y ) ] 2 D(XY)=E(X^{2}Y^{2})-[E(XY)]^{2} D(XY)=E(X2Y2)[E(XY)]2
       = E ( X 2 ) E ( Y 2 ) − [ E ( X ) E ( Y ) ] 2 =E(X^{2})E(Y^{2})-[E(X)E(Y)]^{2} =E(X2)E(Y2)[E(X)E(Y)]2
       = E ( X 2 − 0 2 ) E ( Y 2 − 0 2 ) − [ E ( X ) E ( Y ) ] 2 =E(X^{2}-0^{2})E(Y^{2}-0^{2})-[E(X)E(Y)]^{2} =E(X202)E(Y202)[E(X)E(Y)]2
       = E ( X 2 − [ E ( X ) ] 2 ) E ( Y 2 − [ E ( Y ) ] 2 ) − [ E ( X ) E ( Y ) ] 2 =E(X^{2}-[E(X)]^{2})E(Y^{2}-[E(Y)]^{2})-[E(X)E(Y)]^{2} =E(X2[E(X)]2)E(Y2[E(Y)]2)[E(X)E(Y)]2
       = [ E ( X 2 ) − [ E ( X ) ] 2 ] [ E ( Y 2 ) − [ E ( Y ) ] 2 ] − [ E ( X ) E ( Y ) ] 2 =[E(X^{2})-[E(X)]^{2}][E(Y^{2})-[E(Y)]^{2}]-[E(X)E(Y)]^{2} =[E(X2)[E(X)]2][E(Y2)[E(Y)]2][E(X)E(Y)]2
       = D ( X ) D ( Y ) − [ E ( X ) E ( Y ) ] 2 =D(X)D(Y)-[E(X)E(Y)]^{2} =D(X)D(Y)[E(X)E(Y)]2
       = 1 ∗ 1 − 0 ∗ 0 =1*1-0*0 =1100
       = 1 =1 =1
3、 D ( Q K d k ) = d k ( d k ) 2 = 1 D(\frac{QK}{\sqrt{d_{k}}})=\frac{d_{k}}{(\sqrt{d_{k}})^{2}}=1 D(dk QK)=(dk )2dk=1
需要注意的是, D ( Q K ) = D ( ∑ i = 0 d k q i k i ) = d k ∗ 1 = d k D(QK)=D(\sum_{i=0}^{d_{k}}q_{i}k_{i})=d_{k}*1=d_{k} D(QK)=D(i=0dkqiki)=dk1=dk