S03P04-高斯判别分析

高斯判别分析(GDA)

设二分类样本%5C%7D%7Bi%3D1%7D%5EN%2CX_i%5Cin%5Cmathbb%7BR%7D%5Ep%2Cy_i%5Cin%5C%7B0%2C1%5C%7D#card=math&code=D%3D%5C%7B%28X_i%2Cy_i%29%5C%7D%7Bi%3D1%7D%5EN%2CX_i%5Cin%5Cmathbb%7BR%7D%5Ep%2Cy_i%5Cin%5C%7B0%2C1%5C%7D)

模型的目的就是通过已知的预测，即求#card=math&code=P%28y%7CX%29)。根据贝叶斯定理我们知道

%3D%5Cfrac%7BP(X%2Cy)%7D%7BP(X)%7D%3D%5Cfrac%7BP(X%7Cy)P(y)%7D%7BP(X)%7D%0A#card=math&code=P%28y%7CX%29%3D%5Cfrac%7BP%28X%2Cy%29%7D%7BP%28X%29%7D%3D%5Cfrac%7BP%28X%7Cy%29P%28y%29%7D%7BP%28X%29%7D%0A)

那么如果要求#card=math&code=P%28y%7CX%29)即要求出#card=math&code=P%28X%2Cy%29)。所以高斯判别模型是生成模型，它可以求出联合概率分布#card=math&code=P%28X%2Cy%29)，然后根据这个联合概率分布进一步作出预测

似然函数

因为%3DP(X%7Cy)P(y)#card=math&code=P%28X%2Cy%29%3DP%28X%7Cy%29P%28y%29)，所以我们对于和先给出两个基本假设：

1. 假设先验分布#card=math&code=y%5Csim%20Bernoulli%28%5CPhi%29)，那么%3D%0A%5Cbegin%7Bcases%7D%0A%5CPhi%5Ey%2C%26y%3D1%5C%5C%0A(1-%5CPhi)%5E%7B(1-y)%7D%2C%26y%3D0%0A%5Cend%7Bcases%7D%0A%5CRightarrow%20P(y)%3D%5CPhi%5Ey(1-%5CPhi)%5E%7B(1-y)%7D%0A#card=math&code=P%28y%29%3D%0A%5Cbegin%7Bcases%7D%0A%5CPhi%5Ey%2C%26y%3D1%5C%5C%0A%281-%5CPhi%29%5E%7B%281-y%29%7D%2C%26y%3D0%0A%5Cend%7Bcases%7D%0A%5CRightarrow%20P%28y%29%3D%5CPhi%5Ey%281-%5CPhi%29%5E%7B%281-y%29%7D%0A)

1. 假设和的条件下都服从高斯分布，且两个高斯分布的协方差矩阵相同即%5C%5C%0AX%7Cy%3D0%5Csim%20N(%5Cmu_0%2C%5CSigma)%0A%5Cend%7Bcases%7D%0A%5CRightarrow%20P(X%7Cy)%3DN(%5Cmu_1%2C%5CSigma)%5E%7B(y)%7DN(%5Cmu_0%2C%5CSigma)%5E%7B(1-y)%7D%0A#card=math&code=%5Cbegin%7Bcases%7D%0AX%7Cy%3D1%5Csim%20N%28%5Cmu_1%2C%5CSigma%29%5C%5C%0AX%7Cy%3D0%5Csim%20N%28%5Cmu_0%2C%5CSigma%29%0A%5Cend%7Bcases%7D%0A%5CRightarrow%20P%28X%7Cy%29%3DN%28%5Cmu_1%2C%5CSigma%29%5E%7B%28y%29%7DN%28%5Cmu_0%2C%5CSigma%29%5E%7B%281-y%29%7D%0A)

令#card=math&code=%5Ctheta%3D%28%5CPhi%2C%5Cmu_0%2C%5Cmu_1%2C%5CSigma%29)，对数似然函数为#card=math&code=L%28%5Ctheta%29)

%26%3D%5Clog%20P(X%2CY)%5C%5C%0A%26%3D%5Csum%7Bi%3D1%7D%5EN%5Clog%20P(X_i%2Cy_i)%5C%5C%0A%26%3D%5Csum%7Bi%3D1%7D%5EN%5Clog%20P(Xi%7Cy_i)P(y_i)%5C%5C%0A%26%3D%5Csum%7Bi%3D1%7D%5EN%5B%5Clog%20P(Xi%7Cy_i)%2B%5Clog%20P(y_i)%5D%5C%5C%0A%26%3D%5Csum%7Bi%3D1%7D%5EN%5B%5Clog%20N(%5Cmu1%2C%5CSigma)%5E%7B(y_i)%7DN(%5Cmu_0%2C%5CSigma)%5E%7B(1-y_i)%7D%2B%5Clog%5CPhi%5E%7By_i%7D(1-%5CPhi)%5E%7B(1-y_i)%7D%5D%5C%5C%0A%26%3D%5Csum%7Bi%3D1%7D%5EN%5Byi%5Clog%20N(%5Cmu_1%2C%5CSigma)%2B(1-y_i)%5Clog%20N(%5Cmu_0%2C%5CSigma)%2By_i%5Clog%5CPhi%2B(1-y_i)%5Clog(1-%5CPhi)%5D%0A%5Cend%7Baligned%7D%0A#card=math&code=%5Cbegin%7Baligned%7D%0AL%28%5Ctheta%29%26%3D%5Clog%20P%28X%2CY%29%5C%5C%0A%26%3D%5Csum%7Bi%3D1%7D%5EN%5Clog%20P%28Xi%2Cy_i%29%5C%5C%0A%26%3D%5Csum%7Bi%3D1%7D%5EN%5Clog%20P%28Xi%7Cy_i%29P%28y_i%29%5C%5C%0A%26%3D%5Csum%7Bi%3D1%7D%5EN%5B%5Clog%20P%28Xi%7Cy_i%29%2B%5Clog%20P%28y_i%29%5D%5C%5C%0A%26%3D%5Csum%7Bi%3D1%7D%5EN%5B%5Clog%20N%28%5Cmu1%2C%5CSigma%29%5E%7B%28y_i%29%7DN%28%5Cmu_0%2C%5CSigma%29%5E%7B%281-y_i%29%7D%2B%5Clog%5CPhi%5E%7By_i%7D%281-%5CPhi%29%5E%7B%281-y_i%29%7D%5D%5C%5C%0A%26%3D%5Csum%7Bi%3D1%7D%5EN%5By_i%5Clog%20N%28%5Cmu_1%2C%5CSigma%29%2B%281-y_i%29%5Clog%20N%28%5Cmu_0%2C%5CSigma%29%2By_i%5Clog%5CPhi%2B%281-y_i%29%5Clog%281-%5CPhi%29%5D%0A%5Cend%7Baligned%7D%0A)

参数求解

设的样本集为，的样本样本集为
令%3DL_0%2BL_1%2BL_2#card=math&code=L%28%5Ctheta%29%3DL_0%2BL_1%2BL_2)，其中

%5Clog%20N(%5Cmu0%2C%5CSigma)%5C%5C%0AL_1%3D%5Csum%7Bi%3D1%7D%5ENyi%5Clog%20N(%5Cmu_1%2C%5CSigma)%5C%5C%0AL_2%3D%5Csum%7Bi%3D1%7D%5ENyi%5Clog%5CPhi%2B(1-y_i)%5Clog(1-%5CPhi)%0A%5Cend%7Bgathered%7D%0A#card=math&code=%5Cbegin%7Bgathered%7D%0AL_0%3D%5Csum%7Bi%3D1%7D%5EN%281-yi%29%5Clog%20N%28%5Cmu_0%2C%5CSigma%29%5C%5C%0AL_1%3D%5Csum%7Bi%3D1%7D%5ENyi%5Clog%20N%28%5Cmu_1%2C%5CSigma%29%5C%5C%0AL_2%3D%5Csum%7Bi%3D1%7D%5ENy_i%5Clog%5CPhi%2B%281-y_i%29%5Clog%281-%5CPhi%29%0A%5Cend%7Bgathered%7D%0A)

根据极大似然估计有#card=math&code=%5Chat%7B%5Ctheta%7D%3D%5Carg%5Cmax%5Climits_%5Ctheta%20L%28%5Ctheta%29)

1. 求%7D%7B%5Cpartial%5CPhi%7D%3D%5Cfrac%7B%5Cpartial%20L2%7D%7B%5Cpartial%5CPhi%7D%3D%5Csum%7Bi%3D1%7D%5EN%5Cleft%5Byi%7B1%5Cover%5CPhi%7D%2B(1-y_i)(-%7B1%5Cover%7B1-%5CPhi%7D%7D)%5Cright%5D%3D0%5C%5C%0A%5CRightarrow%5CPhi%3D%5Cfrac%7B%5Csum%7Bi%3D1%7D%5ENyi%7D%7BN%7D%3D%5Cfrac%7BN_1%7D%7BN%7D%0A%5Cend%7Bgathered%7D%0A#card=math&code=%5Cbegin%7Bgathered%7D%0A%5Cfrac%7B%5Cpartial%20L%28%5Ctheta%29%7D%7B%5Cpartial%5CPhi%7D%3D%5Cfrac%7B%5Cpartial%20L_2%7D%7B%5Cpartial%5CPhi%7D%3D%5Csum%7Bi%3D1%7D%5EN%5Cleft%5Byi%7B1%5Cover%5CPhi%7D%2B%281-y_i%29%28-%7B1%5Cover%7B1-%5CPhi%7D%7D%29%5Cright%5D%3D0%5C%5C%0A%5CRightarrow%5CPhi%3D%5Cfrac%7B%5Csum%7Bi%3D1%7D%5ENy_i%7D%7BN%7D%3D%5Cfrac%7BN_1%7D%7BN%7D%0A%5Cend%7Bgathered%7D%0A)

1. 求%3D%5Carg%5Cmax%7B%5Cmu_1%7DL_1%5C%5C%0A%26%3D%5Carg%5Cmax%7B%5Cmu1%7D%5Csum%7Bi%3D1%7D%5ENyi%5Clog%20N(%5Cmu_1%2C%5CSigma)%5C%5C%0A%26%3D%5Carg%5Cmax%7B%5Cmu1%7D%5Csum%7Bi%3D1%7D%5ENyi%5Clog%5Cleft(%7B1%5Cover(2%5Cpi)%5E%7Bp%2F2%7D%7C%5CSigma%7C%5E%7B1%2F2%7D%7D%5Cexp%5B-%7B1%5Cover%202%7D(X_i-%5Cmu_1)%5ET%5CSigma%5E%7B-1%7D(X_i-%5Cmu_1)%5D%5Cright)%5C%5C%0A%26%3D%5Carg%5Cmin%7B%5Cmu1%7D%5Csum%7Bi%3D1%7D%5ENyi(X_i-%5Cmu_1)%5ET%5CSigma%5E%7B-1%7D(X_i-%5Cmu_1)%0A%5Cend%7Baligned%7D%0A#card=math&code=%5Cbegin%7Baligned%7D%0A%5Cmu_1%26%3D%5Carg%5Cmax%7B%5Cmu1%7DL%28%5Ctheta%29%3D%5Carg%5Cmax%7B%5Cmu1%7DL_1%5C%5C%0A%26%3D%5Carg%5Cmax%7B%5Cmu1%7D%5Csum%7Bi%3D1%7D%5ENyi%5Clog%20N%28%5Cmu_1%2C%5CSigma%29%5C%5C%0A%26%3D%5Carg%5Cmax%7B%5Cmu1%7D%5Csum%7Bi%3D1%7D%5ENyi%5Clog%5Cleft%28%7B1%5Cover%282%5Cpi%29%5E%7Bp%2F2%7D%7C%5CSigma%7C%5E%7B1%2F2%7D%7D%5Cexp%5B-%7B1%5Cover%202%7D%28X_i-%5Cmu_1%29%5ET%5CSigma%5E%7B-1%7D%28X_i-%5Cmu_1%29%5D%5Cright%29%5C%5C%0A%26%3D%5Carg%5Cmin%7B%5Cmu1%7D%5Csum%7Bi%3D1%7D%5ENy_i%28X_i-%5Cmu_1%29%5ET%5CSigma%5E%7B-1%7D%28X_i-%5Cmu_1%29%0A%5Cend%7Baligned%7D%0A)

令%5ET%5CSigma%5E%7B-1%7D(Xi-%5Cmu_1)#card=math&code=%5CDelta%3D%5Csum%7Bi%3D1%7D%5ENyi%28X_i-%5Cmu_1%29%5ET%5CSigma%5E%7B-1%7D%28X_i-%5Cmu_1%29)%3D0%5C%5C%0A%5CRightarrow%5Chat%7B%5Cmu_1%7D%3D%5Cfrac%7B%5Csum%5Climits%7Bi%3D1%7D%5ENyiX_i%7D%7B%5Csum%7Bi%3D1%7D%5ENyi%7D%3D%5Cfrac%7B%5Csum%5Climits%7BXi%5Cin%20D_1%7DX_i%7D%7BN_1%7D%0A%5Cend%7Bgathered%7D%0A#card=math&code=%5Cbegin%7Bgathered%7D%0A%5Cfrac%7B%5Cpartial%5CDelta%7D%7B%5Cpartial%5Cmu_1%7D%3D2%5CSigma%5E%7B-1%7D%5Csum%7Bi%3D1%7D%5ENyi%28%5Cmu_1-X_i%29%3D0%5C%5C%0A%5CRightarrow%5Chat%7B%5Cmu_1%7D%3D%5Cfrac%7B%5Csum%5Climits%7Bi%3D1%7D%5ENyiX_i%7D%7B%5Csum%7Bi%3D1%7D%5ENyi%7D%3D%5Cfrac%7B%5Csum%5Climits%7BXi%5Cin%20D_1%7DX_i%7D%7BN_1%7D%0A%5Cend%7Bgathered%7D%0A)

1. 求%3D%5Csum%5Climits%7BX_i%5Cin%20D_1%7D%5Clog%20N(%5Cmu_1%2C%5CSigma)%5C%5C%0A%26%3D%5Csum%5Climits%7BXi%5Cin%20D_1%7D%5Clog%5Cleft(%7B1%5Cover(2%5Cpi)%5E%7Bp%2F2%7D%7C%5CSigma%7C%5E%7B1%2F2%7D%7D%5Cexp%5B-%7B1%5Cover%202%7D(X_i-%5Cmu_1)%5ET%5CSigma%5E%7B-1%7D(X_i-%5Cmu_1)%5D%5Cright)%5C%5C%0A%26%3D%5Csum%5Climits%7BXi%5Cin%20D_1%7D%5Cleft%5B%5Clog%7B1%5Cover(2%5Cpi)%5E%7Bp%2F2%7D%7D-%7B1%5Cover%202%7D%5Clog%7C%5CSigma%7C-%7B1%5Cover%202%7D(X_i-%5Cmu_1)%5ET%5CSigma%5E%7B-1%7D(X_i-%5Cmu_1)%5Cright%5D%5C%5C%0A%26%3DC-%7B1%5Cover%202%7D%5Csum%5Climits%7BXi%5Cin%20D_1%7D%5C%7B%5Clog%7C%5CSigma%7C%2Btr%5B(X_i-%5Cmu_1)%5ET%5CSigma%5E%7B-1%7D(X_i-%5Cmu_1)%5D%5C%7D%5C%5C%0A%26%3DC-%7B1%5Cover%202%7D%5Csum%5Climits%7BXi%5Cin%20D_1%7D%5Clog%7C%5CSigma%7C-%7B1%5Cover%202%7D%5Csum%5Climits%7BXi%5Cin%20D_1%7D(tr%5B(X_i-%5Cmu_1)(X_i-%5Cmu_1)%5ET%5CSigma%5E%7B-1%7D%5D)%5C%5C%0A%26%3DC-%7B1%5Cover%202%7DN_1%5Clog%7C%5CSigma%7C-%7B1%5Cover%202%7Dtr%5Cleft%5BN_1%5Cleft(%5Csum%5Climits%7BXi%5Cin%20D_1%7D%7B1%5Cover%7BN_1%7D%7D(X_i-%5Cmu_1)(X_i-%5Cmu_1)%5ET%5Cright)%5CSigma%5E%7B-1%7D%5Cright%5D%5C%5C%0A%26%3DC-%7B1%5Cover%202%7DN_1%5Clog%7C%5CSigma%7C-%7B1%5Cover%202%7DN_1tr(S_1%5CSigma%5E%7B-1%7D)%0A%5Cend%7Baligned%7D%5C%5C%0A%5C%5C%0A%5CRightarrow%5Cfrac%7B%5Cpartial%20L_1%7D%7B%5Cpartial%5CSigma%7D%3D-%7BN_1%5Cover%202%7D(%5CSigma%5E%7B-1%7D-S_1%5CSigma%5E%7B-2%7D)%0A%5Cend%7Bgathered%7D%0A#card=math&code=%5Cbegin%7Bgathered%7D%0A%5Cbegin%7Baligned%7D%0AL_1%26%3D%5Csum%7Bi%3D1%7D%5ENyi%5Clog%20N%28%5Cmu_1%2C%5CSigma%29%3D%5Csum%5Climits%7BXi%5Cin%20D_1%7D%5Clog%20N%28%5Cmu_1%2C%5CSigma%29%5C%5C%0A%26%3D%5Csum%5Climits%7BXi%5Cin%20D_1%7D%5Clog%5Cleft%28%7B1%5Cover%282%5Cpi%29%5E%7Bp%2F2%7D%7C%5CSigma%7C%5E%7B1%2F2%7D%7D%5Cexp%5B-%7B1%5Cover%202%7D%28X_i-%5Cmu_1%29%5ET%5CSigma%5E%7B-1%7D%28X_i-%5Cmu_1%29%5D%5Cright%29%5C%5C%0A%26%3D%5Csum%5Climits%7BXi%5Cin%20D_1%7D%5Cleft%5B%5Clog%7B1%5Cover%282%5Cpi%29%5E%7Bp%2F2%7D%7D-%7B1%5Cover%202%7D%5Clog%7C%5CSigma%7C-%7B1%5Cover%202%7D%28X_i-%5Cmu_1%29%5ET%5CSigma%5E%7B-1%7D%28X_i-%5Cmu_1%29%5Cright%5D%5C%5C%0A%26%3DC-%7B1%5Cover%202%7D%5Csum%5Climits%7BXi%5Cin%20D_1%7D%5C%7B%5Clog%7C%5CSigma%7C%2Btr%5B%28X_i-%5Cmu_1%29%5ET%5CSigma%5E%7B-1%7D%28X_i-%5Cmu_1%29%5D%5C%7D%5C%5C%0A%26%3DC-%7B1%5Cover%202%7D%5Csum%5Climits%7BXi%5Cin%20D_1%7D%5Clog%7C%5CSigma%7C-%7B1%5Cover%202%7D%5Csum%5Climits%7BXi%5Cin%20D_1%7D%28tr%5B%28X_i-%5Cmu_1%29%28X_i-%5Cmu_1%29%5ET%5CSigma%5E%7B-1%7D%5D%29%5C%5C%0A%26%3DC-%7B1%5Cover%202%7DN_1%5Clog%7C%5CSigma%7C-%7B1%5Cover%202%7Dtr%5Cleft%5BN_1%5Cleft%28%5Csum%5Climits%7BX_i%5Cin%20D_1%7D%7B1%5Cover%7BN_1%7D%7D%28X_i-%5Cmu_1%29%28X_i-%5Cmu_1%29%5ET%5Cright%29%5CSigma%5E%7B-1%7D%5Cright%5D%5C%5C%0A%26%3DC-%7B1%5Cover%202%7DN_1%5Clog%7C%5CSigma%7C-%7B1%5Cover%202%7DN_1tr%28S_1%5CSigma%5E%7B-1%7D%29%0A%5Cend%7Baligned%7D%5C%5C%0A%5C%5C%0A%5CRightarrow%5Cfrac%7B%5Cpartial%20L_1%7D%7B%5Cpartial%5CSigma%7D%3D-%7BN_1%5Cover%202%7D%28%5CSigma%5E%7B-1%7D-S_1%5CSigma%5E%7B-2%7D%29%0A%5Cend%7Bgathered%7D%0A)
   %0A#card=math&code=%5Cfrac%7B%5Cpartial%20L_0%7D%7B%5Cpartial%5CSigma%7D%3D-%7BN_0%5Cover%202%7D%28%5CSigma%5E%7B-1%7D-S_0%5CSigma%5E%7B-2%7D%29%0A)
   %7D%7B%5Cpartial%5CSigma%7D%26%3D%5Cfrac%7B%5Cpartial%20L_0%7D%7B%5Cpartial%5CSigma%7D%2B%5Cfrac%7B%5Cpartial%20L_1%7D%7B%5Cpartial%5CSigma%7D%5C%5C%0A%26%3D-%7BN_0%5Cover%202%7D(%5CSigma%5E%7B-1%7D-S_0%5CSigma%5E%7B-2%7D)-%7BN_1%5Cover%202%7D(%5CSigma%5E%7B-1%7D-S_1%5CSigma%5E%7B-2%7D)%5C%5C%0A%26%3D-%7BN%5Cover%202%7D%5CSigma%5E%7B-1%7D%2B%5Cfrac%7BN_0S_0%2BN_1S_1%7D%7B2%7D%5CSigma%5E%7B-2%7D%5C%5C%0A%26%3D0%0A%5Cend%7Baligned%7D%5C%5C%0A%5CRightarrow%5Chat%7B%5CSigma%7D%3D%5Cfrac%7BN_0S_0%2BN_1S_1%7D%7BN%7D%0A%5Cend%7Bgathered%7D%0A#card=math&code=%5Cbegin%7Bgathered%7D%0A%5Cbegin%7Baligned%7D%0A%5Cfrac%7B%5Cpartial%20L%28%5Ctheta%29%7D%7B%5Cpartial%5CSigma%7D%26%3D%5Cfrac%7B%5Cpartial%20L_0%7D%7B%5Cpartial%5CSigma%7D%2B%5Cfrac%7B%5Cpartial%20L_1%7D%7B%5Cpartial%5CSigma%7D%5C%5C%0A%26%3D-%7BN_0%5Cover%202%7D%28%5CSigma%5E%7B-1%7D-S_0%5CSigma%5E%7B-2%7D%29-%7BN_1%5Cover%202%7D%28%5CSigma%5E%7B-1%7D-S_1%5CSigma%5E%7B-2%7D%29%5C%5C%0A%26%3D-%7BN%5Cover%202%7D%5CSigma%5E%7B-1%7D%2B%5Cfrac%7BN_0S_0%2BN_1S_1%7D%7B2%7D%5CSigma%5E%7B-2%7D%5C%5C%0A%26%3D0%0A%5Cend%7Baligned%7D%5C%5C%0A%5CRightarrow%5Chat%7B%5CSigma%7D%3D%5Cfrac%7BN_0S_0%2BN_1S_1%7D%7BN%7D%0A%5Cend%7Bgathered%7D%0A)