投影4-Least square approximation II - 《线性代数笔记》

先回忆记号：设

$投影4-Least square approximation II - 图1$

我们假设当 $投影4-Least square approximation II - 图2$ 时，有 $投影4-Least square approximation II - 图3$ . 所以 $投影4-Least square approximation II - 图4$ %3D2#card=math&code=r%28A%29%3D2&id=leDzL).

最小二乘问题：考虑方程组

$投影4-Least square approximation II - 图5$

求常数 $投影4-Least square approximation II - 图6$ ，使得目标函数 $投影4-Least square approximation II - 图7$ 取得最小值。

解：

我们用正交投影的公式直接计算 $投影4-Least square approximation II - 图8$ #card=math&code=P_U%28%5Cmathbf%7BY%7D%29&id=FCWhc), 从而导出最小二乘问题的法方程。用Schmidt正交化，从 $投影4-Least square approximation II - 图9$ 的基 $投影4-Least square approximation II - 图10$ 中得到正交基 $投影4-Least square approximation II - 图11$ , 其中

$投影4-Least square approximation II - 图12$ %7D%7B(X%2C%20X)%7DX%7D.%0A#card=math&code=%5Cmathbf%7BX%27%20%3D%201%20-%20%5Cfrac%7B%28X%2C%201%29%7D%7B%28X%2C%20X%29%7DX%7D.%0A&id=dliX9)

那么

$投影4-Least square approximation II - 图13$ %20%26%3D%20%5Cmathbf%7B%5Cfrac%7B(Y%2C%20X)%7D%7B(X%2C%20X)%7DX%20%2B%20%5Cfrac%7B(Y%2C%20X’)%7D%7B(X’%2C%20X’)%7DX’%7D%5C%5C%0A%0A%26%3D%0A%0A%5Cbegin%7Bpmatrix%7D%20%5Cmathbf%7BX%7D%20%26%20%5Cmathbf%7BX’%7D%20%5Cend%7Bpmatrix%7D%20%0A%0A%5Cbegin%7Bpmatrix%7D%0A%0A%5C%7C%5Cmathbf%7BX%7D%5C%7C%5E%7B-2%7D%20%26%20%5C%5C%20%26%20%5C%7C%5Cmathbf%7BX’%7D%5C%7C%5E%7B-2%7D%0A%0A%5Cend%7Bpmatrix%7D%0A%0A%5Cbegin%7Bpmatrix%7D%0A%0A%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BY%7D%5C%5C%20%5Cmathbf%7BX’%7D%5E%5Ctop%5Cmathbf%7BY%7D%0A%0A%5Cend%7Bpmatrix%7D.%0A%0A%5Cend%7Baligned%7D%0A#card=math&code=%5Cbegin%7Baligned%7D%0A%0AP_U%28%5Cmathbf%7BY%7D%29%20%26%3D%20%5Cmathbf%7B%5Cfrac%7B%28Y%2C%20X%29%7D%7B%28X%2C%20X%29%7DX%20%2B%20%5Cfrac%7B%28Y%2C%20X%27%29%7D%7B%28X%27%2C%20X%27%29%7DX%27%7D%5C%5C%0A%0A%26%3D%0A%0A%5Cbegin%7Bpmatrix%7D%20%5Cmathbf%7BX%7D%20%26%20%5Cmathbf%7BX%27%7D%20%5Cend%7Bpmatrix%7D%20%0A%0A%5Cbegin%7Bpmatrix%7D%0A%0A%5C%7C%5Cmathbf%7BX%7D%5C%7C%5E%7B-2%7D%20%26%20%5C%5C%20%26%20%5C%7C%5Cmathbf%7BX%27%7D%5C%7C%5E%7B-2%7D%0A%0A%5Cend%7Bpmatrix%7D%0A%0A%5Cbegin%7Bpmatrix%7D%0A%0A%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BY%7D%5C%5C%20%5Cmathbf%7BX%27%7D%5E%5Ctop%5Cmathbf%7BY%7D%0A%0A%5Cend%7Bpmatrix%7D.%0A%0A%5Cend%7Baligned%7D%0A&id=T9vz9)

代入基变换矩阵

$投影4-Least square approximation II - 图14$ %7D%7B(%5Cmathbf%7BX%2C%20X%7D)%7D%20%5C%5C%200%20%26%201%0A%0A%5Cend%7Bpmatrix%7D%2C%0A#card=math&code=%5Cbegin%7Bpmatrix%7D%20%5Cmathbf%7BX%7D%20%26%20%5Cmathbf%7BX%27%7D%20%5Cend%7Bpmatrix%7D%20%20%3D%20%0A%0A%5Cbegin%7Bpmatrix%7D%20%5Cmathbf%7BX%7D%20%26%20%5Cmathbf%7B1%7D%20%5Cend%7Bpmatrix%7D%20%0A%0A%5Cbegin%7Bpmatrix%7D%0A%0A1%20%26%20-%20%5Cfrac%7B%28%5Cmathbf%7BX%2C%201%7D%29%7D%7B%28%5Cmathbf%7BX%2C%20X%7D%29%7D%20%5C%5C%200%20%26%201%0A%0A%5Cend%7Bpmatrix%7D%2C%0A&id=GtEeX)

我们先计算：

$投影4-Least square approximation II - 图15$ %7D%7B(%5Cmathbf%7BX%2C%20X%7D)%7D%20%5C%5C%200%20%26%201%0A%0A%5Cend%7Bpmatrix%7D%5E%7B-1%7D%20%0A%0A%5Cbegin%7Bpmatrix%7D%0A%0A%5C%7C%5Cmathbf%7BX%7D%5C%7C%5E%7B2%7D%20%26%20%5C%5C%20%26%20%5C%7C%5Cmathbf%7BX’%7D%5C%7C%5E%7B2%7D%0A%0A%5Cend%7Bpmatrix%7D%5E%7B-1%7D%0A%0A%5Cbegin%7Bpmatrix%7D%0A%0A1%20%26%200%20%5C%5C%20%0A%0A-%20%5Cfrac%7B(%5Cmathbf%7BX%2C%201%7D)%7D%7B(%5Cmathbf%7BX%2C%20X%7D)%7D%20%26%201%0A%0A%5Cend%7Bpmatrix%7D%5E%7B-1%7D%20%5C%5C%0A%0A%3D%26%20%5Cbigg%5B%20%0A%0A%5Cbegin%7Bpmatrix%7D%0A%0A1%20%26%200%20%5C%5C%20%0A%0A-%20%5Cfrac%7B(%5Cmathbf%7BX%2C%201%7D)%7D%7B(%5Cmathbf%7BX%2C%20X%7D)%7D%20%26%201%0A%0A%5Cend%7Bpmatrix%7D%0A%0A%5Cbegin%7Bpmatrix%7D%0A%0A%5Cmathbf%7B(X%2C%20X)%7D%20%26%20%5C%5C%20%26%20%5Cmathbf%7B(X’%2C%20X’)%7D%0A%0A%5Cend%7Bpmatrix%7D%0A%0A%5Cbegin%7Bpmatrix%7D%0A%0A1%20%26%20-%20%5Cfrac%7B(%5Cmathbf%7BX%2C%201%7D)%7D%7B(%5Cmathbf%7BX%2C%20X%7D)%7D%20%5C%5C%200%20%26%201%0A%0A%5Cend%7Bpmatrix%7D%0A%0A%5Cbigg%5D%5E%7B-1%7D%20%5C%5C%0A%0A%3D%26%5Cbigg%5B%20%5Cbegin%7Bpmatrix%7D%20%5Cmathbf%7BX%7D%5E%5Ctop%20%5C%5C%20%5Cmathbf%7B1%7D%5E%5Ctop%20%5Cend%7Bpmatrix%7D%20%5Cbegin%7Bpmatrix%7D%20%5Cmathbf%7BX%7D%26%5Cmathbf%7B1%7D%5Cend%7Bpmatrix%7D%5Cbigg%5D%5E%7B-1%7D%5C%5C%0A%0A%3D%26%20(A%5E%5Ctop%20A)%5E%7B-1%7D%2C%0A%0A%5Cend%7Baligned%7D%0A#card=math&code=%5Cbegin%7Baligned%7D%0A%0A%26%0A%0A%5Cbegin%7Bpmatrix%7D%0A%0A1%20%26%20-%20%5Cfrac%7B%28%5Cmathbf%7BX%2C%201%7D%29%7D%7B%28%5Cmathbf%7BX%2C%20X%7D%29%7D%20%5C%5C%200%20%26%201%0A%0A%5Cend%7Bpmatrix%7D%5E%7B-1%7D%20%0A%0A%5Cbegin%7Bpmatrix%7D%0A%0A%5C%7C%5Cmathbf%7BX%7D%5C%7C%5E%7B2%7D%20%26%20%5C%5C%20%26%20%5C%7C%5Cmathbf%7BX%27%7D%5C%7C%5E%7B2%7D%0A%0A%5Cend%7Bpmatrix%7D%5E%7B-1%7D%0A%0A%5Cbegin%7Bpmatrix%7D%0A%0A1%20%26%200%20%5C%5C%20%0A%0A-%20%5Cfrac%7B%28%5Cmathbf%7BX%2C%201%7D%29%7D%7B%28%5Cmathbf%7BX%2C%20X%7D%29%7D%20%26%201%0A%0A%5Cend%7Bpmatrix%7D%5E%7B-1%7D%20%5C%5C%0A%0A%3D%26%20%5Cbigg%5B%20%0A%0A%5Cbegin%7Bpmatrix%7D%0A%0A1%20%26%200%20%5C%5C%20%0A%0A-%20%5Cfrac%7B%28%5Cmathbf%7BX%2C%201%7D%29%7D%7B%28%5Cmathbf%7BX%2C%20X%7D%29%7D%20%26%201%0A%0A%5Cend%7Bpmatrix%7D%0A%0A%5Cbegin%7Bpmatrix%7D%0A%0A%5Cmathbf%7B%28X%2C%20X%29%7D%20%26%20%5C%5C%20%26%20%5Cmathbf%7B%28X%27%2C%20X%27%29%7D%0A%0A%5Cend%7Bpmatrix%7D%0A%0A%5Cbegin%7Bpmatrix%7D%0A%0A1%20%26%20-%20%5Cfrac%7B%28%5Cmathbf%7BX%2C%201%7D%29%7D%7B%28%5Cmathbf%7BX%2C%20X%7D%29%7D%20%5C%5C%200%20%26%201%0A%0A%5Cend%7Bpmatrix%7D%0A%0A%5Cbigg%5D%5E%7B-1%7D%20%5C%5C%0A%0A%3D%26%5Cbigg%5B%20%5Cbegin%7Bpmatrix%7D%20%5Cmathbf%7BX%7D%5E%5Ctop%20%5C%5C%20%5Cmathbf%7B1%7D%5E%5Ctop%20%5Cend%7Bpmatrix%7D%20%5Cbegin%7Bpmatrix%7D%20%5Cmathbf%7BX%7D%26%5Cmathbf%7B1%7D%5Cend%7Bpmatrix%7D%5Cbigg%5D%5E%7B-1%7D%5C%5C%0A%0A%3D%26%20%28A%5E%5Ctop%20A%29%5E%7B-1%7D%2C%0A%0A%5Cend%7Baligned%7D%0A&id=a5FF5)

由此可见

$投影4-Least square approximation II - 图16$ %3D%20A(A%5E%5Ctop%20A)%5E%7B-1%7D%20A%5E%5Ctop%20%5Cmathbf%7BY%7D.%0A#card=math&code=P_U%28%5Cmathbf%7BY%7D%29%3D%20A%28A%5E%5Ctop%20A%29%5E%7B-1%7D%20A%5E%5Ctop%20%5Cmathbf%7BY%7D.%0A&id=rN9nA)

余下就是解出法方程 $投影4-Least square approximation II - 图17$ %20%3D%20A%5E%5Ctop%20%5Cmathbf%7BY%7D#card=math&code=A%5E%5Ctop%20A%20%5Cbig%28%20%5Cbegin%7Bsmallmatrix%7D%20a%20%5C%5C%20b%5Cend%7Bsmallmatrix%7D%5Cbig%29%20%3D%20A%5E%5Ctop%20%5Cmathbf%7BY%7D&id=ADHgM). 搞掂！

下面简介高维（即多变量）的线性最小二乘回归问题：给定了 $投影4-Least square approximation II - 图18$ 中的 $投影4-Least square approximation II - 图19$ #card=math&code=n%20%5C%2C%20%28n%20%3E%20d%2B1%29&id=Xm7tY)个数据点 $投影4-Least square approximation II - 图20$ %2C%20%5C%2C%20(1%5Cleq%20i%20%5Cleq%20n)#card=math&code=Pi%28a%7Bi1%7D%2C%20%5Cldots%2C%20a_%7Bid%7D%2C%20b_i%29%2C%20%5C%2C%20%281%5Cleq%20i%20%5Cleq%20n%29&id=S16oI). 求实数 $投影4-Least square approximation II - 图21$ , 使得目标函数

$投影4-Least square approximation II - 图22$ %5E2%0A#card=math&code=%5Csum%7Bi%3D1%7D%5En%20%28a%7Bi1%7Dc1%20%2B%20a%7Bi2%7Dc2%20%2B%20%5Cldots%20%2B%20a%7Bid%7Dc_d%20-%20b_i%29%5E2%0A&id=EvbJM)

取最小值.

记

$投影4-Least square approximation II - 图23$

那么目标函数可以表达为

$投影4-Least square approximation II - 图24$

使目标函数取得最小值的解 $投影4-Least square approximation II - 图25$ 称为最小二乘解. 之前我们讨论的是 $投影4-Least square approximation II - 图26$ 的情形. 一般情况的解法完全一样. 即，最小二乘解 $投影4-Least square approximation II - 图27$ 应该使得 $投影4-Least square approximation II - 图28$ 与 $投影4-Least square approximation II - 图29$ 的列向量 $投影4-Least square approximation II - 图30$ 张成的线性空间正交. 为此，必须且只需

$投影4-Least square approximation II - 图31$ %20%3D%20%5Cmathbf%7Ba%7D_i%5E%5Ctop%20(AC-B)%20%3D%200%2C%20%5Cquad%201%5Cleq%20i%20%5Cleq%20d.%20%0A%0A%5Cend%7Bequation%7D%0A#card=math&code=%5Cbegin%7Bequation%7D%5Clabel%7Beq%3Alsq_normal_eq%7D%0A%0A%28AC-B%2C%20%5Cmathbf%7Ba%7D_i%29%20%3D%20%5Cmathbf%7Ba%7D_i%5E%5Ctop%20%28AC-B%29%20%3D%200%2C%20%5Cquad%201%5Cleq%20i%20%5Cleq%20d.%20%0A%0A%5Cend%7Bequation%7D%0A&id=fdyln)

由于向量投影4-Least square approximation II - 图32 按行恰好排成矩阵于是上述条件合起来，用矩阵来表示，正好是

$投影4-Least square approximation II - 图34$ %20%3D%20%20%5Cmathbf%7B0%7D%20%5CLongleftrightarrow%20A%5E%5Ctop%20A%20C%20%3D%20A%5E%5Ctop%20B.%0A%0A%5Cend%7Bequation%7D%0A#card=math&code=%5Cbegin%7Bequation%7D%0A%0AA%5E%5Ctop%20%28AC-B%29%20%3D%20%20%5Cmathbf%7B0%7D%20%5CLongleftrightarrow%20A%5E%5Ctop%20A%20C%20%3D%20A%5E%5Ctop%20B.%0A%0A%5Cend%7Bequation%7D%0A&id=iJJlz)

这就是线性最小二乘回归问题的法方程.