Computing Parameters Analytically

Normal Equation

一般译作正规方程，标准方程

Normal equation: Method to solve for \theta analytically.

无需像gradient descent 一样迭代，而是一步求出结果

Review:

%7D%3D%5Ctext%20%7B%20value%20of%20feature%20%7D%20j%20%5Ctext%20%7B%20in%20%7D%20i%5E%7B%5Ctext%20%7Bth%20%7D%7D%20%5Ctext%20%7B%20training%20example%20%7D%24%24%EF%BC%8C%20%E4%B8%80%E8%88%AC%E6%88%91%E4%BB%AC%E8%AE%BE%E5%AE%9A%24%24x%7B0%7D%20%3D%201%0A#card=math&code=x%7Bj%7D%5E%7B%28i%29%7D%3D%5Ctext%20%7B%20value%20of%20feature%20%7D%20j%20%5Ctext%20%7B%20in%20%7D%20i%5E%7B%5Ctext%20%7Bth%20%7D%7D%20%5Ctext%20%7B%20training%20example%20%7D%24%24%EF%BC%8C%20%E4%B8%80%E8%88%AC%E6%88%91%E4%BB%AC%E8%AE%BE%E5%AE%9A%24%24x_%7B0%7D%20%3D%201%0A)

%7D%2C%20y%5E%7B(1)%7D%5Cright)%2C%20%5Cldots%2C%5Cleft(x%5E%7B(m)%7D%2C%20y%5E%7B(m)%7D%5Cright)%24%24%3B%20%24%24n%24%24%20features%0A#card=math&code=m%24%24%20examples%20%24%24%5Cleft%28x%5E%7B%281%29%7D%2C%20y%5E%7B%281%29%7D%5Cright%29%2C%20%5Cldots%2C%5Cleft%28x%5E%7B%28m%29%7D%2C%20y%5E%7B%28m%29%7D%5Cright%29%24%24%3B%20%24%24n%24%24%20features%0A)

{x^{(i)}}=\left[\begin{array}{c}
x{0}^{(i)} \
x{1}^{(i)} \
x{2}^{(i)} \
\vdots \
x{n}^{(i)}
\end{array}\right] \in \mathbb{R}^{n+1}

X=\left[\begin{array}{c}
(x{T} \
(x{T} \
(x{T}\
\vdots \
(x{T}
\end{array}\right]=\left[\begin{array}{ccccc}
x{0}^{(1)} & x{1}^{(1)} & x{2}^{(1)} & \cdots & x{n}^{(1)} \
x{0}^{(2)} & x{1}^{(2)} & x{2}^{(2)} & \cdots & x{n}^{(2)} \
x{0}^{(3)} & x{1}^{(3)} & x{2}^{(3)} & \cdots & x{n}^{(3)} \
\vdots & \vdots & \vdots & \ddots & \vdots \
x{0}^{(m)} & x{1}^{(m)} & x{2}^{(m)} & \cdots & x{n}^{(m)}
\end{array}\right] \in \mathbb{R}^{n+1}

%24%24%E7%9A%84%E7%9F%A9%E9%98%B5%EF%BC%8C%24%24Y%24%24%E6%98%AF%E4%B8%80%E4%B8%AA%24%24m-dimension%24%24%E7%9A%84%E5%90%91%E9%87%8F%0A%0A#card=math&code=%0A%3E%20%24%24X%24%24%20%E6%98%AF%E4%B8%80%E4%B8%AA%24%24m%20%C3%97%20%28n%20%2B%201%29%24%24%E7%9A%84%E7%9F%A9%E9%98%B5%EF%BC%8C%24%24Y%24%24%E6%98%AF%E4%B8%80%E4%B8%AA%24%24m-dimension%24%24%E7%9A%84%E5%90%91%E9%87%8F%0A%0A)

y=\left[\begin{array}{c}
y^{(1)} \
y^{(2)} \
y^{(3)} \
\vdots \
y^{(m)}
\end{array}\right]

\theta=\left(X^{T} X\right)^{-1} X^{T} y

-dimension%24%24%E7%9A%84%E5%90%91%E9%87%8F%0A%0Aexample%3A%0A#card=math&code=%0A%3E%E8%AE%A1%E7%AE%97%E7%9A%84%E7%BB%93%E6%9E%9C%E6%98%AF%E4%B8%80%E4%B8%AA%24%24%28n%20%2B%201%29-dimension%24%24%E7%9A%84%E5%90%91%E9%87%8F%0A%0A%2Aexample%3A%2A%0A)

\begin{array}{c|c|c|c|c|c}
& \text { Size (feet }^{2} \text { ) } & \begin{array}{c}
\text { Number of } \
\text { bedrooms }
\end{array} & \begin{array}{c}
\text { Number of } \
\text { floors }
\end{array} & \begin{array}{c}
\text { Age of home } \
\text { (years) }
\end{array} & \text { Price ($1000) } \
x{0} & x{1} & x{2} & x{3} & x_{4} & y \
\hline 1 & 2104 & 5 & 1 & 45 & 460 \
1 & 1416 & 3 & 2 & 40 & 232 \
1 & 1534 & 3 & 2 & 30 & 315 \
1 & 852 & 2 & 1 & 36 & 178
\end{array}

X=\left[\begin{array}{ccccc}
1 & 2104 & 5 & 1 & 45 \
1 & 1416 & 3 & 2 & 40 \
1 & 1534 & 3 & 2 & 30 \
1 & 852 & 2 & 1 & 36
\end{array}\right] \ y=\left[\begin{array}{l}
460 \
232 \
315 \
178
\end{array}\right]

%0A%0A%23%23%20Normal%20Equation%20Noninevitability(%E4%B8%8D%E5%8F%AF%E9%80%86%E6%80%A7)%0A%0AWhen%20implementing%20the%20normal%20equation%20in%20octave%20we%20want%20to%20use%20the%20’pinv’%20function%20rather%20than%20’inv.’%20The%20’pinv’%20function%20will%20give%20you%20a%20value%20of%20%24%5Ctheta%24%20even%20if%20%24X%5E%7BT%7D%20X%24%20is%20not%20invertible.%0AIf%20%24X%5E%7BT%7D%20X%24%20is%20noninvertible%2C%20the%20common%20causes%20might%20be%20having%20%3A%0A%0A-%20Redundant%20features%2C%20where%20two%20features%20are%20very%20closely%20related%20(i.e.%20they%20are%20linearly%20dependent)%0A-%20Too%20many%20features%20(e.g.%20%24m%20%5Cleq%20n)%24.%20In%20this%20case%2C%20delete%20some%20features%20or%20use%20%22regularization%22%20(to%20be%20explained%20in%20a%20later%20lesson).%0ASolutions%20to%20the%20above%20problems%20include%20deleting%20a%20feature%20that%20is%20linearly%20dependent%20with%20another%20or%20deleting%20one%20or%20more%20features%20when%20there%20are%20too%20many%20features.%0A%0A%23%23%20%E6%8E%A8%E5%AF%BCNormal%20Equation%0A%0A%23%23%23%20%E5%89%8D%E6%8F%90%0A%0A#card=math&code=%0A%23%23%23%20Gradient%20Descent%20or%20Normal%20Equation%20%3F%0A%0A%21%5Bimage-20210416170917240%5D%28https%3A%2F%2Fraw.githubusercontent.com%2FRainGivingU%2FPicture-warehouse%2Fmaster%2Fimg%2F20210416170917.png%29%0A%0A%23%23%20Normal%20Equation%20Noninevitability%28%E4%B8%8D%E5%8F%AF%E9%80%86%E6%80%A7%29%0A%0AWhen%20implementing%20the%20normal%20equation%20in%20octave%20we%20want%20to%20use%20the%20%27pinv%27%20function%20rather%20than%20%27inv.%27%20The%20%27pinv%27%20function%20will%20give%20you%20a%20value%20of%20%24%5Ctheta%24%20even%20if%20%24X%5E%7BT%7D%20X%24%20is%20not%20invertible.%0AIf%20%24X%5E%7BT%7D%20X%24%20is%20noninvertible%2C%20the%20common%20causes%20might%20be%20having%20%3A%0A%0A-%20Redundant%20features%2C%20where%20two%20features%20are%20very%20closely%20related%20%28i.e.%20they%20are%20linearly%20dependent%29%0A-%20Too%20many%20features%20%28e.g.%20%24m%20%5Cleq%20n%29%24.%20In%20this%20case%2C%20delete%20some%20features%20or%20use%20%22regularization%22%20%28to%20be%20explained%20in%20a%20later%20lesson%29.%0ASolutions%20to%20the%20above%20problems%20include%20deleting%20a%20feature%20that%20is%20linearly%20dependent%20with%20another%20or%20deleting%20one%20or%20more%20features%20when%20there%20are%20too%20many%20features.%0A%0A%23%23%20%E6%8E%A8%E5%AF%BCNormal%20Equation%0A%0A%23%23%23%20%E5%89%8D%E6%8F%90%0A%0A)

{x^{(i)}}=\left[\begin{array}{c}
x{0}^{(i)} \
x{1}^{(i)} \
x{2}^{(i)} \
\vdots \
x{n}^{(i)}
\end{array}\right] \in \mathbb{R}^{n+1}\tag{1}\

X=\left[\begin{array}{c}
(x{T} \
(x{T} \
(x{T}\
\vdots \
(x{T}
\end{array}\right]=\left[\begin{array}{ccccc}
x{0}^{(1)} & x{1}^{(1)} & x{2}^{(1)} & \cdots & x{n}^{(1)} \
x{0}^{(2)} & x{1}^{(2)} & x{2}^{(2)} & \cdots & x{n}^{(2)} \
x{0}^{(3)} & x{1}^{(3)} & x{2}^{(3)} & \cdots & x{n}^{(3)} \
\vdots & \vdots & \vdots & \ddots & \vdots \
x{0}^{(m)} & x{1}^{(m)} & x{2}^{(m)} & \cdots & x{n}^{(m)}
\end{array}\right] \in \mathbb{R}^{n+1}\tag{2}\

y=\left[\begin{array}{c}
y^{(1)} \
y^{(2)} \
y^{(3)} \
\vdots \
y^{(m)}
\end{array}\right]\tag{3}

\theta=\left[\begin{array}{c}
\theta{0} \
\theta{1} \
\theta{2} \
\vdots \
\theta{n}
\end{array}\right]\tag{4}

h_{\theta}\left(\boldsymbol{x}{(i)}\right)^{T} \theta\tag{5}

J\left(\theta{0}, \theta{1}, \theta{2}, \ldots, \theta{n}\right)=\frac{1}{2 m} \sum_{i=1}{(i)}\right)-y{2}\tag{6}

\begin{array}{c}
\frac{\partial}{\partial \theta{0}} {J}\left(\theta{0}, \theta{1}, \theta{2}, \ldots, \theta{n}\right)=\frac{1}{m} \sum{i=1}{(i)}\right)-y^{(i)}\right) x{0}^{(i)}=0 \
\frac{\partial}{\partial \theta{1}} {J}\left(\theta{0}, \theta{1}, \theta{2}, \ldots, \theta{n}\right)=\frac{1}{m} \sum{i=1}{(i)}\right)-y^{(i)}\right) x{1}^{(i)}=0 \
\frac{\partial}{\partial \theta{2}} {J}\left(\theta{0}, \theta{1}, \theta{2}, \ldots, \theta{n}\right)=\frac{1}{m} \sum{i=1}{(i)}\right)-y^{(i)}\right) x{2}^{(i)}=0 \
\vdots \
\frac{\partial}{\partial \theta{n}} {J}\left(\theta{0}, \theta{1}, \theta{2}, \ldots, \theta{n}\right)=\frac{1}{m} \sum{i=1}{(i)}\right)-y^{(i)}\right) x{n}^{(i)}=0
\end{array}\tag{7}

%7D%5Cright)-y%5E%7B(i)%7D%5Cright)%20x%7B0%7D%5E%7B(i)%7D%3D0%20%5C%5C#card=math&code=%0A%0A%E6%88%91%E4%BB%AC%E6%8C%91%E4%B8%80%E4%B8%AA%24%24%5Ctheta%7B0%7D%24%24%E5%AF%B9%E5%BA%94%E7%9A%84%E7%AD%89%E5%BC%8F%E8%BF%9B%E8%A1%8C%E5%88%86%E6%9E%90%24%24%5Cfrac%7B1%7D%7Bm%7D%20%5Csum%7Bi%3D1%7D%5E%7Bm%7D%5Cleft%28h%7B%5Ctheta%7D%5Cleft%28%5Cboldsymbol%7Bx%7D%5E%7B%28i%29%7D%5Cright%29-y%5E%7B%28i%29%7D%5Cright%29%20x_%7B0%7D%5E%7B%28i%29%7D%3D0%20%5C%5C)

转化为矩阵运算

%7D%20%26%20x%7B0%7D%5E%7B(2)%7D%20%26%20x%7B0%7D%5E%7B(3)%7D%20%26%20%5Ccdots%20%26%20x%7B0%7D%5E%7B(m)%7D%0A%5Cend%7Barray%7D%5Cright%5D%5Cleft%5B%5Cbegin%7Barray%7D%7Bc%7D%0Ah%7B%5Ctheta%7D%5Cleft(%5Cboldsymbol%7Bx%7D%5E%7B(1)%7D%5Cright)-y%5E%7B(1)%7D%20%5C%5C%0Ah%7B%5Ctheta%7D%5Cleft(%5Cboldsymbol%7Bx%7D%5E%7B(2)%7D%5Cright)-y%5E%7B(2)%7D%20%5C%5C%0Ah%7B%5Ctheta%7D%5Cleft(%5Cboldsymbol%7Bx%7D%5E%7B(3)%7D%5Cright)-y%5E%7B(3)%7D%20%5C%5C%0A%5Cvdots%20%5C%5C%0Ah%7B%5Ctheta%7D%5Cleft(%5Cboldsymbol%7Bx%7D%5E%7B(%5Cboldsymbol%7Bm%7D)%7D%5Cright)-y%5E%7B(m)%7D%0A%5Cend%7Barray%7D%5Cright%5D%3D0%5Ctag%7B8%7D%0A#card=math&code=%5Cfrac%7B1%7D%7Bm%7D%5Cleft%5B%5Cbegin%7Barray%7D%7Blllll%7D%0Ax%7B0%7D%5E%7B%281%29%7D%20%26%20x%7B0%7D%5E%7B%282%29%7D%20%26%20x%7B0%7D%5E%7B%283%29%7D%20%26%20%5Ccdots%20%26%20x%7B0%7D%5E%7B%28m%29%7D%0A%5Cend%7Barray%7D%5Cright%5D%5Cleft%5B%5Cbegin%7Barray%7D%7Bc%7D%0Ah%7B%5Ctheta%7D%5Cleft%28%5Cboldsymbol%7Bx%7D%5E%7B%281%29%7D%5Cright%29-y%5E%7B%281%29%7D%20%5C%5C%0Ah%7B%5Ctheta%7D%5Cleft%28%5Cboldsymbol%7Bx%7D%5E%7B%282%29%7D%5Cright%29-y%5E%7B%282%29%7D%20%5C%5C%0Ah%7B%5Ctheta%7D%5Cleft%28%5Cboldsymbol%7Bx%7D%5E%7B%283%29%7D%5Cright%29-y%5E%7B%283%29%7D%20%5C%5C%0A%5Cvdots%20%5C%5C%0Ah_%7B%5Ctheta%7D%5Cleft%28%5Cboldsymbol%7Bx%7D%5E%7B%28%5Cboldsymbol%7Bm%7D%29%7D%5Cright%29-y%5E%7B%28m%29%7D%0A%5Cend%7Barray%7D%5Cright%5D%3D0%5Ctag%7B8%7D%0A)

拆分第二个矩阵，顺便消去\frac{1}{m}

%7D%20%26%20x%7B0%7D%5E%7B(2)%7D%20%26%20x%7B0%7D%5E%7B(3)%7D%20%26%20%5Ccdots%20%26%20x%7B0%7D%5E%7B(m)%7D%0A%5Cend%7Barray%7D%5Cright%5D%5Cleft(%5Cleft%5B%5Cbegin%7Barray%7D%7Bc%7D%0Ah%7B%5Ctheta%7D%5Cleft(%5Cboldsymbol%7Bx%7D%5E%7B(1)%7D%5Cright)%20%5C%5C%0Ah%7B%5Ctheta%7D%5Cleft(%5Cboldsymbol%7Bx%7D%5E%7B(2)%7D%5Cright)%20%5C%5C%0Ah%7B%5Ctheta%7D%5Cleft(%5Cboldsymbol%7Bx%7D%5E%7B(3)%7D%5Cright)%20%5C%5C%0A%5Cvdots%20%5C%5C%0Ah%7B%5Ctheta%7D%5Cleft(%5Cboldsymbol%7Bx%7D%5E%7B(%5Cboldsymbol%7Bm%7D)%7D%5Cright)%0A%5Cend%7Barray%7D%5Cright%5D-%5Cleft%5B%5Cbegin%7Barray%7D%7Bc%7D%0Ay%5E%7B(1)%7D%20%5C%5C%0Ay%5E%7B(2)%7D%20%5C%5C%0Ay%5E%7B(3)%7D%20%5C%5C%0A%5Cvdots%20%5C%5C%0Ay%5E%7B(m)%7D%0A%5Cend%7Barray%7D%5Cright%5D%5Cright)%3D0%5Ctag%7B9%7D%0A#card=math&code=%5Cleft%5B%5Cbegin%7Barray%7D%7Blllll%7D%0Ax%7B0%7D%5E%7B%281%29%7D%20%26%20x%7B0%7D%5E%7B%282%29%7D%20%26%20x%7B0%7D%5E%7B%283%29%7D%20%26%20%5Ccdots%20%26%20x%7B0%7D%5E%7B%28m%29%7D%0A%5Cend%7Barray%7D%5Cright%5D%5Cleft%28%5Cleft%5B%5Cbegin%7Barray%7D%7Bc%7D%0Ah%7B%5Ctheta%7D%5Cleft%28%5Cboldsymbol%7Bx%7D%5E%7B%281%29%7D%5Cright%29%20%5C%5C%0Ah%7B%5Ctheta%7D%5Cleft%28%5Cboldsymbol%7Bx%7D%5E%7B%282%29%7D%5Cright%29%20%5C%5C%0Ah%7B%5Ctheta%7D%5Cleft%28%5Cboldsymbol%7Bx%7D%5E%7B%283%29%7D%5Cright%29%20%5C%5C%0A%5Cvdots%20%5C%5C%0Ah_%7B%5Ctheta%7D%5Cleft%28%5Cboldsymbol%7Bx%7D%5E%7B%28%5Cboldsymbol%7Bm%7D%29%7D%5Cright%29%0A%5Cend%7Barray%7D%5Cright%5D-%5Cleft%5B%5Cbegin%7Barray%7D%7Bc%7D%0Ay%5E%7B%281%29%7D%20%5C%5C%0Ay%5E%7B%282%29%7D%20%5C%5C%0Ay%5E%7B%283%29%7D%20%5C%5C%0A%5Cvdots%20%5C%5C%0Ay%5E%7B%28m%29%7D%0A%5Cend%7Barray%7D%5Cright%5D%5Cright%29%3D0%5Ctag%7B9%7D%0A)

代入(3)，(5)

%7D%20%26%20x%7B0%7D%5E%7B(2)%7D%20%26%20x%7B0%7D%5E%7B(3)%7D%20%26%20%5Ccdots%20%26%20x%7B0%7D%5E%7B(m)%7D%0A%5Cend%7Barray%7D%5Cright%5D%5Cleft(%5Cleft%5B%5Cbegin%7Barray%7D%7Bc%7D%0A%5Cleft(%5Cboldsymbol%7Bx%7D%5E%7B(1)%7D%5Cright)%5E%7BT%7D%20%5C%5C%0A%5Cleft(%5Cboldsymbol%7Bx%7D%5E%7B(2)%7D%5Cright)%5E%7BT%7D%20%5C%5C%0A%5Cleft(%5Cboldsymbol%7Bx%7D%5E%7B(3)%7D%5Cright)%5E%7BT%7D%20%5C%5C%0A%5Cvdots%20%5C%5C%0A%5Cleft(%5Cboldsymbol%7Bx%7D%5E%7B(%5Cboldsymbol%7Bm%7D)%7D%5Cright)%5E%7BT%7D%0A%5Cend%7Barray%7D%5Cright%5D%20%5Ctheta-%5Cmathrm%7By%7D%5Cright)%3D0%5Ctag%7B10%7D%0A#card=math&code=%5Cleft%5B%5Cbegin%7Barray%7D%7Blllll%7D%0Ax%7B0%7D%5E%7B%281%29%7D%20%26%20x%7B0%7D%5E%7B%282%29%7D%20%26%20x%7B0%7D%5E%7B%283%29%7D%20%26%20%5Ccdots%20%26%20x_%7B0%7D%5E%7B%28m%29%7D%0A%5Cend%7Barray%7D%5Cright%5D%5Cleft%28%5Cleft%5B%5Cbegin%7Barray%7D%7Bc%7D%0A%5Cleft%28%5Cboldsymbol%7Bx%7D%5E%7B%281%29%7D%5Cright%29%5E%7BT%7D%20%5C%5C%0A%5Cleft%28%5Cboldsymbol%7Bx%7D%5E%7B%282%29%7D%5Cright%29%5E%7BT%7D%20%5C%5C%0A%5Cleft%28%5Cboldsymbol%7Bx%7D%5E%7B%283%29%7D%5Cright%29%5E%7BT%7D%20%5C%5C%0A%5Cvdots%20%5C%5C%0A%5Cleft%28%5Cboldsymbol%7Bx%7D%5E%7B%28%5Cboldsymbol%7Bm%7D%29%7D%5Cright%29%5E%7BT%7D%0A%5Cend%7Barray%7D%5Cright%5D%20%5Ctheta-%5Cmathrm%7By%7D%5Cright%29%3D0%5Ctag%7B10%7D%0A)

代入(2)

%7D%20%26%20x%7B0%7D%5E%7B(2)%7D%20%26%20x%7B0%7D%5E%7B(3)%7D%20%26%20%5Ccdots%20%26%20x%7B0%7D%5E%7B(m)%7D%0A%5Cend%7Barray%7D%5Cright%5D(X%20%5Ctheta-%5Cmathrm%7By%7D)%3D0%5Ctag%7B11%7D%0A#card=math&code=%5Cleft%5B%5Cbegin%7Barray%7D%7Blllll%7D%0Ax%7B0%7D%5E%7B%281%29%7D%20%26%20x%7B0%7D%5E%7B%282%29%7D%20%26%20x%7B0%7D%5E%7B%283%29%7D%20%26%20%5Ccdots%20%26%20x_%7B0%7D%5E%7B%28m%29%7D%0A%5Cend%7Barray%7D%5Cright%5D%28X%20%5Ctheta-%5Cmathrm%7By%7D%29%3D0%5Ctag%7B11%7D%0A)

于是

%3D%5Cleft%5B%5Cbegin%7Barray%7D%7Blllll%7D%0Ax%7B0%7D%5E%7B(1)%7D%20%26%20x%7B0%7D%5E%7B(2)%7D%20%26%20x%7B0%7D%5E%7B(3)%7D%20%26%20%5Ccdots%20%26%20x%7B0%7D%5E%7B(m)%7D%0A%5Cend%7Barray%7D%5Cright%5D(X%20%5Ctheta-%5Cmathrm%7By%7D)%3D0%5C%5C%0A%26%5Cfrac%7B%5Cpartial%7D%7B%5Cpartial%20%5Ctheta%7B1%7D%7D%7BJ%7D%5Cleft(%5Ctheta%7B0%7D%2C%20%5Ctheta%7B1%7D%2C%20%5Ctheta%7B2%7D%2C%20%5Cldots%2C%20%5Ctheta%7Bn%7D%5Cright)%3D%5Cleft%5B%5Cbegin%7Barray%7D%7Blllll%7D%0Ax%7B1%7D%5E%7B(1)%7D%20%26%20x%7B1%7D%5E%7B(2)%7D%20%26%20x%7B1%7D%5E%7B(3)%7D%20%26%20%5Ccdots%20%26%20x%7B1%7D%5E%7B(m)%7D%0A%5Cend%7Barray%7D%5Cright%5D(X%20%5Ctheta-%5Cmathrm%7By%7D)%3D0%5C%5C%0A%26%5Cfrac%7B%5Cpartial%7D%7B%5Cpartial%20%5Ctheta%7B2%7D%7D%20%7BJ%7D%5Cleft(%5Ctheta%7B0%7D%2C%20%5Ctheta%7B1%7D%2C%20%5Ctheta%7B2%7D%2C%20%5Cldots%2C%20%5Ctheta%7Bn%7D%5Cright)%3D%5Cleft%5B%5Cbegin%7Barray%7D%7Blllll%7D%0Ax%7B2%7D%5E%7B(1)%7D%20%26%20x%7B2%7D%5E%7B(2)%7D%20%26%20x%7B2%7D%5E%7B(3)%7D%20%26%20%5Ccdots%20%26%20x%7B2%7D%5E%7B(m)%7D%0A%5Cend%7Barray%7D%5Cright%5D(X%20%5Ctheta-%5Cmathrm%7By%7D)%3D0%5C%5C%0A%26%5Cfrac%7B%5Cpartial%7D%7B%5Cpartial%20%5Ctheta%7Bn%7D%7D%20%7BJ%7D%5Cleft(%5Ctheta%7B0%7D%2C%20%5Ctheta%7B1%7D%2C%20%5Ctheta%7B2%7D%2C%20%5Cldots%2C%20%5Ctheta%7Bn%7D%5Cright)%3D%5Cleft%5Bx%7Bn%7D%5E%7B(1)%7D%20%5Cquad%20x%7Bn%7D%5E%7B(2)%7D%20%5Cquad%20x%7Bn%7D%5E%7B(3)%7D%20%5Cquad%20%5Cldots%20%5Cquad%20x%7Bn%7D%5E%7B(m)%7D%5Cright%5D(X%20%5Ctheta-%5Cmathrm%7By%7D)%3D0%0A%5Cend%7Baligned%7D%5Ctag%7B12%7D%0A#card=math&code=%5Cbegin%7Baligned%7D%0A%26%5Cfrac%7B%5Cpartial%7D%7B%5Cpartial%20%5Ctheta%7B0%7D%7D%20%7BJ%7D%5Cleft%28%5Ctheta%7B0%7D%2C%20%5Ctheta%7B1%7D%2C%20%5Ctheta%7B2%7D%2C%20%5Cldots%2C%20%5Ctheta%7Bn%7D%5Cright%29%3D%5Cleft%5B%5Cbegin%7Barray%7D%7Blllll%7D%0Ax%7B0%7D%5E%7B%281%29%7D%20%26%20x%7B0%7D%5E%7B%282%29%7D%20%26%20x%7B0%7D%5E%7B%283%29%7D%20%26%20%5Ccdots%20%26%20x%7B0%7D%5E%7B%28m%29%7D%0A%5Cend%7Barray%7D%5Cright%5D%28X%20%5Ctheta-%5Cmathrm%7By%7D%29%3D0%5C%5C%0A%26%5Cfrac%7B%5Cpartial%7D%7B%5Cpartial%20%5Ctheta%7B1%7D%7D%7BJ%7D%5Cleft%28%5Ctheta%7B0%7D%2C%20%5Ctheta%7B1%7D%2C%20%5Ctheta%7B2%7D%2C%20%5Cldots%2C%20%5Ctheta%7Bn%7D%5Cright%29%3D%5Cleft%5B%5Cbegin%7Barray%7D%7Blllll%7D%0Ax%7B1%7D%5E%7B%281%29%7D%20%26%20x%7B1%7D%5E%7B%282%29%7D%20%26%20x%7B1%7D%5E%7B%283%29%7D%20%26%20%5Ccdots%20%26%20x%7B1%7D%5E%7B%28m%29%7D%0A%5Cend%7Barray%7D%5Cright%5D%28X%20%5Ctheta-%5Cmathrm%7By%7D%29%3D0%5C%5C%0A%26%5Cfrac%7B%5Cpartial%7D%7B%5Cpartial%20%5Ctheta%7B2%7D%7D%20%7BJ%7D%5Cleft%28%5Ctheta%7B0%7D%2C%20%5Ctheta%7B1%7D%2C%20%5Ctheta%7B2%7D%2C%20%5Cldots%2C%20%5Ctheta%7Bn%7D%5Cright%29%3D%5Cleft%5B%5Cbegin%7Barray%7D%7Blllll%7D%0Ax%7B2%7D%5E%7B%281%29%7D%20%26%20x%7B2%7D%5E%7B%282%29%7D%20%26%20x%7B2%7D%5E%7B%283%29%7D%20%26%20%5Ccdots%20%26%20x%7B2%7D%5E%7B%28m%29%7D%0A%5Cend%7Barray%7D%5Cright%5D%28X%20%5Ctheta-%5Cmathrm%7By%7D%29%3D0%5C%5C%0A%26%5Cfrac%7B%5Cpartial%7D%7B%5Cpartial%20%5Ctheta%7Bn%7D%7D%20%7BJ%7D%5Cleft%28%5Ctheta%7B0%7D%2C%20%5Ctheta%7B1%7D%2C%20%5Ctheta%7B2%7D%2C%20%5Cldots%2C%20%5Ctheta%7Bn%7D%5Cright%29%3D%5Cleft%5Bx%7Bn%7D%5E%7B%281%29%7D%20%5Cquad%20x%7Bn%7D%5E%7B%282%29%7D%20%5Cquad%20x%7Bn%7D%5E%7B%283%29%7D%20%5Cquad%20%5Cldots%20%5Cquad%20x_%7Bn%7D%5E%7B%28m%29%7D%5Cright%5D%28X%20%5Ctheta-%5Cmathrm%7By%7D%29%3D0%0A%5Cend%7Baligned%7D%5Ctag%7B12%7D%0A)

将(12)中的多个式子通过矩阵结合起来

%7D%20%26%20x%7B0%7D%5E%7B(2)%7D%20%26%20x%7B0%7D%5E%7B(3)%7D%20%26%20%5Ccdots%20%26%20x%7B0%7D%5E%7B(m)%7D%20%5C%5C%0Ax%7B1%7D%5E%7B(1)%7D%20%26%20x%7B1%7D%5E%7B(2)%7D%20%26%20x%7B1%7D%5E%7B(3)%7D%20%26%20%5Ccdots%20%26%20x%7B1%7D%5E%7B(m)%7D%20%5C%5C%0Ax%7B2%7D%5E%7B(1)%7D%20%26%20x%7B2%7D%5E%7B(2)%7D%20%26%20x%7B2%7D%5E%7B(3)%7D%20%26%20%5Ccdots%20%26%20x%7B2%7D%5E%7B(m)%7D%20%5C%5C%0A%5Cvdots%20%26%20%5Cvdots%20%26%20%5Cvdots%20%26%20%5Cddots%20%26%20%5Cvdots%20%5C%5C%0Ax%7Bn%7D%5E%7B(1)%7D%20%26%20x%7Bn%7D%5E%7B(2)%7D%20%26%20x%7Bn%7D%5E%7B(3)%7D%20%26%20%5Ccdots%20%26%20x%7Bn%7D%5E%7B(m)%7D%0A%5Cend%7Barray%7D%5Cright%5D(X%20%5Ctheta-%5Cmathrm%7By%7D)%3D%5Cleft%5B%5Cbegin%7Barray%7D%7Bc%7D%0A0%20%5C%5C%0A0%20%5C%5C%0A0%20%5C%5C%0A%5Cvdots%20%5C%5C%0A0%0A%5Cend%7Barray%7D%5Cright%5D%5Ctag%7B13%7D%0A#card=math&code=%5Cleft%5B%5Cbegin%7Barray%7D%7Bccccc%7D%0Ax%7B0%7D%5E%7B%281%29%7D%20%26%20x%7B0%7D%5E%7B%282%29%7D%20%26%20x%7B0%7D%5E%7B%283%29%7D%20%26%20%5Ccdots%20%26%20x%7B0%7D%5E%7B%28m%29%7D%20%5C%5C%0Ax%7B1%7D%5E%7B%281%29%7D%20%26%20x%7B1%7D%5E%7B%282%29%7D%20%26%20x%7B1%7D%5E%7B%283%29%7D%20%26%20%5Ccdots%20%26%20x%7B1%7D%5E%7B%28m%29%7D%20%5C%5C%0Ax%7B2%7D%5E%7B%281%29%7D%20%26%20x%7B2%7D%5E%7B%282%29%7D%20%26%20x%7B2%7D%5E%7B%283%29%7D%20%26%20%5Ccdots%20%26%20x%7B2%7D%5E%7B%28m%29%7D%20%5C%5C%0A%5Cvdots%20%26%20%5Cvdots%20%26%20%5Cvdots%20%26%20%5Cddots%20%26%20%5Cvdots%20%5C%5C%0Ax%7Bn%7D%5E%7B%281%29%7D%20%26%20x%7Bn%7D%5E%7B%282%29%7D%20%26%20x%7Bn%7D%5E%7B%283%29%7D%20%26%20%5Ccdots%20%26%20x_%7Bn%7D%5E%7B%28m%29%7D%0A%5Cend%7Barray%7D%5Cright%5D%28X%20%5Ctheta-%5Cmathrm%7By%7D%29%3D%5Cleft%5B%5Cbegin%7Barray%7D%7Bc%7D%0A0%20%5C%5C%0A0%20%5C%5C%0A0%20%5C%5C%0A%5Cvdots%20%5C%5C%0A0%0A%5Cend%7Barray%7D%5Cright%5D%5Ctag%7B13%7D%0A)

又

%7D%20%26%20x%7B0%7D%5E%7B(2)%7D%20%26%20x%7B0%7D%5E%7B(3)%7D%20%26%20%5Ccdots%20%26%20x%7B0%7D%5E%7B(m)%7D%20%5C%5C%0Ax%7B1%7D%5E%7B(1)%7D%20%26%20x%7B1%7D%5E%7B(2)%7D%20%26%20x%7B1%7D%5E%7B(3)%7D%20%26%20%5Ccdots%20%26%20x%7B1%7D%5E%7B(m)%7D%20%5C%5C%0Ax%7B2%7D%5E%7B(1)%7D%20%26%20x%7B2%7D%5E%7B(2)%7D%20%26%20x%7B2%7D%5E%7B(3)%7D%20%26%20%5Ccdots%20%26%20x%7B2%7D%5E%7B(m)%7D%20%5C%5C%0A%5Cvdots%20%26%20%5Cvdots%20%26%20%5Cvdots%20%26%20%5Cddots%20%26%20%5Cvdots%20%5C%5C%0Ax%7Bn%7D%5E%7B(1)%7D%20%26%20x%7Bn%7D%5E%7B(2)%7D%20%26%20x%7Bn%7D%5E%7B(3)%7D%20%26%20%5Ccdots%20%26%20x%7Bn%7D%5E%7B(m)%7D%0A%5Cend%7Barray%7D%5Cright%5D%3D%5Cleft%5B%5Cbegin%7Barray%7D%7Bccccc%7D%0Ax%7B0%7D%5E%7B(1)%7D%20%26%20x%7B1%7D%5E%7B(1)%7D%20%26%20x%7B2%7D%5E%7B(1)%7D%20%26%20%5Ccdots%20%26%20x%7Bn%7D%5E%7B(1)%7D%20%5C%5C%0Ax%7B0%7D%5E%7B(2)%7D%20%26%20x%7B1%7D%5E%7B(2)%7D%20%26%20x%7B2%7D%5E%7B(2)%7D%20%26%20%5Ccdots%20%26%20x%7Bn%7D%5E%7B(2)%7D%20%5C%5C%0Ax%7B0%7D%5E%7B(3)%7D%20%26%20x%7B1%7D%5E%7B(3)%7D%20%26%20x%7B2%7D%5E%7B(3)%7D%20%26%20%5Ccdots%20%26%20x%7Bn%7D%5E%7B(3)%7D%20%5C%5C%0A%5Cvdots%20%26%20%5Cvdots%20%26%20%5Cvdots%20%26%20%5Cddots%20%26%20%5Cvdots%20%5C%5C%0Ax%7B0%7D%5E%7B(m)%7D%20%26%20x%7B1%7D%5E%7B(m)%7D%20%26%20x%7B2%7D%5E%7B(m)%7D%20%26%20%5Ccdots%20%26%20x%7Bn%7D%5E%7B(m)%7D%0A%5Cend%7Barray%7D%5Cright%5D%5E%7BT%7D%3DX%5E%7BT%7D%20%5Ctag%7B14%7D%0A#card=math&code=%5Cleft%5B%5Cbegin%7Barray%7D%7Bccccc%7D%0Ax%7B0%7D%5E%7B%281%29%7D%20%26%20x%7B0%7D%5E%7B%282%29%7D%20%26%20x%7B0%7D%5E%7B%283%29%7D%20%26%20%5Ccdots%20%26%20x%7B0%7D%5E%7B%28m%29%7D%20%5C%5C%0Ax%7B1%7D%5E%7B%281%29%7D%20%26%20x%7B1%7D%5E%7B%282%29%7D%20%26%20x%7B1%7D%5E%7B%283%29%7D%20%26%20%5Ccdots%20%26%20x%7B1%7D%5E%7B%28m%29%7D%20%5C%5C%0Ax%7B2%7D%5E%7B%281%29%7D%20%26%20x%7B2%7D%5E%7B%282%29%7D%20%26%20x%7B2%7D%5E%7B%283%29%7D%20%26%20%5Ccdots%20%26%20x%7B2%7D%5E%7B%28m%29%7D%20%5C%5C%0A%5Cvdots%20%26%20%5Cvdots%20%26%20%5Cvdots%20%26%20%5Cddots%20%26%20%5Cvdots%20%5C%5C%0Ax%7Bn%7D%5E%7B%281%29%7D%20%26%20x%7Bn%7D%5E%7B%282%29%7D%20%26%20x%7Bn%7D%5E%7B%283%29%7D%20%26%20%5Ccdots%20%26%20x%7Bn%7D%5E%7B%28m%29%7D%0A%5Cend%7Barray%7D%5Cright%5D%3D%5Cleft%5B%5Cbegin%7Barray%7D%7Bccccc%7D%0Ax%7B0%7D%5E%7B%281%29%7D%20%26%20x%7B1%7D%5E%7B%281%29%7D%20%26%20x%7B2%7D%5E%7B%281%29%7D%20%26%20%5Ccdots%20%26%20x%7Bn%7D%5E%7B%281%29%7D%20%5C%5C%0Ax%7B0%7D%5E%7B%282%29%7D%20%26%20x%7B1%7D%5E%7B%282%29%7D%20%26%20x%7B2%7D%5E%7B%282%29%7D%20%26%20%5Ccdots%20%26%20x%7Bn%7D%5E%7B%282%29%7D%20%5C%5C%0Ax%7B0%7D%5E%7B%283%29%7D%20%26%20x%7B1%7D%5E%7B%283%29%7D%20%26%20x%7B2%7D%5E%7B%283%29%7D%20%26%20%5Ccdots%20%26%20x%7Bn%7D%5E%7B%283%29%7D%20%5C%5C%0A%5Cvdots%20%26%20%5Cvdots%20%26%20%5Cvdots%20%26%20%5Cddots%20%26%20%5Cvdots%20%5C%5C%0Ax%7B0%7D%5E%7B%28m%29%7D%20%26%20x%7B1%7D%5E%7B%28m%29%7D%20%26%20x%7B2%7D%5E%7B%28m%29%7D%20%26%20%5Ccdots%20%26%20x_%7Bn%7D%5E%7B%28m%29%7D%0A%5Cend%7Barray%7D%5Cright%5D%5E%7BT%7D%3DX%5E%7BT%7D%20%5Ctag%7B14%7D%0A)

将(14)代入(13)

%3D%5Cmathbf%7B0%7D%5Ctag%7B15%7D%0A#card=math&code=X%5E%7BT%7D%28X%20%5Ctheta-%5Cmathrm%7By%7D%29%3D%5Cmathbf%7B0%7D%5Ctag%7B15%7D%0A)

下面对(15)进行简单变换，推导完毕

%5E%7B-1%7D%20X%5E%7BT%7D%20X%20%5Ctheta%3D%5Cleft(X%5E%7BT%7D%20X%5Cright)%5E%7B-1%7D%20X%5E%7BT%7D%20%5Cmathrm%7By%7D%5C%5C%0A%5Ctheta%3D%5Cleft(X%5E%7BT%7D%20X%5Cright)%5E%7B-1%7D%20X%5E%7BT%7D%20%5Cmathrm%7By%7D%5C%5C%0A%5Ctag%7B16%7D%0A#card=math&code=X%5E%7BT%7D%20X%20%5Ctheta-X%5E%7BT%7D%20%5Cmathrm%7By%7D%3D%5Cmathbf%7B0%7D%5C%5C%0AX%5E%7BT%7D%20X%20%5Ctheta%3DX%5E%7BT%7D%20%5Cmathrm%7By%7D%5C%5C%0A%5Cleft%28X%5E%7BT%7D%20X%5Cright%29%5E%7B-1%7D%20X%5E%7BT%7D%20X%20%5Ctheta%3D%5Cleft%28X%5E%7BT%7D%20X%5Cright%29%5E%7B-1%7D%20X%5E%7BT%7D%20%5Cmathrm%7By%7D%5C%5C%0A%5Ctheta%3D%5Cleft%28X%5E%7BT%7D%20X%5Cright%29%5E%7B-1%7D%20X%5E%7BT%7D%20%5Cmathrm%7By%7D%5C%5C%0A%5Ctag%7B16%7D%0A)

矩阵求导方法（待补）