GRU 的基本概念

LSTM 存在很多变体，其中门控循环单元（Gated Recurrent Unit, GRU）是最常见的一种，也是目前比较流行的一种。GRU是由 Cho 等人在2014年提出的，它对LSTM做了一些简化：

  1. GRU将LSTM原来的三个门简化成为两个：重置门 $$r_t$$（Reset Gate）和更新门 $$z_t$$ (Update Gate)。
  2. GRU不保留单元状态 $$c_t$$，只保留隐藏状态 $$h_t$$作为单元输出，这样就和传统RNN的结构保持一致。
  3. 重置门直接作用于前一时刻的隐藏状态 $$h_{t-1}$$。

GRU的前向计算

GRU的单元结构

图20-7展示了GRU的单元结构。

GRU基本原理与实现 - 图1

GRU单元的前向计算公式如下：

  1. 更新门
    zt = \sigma(h{t-1} \cdot W_z + x_t \cdot U_z) \tag{1}

  2. 重置门
    rt = \sigma(h{t-1} \cdot W_r + x_t \cdot U_r) \tag{2}

  3. 候选隐藏状态
    \tilde{h}t = \tanh((r_t \circ h{t-1}) \cdot W_h + x_t \cdot U_h) \tag{3}

  4. 隐藏状态
    h = (1 - zt) \circ h{t-1} + z_t \circ \tilde{h}_t \tag{4}

GRU的原理浅析

从上面的公式可以看出，GRU通过更新们和重置门控制长期状态的遗忘和保留，以及当前输入信息的选择。更新门和重置门通过sigmoid函数，将输入信息映射到[0,1]区间，实现门控功能。

首先，上一时刻的状态h_{t-1}通过重置门，加上当前时刻输入信息，共同构成当前时刻的即时状态\tilde{h}_t，并通过\tanh函数映射到[-1,1]区间。

然后，通过更新门实现遗忘和记忆两个部分。从隐藏状态的公式可以看出，通过z_t进行选择性的遗忘和记忆。(1-z_t)和z_t有联动关系，上一时刻信息遗忘的越多，当前信息记住的就越多，实现了LSTM中f_t和i_t的功能。

GRU的反向传播

学习了LSTM的反向传播的推导，GRU的推导就相对简单了。我们仍然以l层t时刻的GRU单元为例，推导反向传播过程。

同LSTM， 令：l层t时刻传入误差为\delta{t}l和上一层传入误差\delta{xt}^{l+1}之和，简写为\delta{t}。

令：

z{zt} = h{t-1} \cdot W_z + x_t \cdot U_z \tag{5}

z{rt} = h{t-1} \cdot W_r + x_t \cdot U_r \tag{6}

z_{\tilde{h}t} = (r_t \circ h{t-1}) \cdot W_h + x_t \cdot U_h \tag{7}

则：

\begin{aligned} \delta{z{zt}} = \frac{\partial{loss}}{\partial{ht}} \cdot \frac{\partial{h_t}}{\partial{z_t}} \cdot \frac{\partial{z_t}}{\partial{z{zt}}} \ = \delta_t \cdot (-diag[h{t-1}] + diag[\tilde{h}_t]) \cdot diag[z_t \circ (1-z_t)] \ = \delta_t \circ (\tilde{h}t - h{t-1}) \circ z_t \circ (1-z_t) \end{aligned} \tag{8}

\begin{aligned} \delta{z{\tilde{h}t}} = \frac{\partial{loss}}{\partial{ht}} \cdot \frac{\partial{h_t}}{\partial{\tilde{h}_t}} \cdot \frac{\partial{\tilde{h}t}}{\partial{z{\tilde{h}_t}}} \ = \delta_t \cdot diag[z_t] \cdot diag[1-(\tilde{h}_t)^2] \ &= \delta_t \circ z_t \circ (1-(\tilde{h}_t)^2) \end{aligned} \tag{9} \begin{aligned} \delta{z{rt}} = \frac{\partial{loss}}{\partial{\tilde{h}t}} \cdot \frac{\partial{\tilde{h}t}}{\partial{z{\tilde{h}t}}} \cdot \frac{\partial{z{\tilde{h}t}}}{\partial{r_t}} \cdot \frac{\partial{r_t}}{\partial{z{r_t}}} \ = \delta{z{\tilde{h}t}} \cdot Wh^T \cdot diag[h{t-1}] \cdot diag[rt \circ (1-r_t)] \ &= \delta{z{\tilde{h}t}} \cdot W_h^T \circ h{t-1} \circ r_t \circ (1-r_t) \end{aligned} \tag{10}

由此可求出，t时刻各个可学习参数的误差：

\begin{aligned} d{W{h,t}} = \frac{\partial{loss}}{\partial{z{\tilde{h}t}}} \cdot \frac{\partial{z{\tilde{h}t}}}{\partial{W_h}} = (r_t \circ h{t-1})^T \cdot \delta{z_{\tilde{h}t}} \end{aligned} \tag{11}

\begin{aligned} d{U{h,t}} = \frac{\partial{loss}}{\partial{z{\tilde{h}t}}} \cdot \frac{\partial{z{\tilde{h}t}}}{\partial{U_h}} = x_t^T \cdot \delta{z{\tilde{h}t}} \end{aligned} \tag{12}

\begin{aligned} d{W{r,t}} = \frac{\partial{loss}}{\partial{z{r_t}}} \cdot \frac{\partial{z{rt}}}{\partial{W_r}} = h{t-1}^T \cdot \delta{z{rt}} \end{aligned} \tag{13}

\begin{aligned} d{U{r,t}} = \frac{\partial{loss}}{\partial{z{r_t}}} \cdot \frac{\partial{z{rt}}}{\partial{U_r}} = x_t^T \cdot \delta{z_{rt}} \end{aligned} \tag{14}

\begin{aligned} d{W{z,t}} = \frac{\partial{loss}}{\partial{z{z_t}}} \cdot \frac{\partial{z{zt}}}{\partial{W_z}} = h{t-1}^T \cdot \delta{z{zt}} \end{aligned} \tag{15}

\begin{aligned} d{U{z,t}} = \frac{\partial{loss}}{\partial{z{z_t}}} \cdot \frac{\partial{z{zt}}}{\partial{U_z}} = x_t^T \cdot \delta{z_{zt}} \end{aligned} \tag{16}

可学习参数的最终误差为各个时刻误差之和，即：

d{W_h} = \sum{t=1}^{\tau} d{W{h,t}} = \sum{t=1}^{\tau} (r_t \circ h{t-1})^T \cdot \delta{z{\tilde{h}t}} \tag{17}

d{U_h} = \sum{t=1}^{\tau} d{U{h,t}} = \sum{t=1}^{\tau} x_t^T \cdot \delta{z_{\tilde{h}t}} \tag{18}

d{W_r} = \sum{t=1}^{\tau} d{W{r,t}} = \sum{t=1}^{\tau} h{t-1}^T \cdot \delta{z{rt}} \tag{19}

d{U_r} = \sum{t=1}^{\tau} d{U{r,t}} = \sum{t=1}^{\tau} x_t^T \cdot \delta{z_{rt}} \tag{20}

d{W_z} = \sum{t=1}^{\tau} d{W{z,t}} = \sum{t=1}^{\tau} h{t-1}^T \cdot \delta{z{zt}} \tag{21}

d{U_z} = \sum{t=1}^{\tau} d{U{z,t}} = \sum{t=1}^{\tau} x_t^T \cdot \delta{z_{zt}} \tag{22}

当前GRU cell分别向前一时刻（$t-1$）和下一层（$l-1$）传递误差，公式如下：

沿时间向前传递：

$$
\begin{aligned} \delta{h{t-1}} = \frac{\partial{loss}}{\partial{h{t-1}}} \ = \frac{\partial{loss}}{\partial{h_t}} \cdot \frac{\partial{h_t}}{\partial{h{t-1}}} + \frac{\partial{loss}}{\partial{z{\tilde{h}t}}} \cdot \frac{\partial{z{\tilde{h}t}}}{\partial{h{t-1}}} \ &+ \frac{\partial{loss}}{\partial{z{rt}}} \cdot \frac{\partial{z{rt}}}{\partial{h{t-1}}} + \frac{\partial{loss}}{\partial{z{zt}}} \cdot \frac{\partial{z{zt}}}{\partial{h{t-1}}} \ = \delta{t} \circ (1-zt) + \delta{z{\tilde{h}t}} \cdot W_h^T \circ r_t \ &+ \delta{z{rt}} \cdot W_r^T + \delta{z_{zt}} \cdot W_z^T \end{aligned} \tag{23}
$$

沿层次向下传递：

\begin{aligned} \delta{x_t} = \frac{\partial{loss}}{\partial{x_t}} = \frac{\partial{loss}}{\partial{z{\tilde{h}t}}} \cdot \frac{\partial{z{\tilde{h}t}}}{\partial{xt}} \ &+ \frac{\partial{loss}}{\partial{z{r_t}}} \cdot \frac{\partial{z{rt}}}{\partial{x_t}} + \frac{\partial{loss}}{\partial{z{zt}}} \cdot \frac{\partial{z{zt}}}{\partial{x_t}} \ = \delta{z{\tilde{h}t}} \cdot U_h^T + \delta{z{rt}} \cdot U_r^T + \delta{z_{zt}} \cdot U_z^T \end{aligned} \tag{24}

以上，GRU反向传播公式推导完毕。

代码实现

本节进行了GRU网络单元前向计算和反向传播的实现。为了统一和简单，测试用例依然是二进制减法。

初始化

本案例实现了没有bias的GRU单元，只需初始化输入维度和隐层维度。

  1. def __init__(self, input_size, hidden_size):
  2.         self.input_size = input_size
  3.         self.hidden_size = hidden_size

前向计算

  1. def forward(self, x, h_p, W, U):
  2.     self.get_params(W, U)
  3.     self.x = x
  5.     self.z = Sigmoid().forward(np.dot(h_p, self.wz) + np.dot(x, self.uz))
  6.     self.r = Sigmoid().forward(np.dot(h_p, self.wr) + np.dot(x, self.ur))
  7.     self.n = Tanh().forward(np.dot((self.r * h_p), self.wn) + np.dot(x, self.un))
  8.     self.h = (1 - self.z) * h_p + self.z * self.n
  10. def split_params(self, w, size):
  11.         s=[]
  12.         for i in range(3):
  13.             s.append(w[(i*size):((i+1)*size)])
  14.         return s[0], s[1], s[2]
  16. # Get shared parameters, and split them to fit 3 gates, in the order of z, r, \tilde{h} (n stands for \tilde{h} in code)
  17. def get_params(self, W, U):
  18.     self.wz, self.wr, self.wn = self.split_params(W, self.hidden_size)
  19.     self.uz, self.ur, self.un = self.split_params(U, self.input_size)

反向传播

  1. def backward(self, h_p, in_grad):
  2.     self.dzz = in_grad * (self.n - h_p) * self.z * (1 - self.z)
  3.     self.dzn = in_grad * self.z * (1 - self.n * self.n)
  4.     self.dzr = np.dot(self.dzn, self.wn.T) * h_p * self.r * (1 - self.r)
  6.     self.dwn = np.dot((self.r * h_p).T, self.dzn)
  7.     self.dun = np.dot(self.x.T, self.dzn)
  8.     self.dwr = np.dot(h_p.T, self.dzr)
  9.     self.dur = np.dot(self.x.T, self.dzr)
  10.     self.dwz = np.dot(h_p.T, self.dzz)
  11.     self.duz = np.dot(self.x.T, self.dzz)
  13.     self.merge_params()
  15.     # pass to previous time step
  16.     self.dh = in_grad * (1 - self.z) + np.dot(self.dzn, self.wn.T) * self.r + np.dot(self.dzr, self.wr.T) + np.dot(self.dzz, self.wz.T)
  17.     # pass to previous layer
  18.     self.dx = np.dot(self.dzn, self.un.T) + np.dot(self.dzr, self.ur.T) + np.dot(self.dzz, self.uz.T)

我们将所有拆分的参数merge到一起，便于更新梯度。

  1. def merge_params(self):
  2.     self.dW = np.concatenate((self.dwz, self.dwr, self.dwn), axis=0)
  3.     self.dU = np.concatenate((self.duz, self.dur, self.dun), axis=0)

最终结果

图20-8展示了训练过程，以及loss和accuracy的曲线变化。

GRU基本原理与实现 - 图2

该模型在验证集上可得100%的正确率。网络正确性得到验证。

  1.   x1: [1, 1, 1, 0]
  2. - x2: [1, 0, 0, 0]
  3. ------------------
  4. true: [0, 1, 1, 0]
  5. pred: [0, 1, 1, 0]
  6. 14 - 8 = 6
  7. ====================
  8.   x1: [1, 1, 0, 0]
  9. - x2: [0, 0, 0, 0]
  10. ------------------
  11. true: [1, 1, 0, 0]
  12. pred: [1, 1, 0, 0]
  13. 12 - 0 = 12
  14. ====================
  15.   x1: [1, 0, 1, 0]
  16. - x2: [0, 0, 0, 1]
  17. ------------------
  18. true: [1, 0, 0, 1]
  19. pred: [1, 0, 0, 1]
  20. 10 - 1 = 9

keras实现

  1. import os
  2. os.environ['KMP_DUPLICATE_LIB_OK']='True'
  4. import matplotlib.pyplot as plt
  6. from MiniFramework.DataReader_2_0 import *
  8. from keras.models import Sequential
  9. from keras.layers import GRU, Dense
  11. train_file = "../data/ch19.train_minus.npz"
  12. test_file = "../data/ch19.test_minus.npz"
  15. def load_data():
  16.     dataReader = DataReader_2_0(train_file, test_file)
  17.     dataReader.ReadData()
  18.     dataReader.Shuffle()
  19.     dataReader.GenerateValidationSet(k=10)
  20.     x_train, y_train = dataReader.XTrain, dataReader.YTrain
  21.     x_test, y_test = dataReader.XTest, dataReader.YTest
  22.     x_val, y_val = dataReader.XDev, dataReader.YDev
  23.     return x_train, y_train, x_test, y_test, x_val, y_val
  26. def build_model():
  27.     model = Sequential()
  28.     model.add(GRU(input_shape=(4,2),
  29.                         units=4))
  30.     model.add(Dense(4, activation='sigmoid'))
  31.     model.compile(optimizer='Adam',
  32.                   loss='binary_crossentropy',
  33.                   metrics=['accuracy'])
  34.     return model
  36. #画出训练过程中训练和验证的精度与损失
  37. def draw_train_history(history):
  38.     plt.figure(1)
  40.     # summarize history for accuracy
  41.     plt.subplot(211)
  42.     plt.plot(history.history['accuracy'])
  43.     plt.plot(history.history['val_accuracy'])
  44.     plt.title('model accuracy')
  45.     plt.ylabel('accuracy')
  46.     plt.xlabel('epoch')
  47.     plt.legend(['train', 'validation'])
  49.     # summarize history for loss
  50.     plt.subplot(212)
  51.     plt.plot(history.history['loss'])
  52.     plt.plot(history.history['val_loss'])
  53.     plt.title('model loss')
  54.     plt.ylabel('loss')
  55.     plt.xlabel('epoch')
  56.     plt.legend(['train', 'validation'])
  57.     plt.show()
  60. def test(x_test, y_test, model):
  61.     print("testing...")
  62.     count = x_test.shape[0]
  63.     result = model.predict(x_test)
  64.     r = np.random.randint(0, count, 10)
  65.     for i in range(10):
  66.         idx = r[i]
  67.         x1 = x_test[idx, :, 0]
  68.         x2 = x_test[idx, :, 1]
  69.         print("  x1:", reverse(x1))
  70.         print("- x2:", reverse(x2))
  71.         print("------------------")
  72.         print("true:", reverse(y_test[idx]))
  73.         print("pred:", reverse(result[idx]))
  74.         x1_dec = int("".join(map(str, reverse(x1))), 2)
  75.         x2_dec = int("".join(map(str, reverse(x2))), 2)
  76.         print("{0} - {1} = {2}".format(x1_dec, x2_dec, (x1_dec - x2_dec)))
  77.         print("====================")
  78.     # end for
  81. def reverse(a):
  82.     l = a.tolist()
  83.     l.reverse()
  84.     return l
  87. if __name__ == '__main__':
  88.     x_train, y_train, x_test, y_test, x_val, y_val = load_data()
  89.     print(x_train.shape)
  90.     print(y_train.shape)
  91.     print(x_test.shape)
  92.     print(x_val.shape)
  94.     model = build_model()
  95.     history = model.fit(x_train, y_train,
  96.                         epochs=200,
  97.                         batch_size=64,
  98.                         validation_data=(x_val, y_val))
  99.     print(model.summary())
  100.     draw_train_history(history)
  102.     loss, accuracy = model.evaluate(x_test, y_test)
  103.     print("test loss: {}, test accuracy: {}".format(loss, accuracy))
  105.     test(x_test, y_test, model)

模型输出

test loss: 0.6068302603328929, test accuracy: 0.623161792755127

损失以及准确率曲线

GRU基本原理与实现 - 图3

代码位置

原代码位置：ch20, Level2

个人代码：GRU_BinaryNumberMinus**