Implementation of Recurrent Neural Networks from Scratch

:label:sec_rnn_scratch

In this section we will implement an RNN from scratch for a character-level language model, according to our descriptions in :numref:sec_rnn. Such a model will be trained on H. G. Wells’ The Time Machine. As before, we start by reading the dataset first, which is introduced in :numref:sec_language_model.

```{.python .input} %matplotlib inline from d2l import mxnet as d2l import math from mxnet import autograd, gluon, np, npx npx.set_np()

  2. ```{.python .input}
  3. #@tab pytorch
  4. %matplotlib inline
  5. from d2l import torch as d2l
  6. import math
  7. import torch
  8. from torch import nn
  9. from torch.nn import functional as F

```{.python .input}

@tab tensorflow

%matplotlib inline from d2l import tensorflow as d2l import math import numpy as np import tensorflow as tf

  2. ```{.python .input}
  3. #@tab all
  4. batch_size, num_steps = 32, 35
  5. train_iter, vocab = d2l.load_data_time_machine(batch_size, num_steps)

```{.python .input}

@tab tensorflow

train_random_iter, vocab_random_iter = d2l.load_data_time_machine( batch_size, num_steps, use_random_iter=True)

  2. ## One-Hot Encoding
  4. Recall that each token is represented as a numerical index in `train_iter`.
  5. Feeding these indices directly to a neural network might make it hard to
  6. learn.
  7. We often represent each token as a more expressive feature vector.
  8. The easiest representation is called *one-hot encoding*,
  9. which is introduced
  10. in :numref:`subsec_classification-problem`.
  12. In a nutshell, we map each index to a different unit vector: assume that the number of different tokens in the vocabulary is $N$ (`len(vocab)`) and the token indices range from 0 to $N-1$.
  13. If the index of a token is the integer $i$, then we create a vector of all 0s with a length of $N$ and set the element at position $i$ to 1.
  14. This vector is the one-hot vector of the original token. The one-hot vectors with indices 0 and 2 are shown below.
  16. ```{.python .input}
  17. npx.one_hot(np.array([0, 2]), len(vocab))

```{.python .input}

@tab pytorch

F.one_hot(torch.tensor([0, 2]), len(vocab))

  2. ```{.python .input}
  3. #@tab tensorflow
  4. tf.one_hot(tf.constant([0, 2]), len(vocab))

The shape of the minibatch that we sample each time is (batch size, number of time steps). The one_hot function transforms such a minibatch into a three-dimensional tensor with the last dimension equals to the vocabulary size (len(vocab)). We often transpose the input so that we will obtain an output of shape (number of time steps, batch size, vocabulary size). This will allow us to more conveniently loop through the outermost dimension for updating hidden states of a minibatch, time step by time step.

```{.python .input} X = d2l.reshape(d2l.arange(10), (2, 5)) npx.one_hot(X.T, 28).shape

  2. ```{.python .input}
  3. #@tab pytorch
  4. X = d2l.reshape(d2l.arange(10), (2, 5))
  5. F.one_hot(X.T, 28).shape

```{.python .input}

@tab tensorflow

X = d2l.reshape(d2l.arange(10), (2, 5)) tf.one_hot(tf.transpose(X), 28).shape

  2. ## Initializing the Model Parameters
  4. Next, we initialize the model parameters for
  5. the RNN model.
  6. The number of hidden units `num_hiddens` is a tunable hyperparameter.
  7. When training language models,
  8. the inputs and outputs are from the same vocabulary.
  9. Hence, they have the same dimension,
  10. which is equal to the vocabulary size.
  12. ```{.python .input}
  13. def get_params(vocab_size, num_hiddens, device):
  14.     num_inputs = num_outputs = vocab_size
  16.     def normal(shape):
  17.         return np.random.normal(scale=0.01, size=shape, ctx=device)
  19.     # Hidden layer parameters
  20.     W_xh = normal((num_inputs, num_hiddens))
  21.     W_hh = normal((num_hiddens, num_hiddens))
  22.     b_h = d2l.zeros(num_hiddens, ctx=device)
  23.     # Output layer parameters
  24.     W_hq = normal((num_hiddens, num_outputs))
  25.     b_q = d2l.zeros(num_outputs, ctx=device)
  26.     # Attach gradients
  27.     params = [W_xh, W_hh, b_h, W_hq, b_q]
  28.     for param in params:
  29.         param.attach_grad()
  30.     return params

```{.python .input}

@tab pytorch

def get_params(vocab_size, num_hiddens, device): num_inputs = num_outputs = vocab_size

  1. def normal(shape):
  2.     return torch.randn(size=shape, device=device) * 0.01
  4. # Hidden layer parameters
  5. W_xh = normal((num_inputs, num_hiddens))
  6. W_hh = normal((num_hiddens, num_hiddens))
  7. b_h = d2l.zeros(num_hiddens, device=device)
  8. # Output layer parameters
  9. W_hq = normal((num_hiddens, num_outputs))
  10. b_q = d2l.zeros(num_outputs, device=device)
  11. # Attach gradients
  12. params = [W_xh, W_hh, b_h, W_hq, b_q]
  13. for param in params:
  14.     param.requires_grad_(True)
  15. return params
  2. ```{.python .input}
  3. #@tab tensorflow
  4. def get_params(vocab_size, num_hidden):
  5.     num_inputs = num_outputs = vocab_size
  7.     def normal(shape):
  8.         return d2l.normal(shape=shape,stddev=0.01,mean=0,dtype=tf.float32)
  10.     # Hidden layer parameters
  11.     W_xh = tf.Variable(normal((num_inputs, num_hiddens)), dtype=tf.float32)
  12.     W_hh = tf.Variable(normal((num_hiddens, num_hiddens)), dtype=tf.float32)
  13.     b_h = tf.Variable(d2l.zeros(num_hiddens), dtype=tf.float32)
  14.     # Output layer parameters
  15.     W_hq = tf.Variable(normal((num_hiddens, num_outputs)), dtype=tf.float32)
  16.     b_q = tf.Variable(d2l.zeros(num_outputs), dtype=tf.float32)
  17.     params = [W_xh, W_hh, b_h, W_hq, b_q]
  18.     return params

RNN Model

To define an RNN model, we first need an init_rnn_state function to return the hidden state at initialization. It returns a tensor filled with 0 and with a shape of (batch size, number of hidden units). Using tuples makes it easier to handle situations where the hidden state contains multiple variables, which we will encounter in later sections.

```{.python .input} def init_rnn_state(batch_size, num_hiddens, device): return (d2l.zeros((batch_size, num_hiddens), ctx=device), )

  2. ```{.python .input}
  3. #@tab pytorch
  4. def init_rnn_state(batch_size, num_hiddens, device):
  5.     return (d2l.zeros((batch_size, num_hiddens), device=device), )

```{.python .input}

@tab tensorflow

def init_rnn_state(batch_size, num_hiddens): return (d2l.zeros((batch_size, num_hiddens)), )

  2. The following `rnn` function defines how to compute the hidden state and output
  3. at a time step.
  4. Note that
  5. the RNN model
  6. loops through the outermost dimension of `inputs`
  7. so that it updates hidden states `H` of a minibatch,
  8. time step by time step.
  9. Besides,
  10. the activation function here uses the $\tanh$ function.
  11. As
  12. described in :numref:`sec_mlp`, the
  13. mean value of the $\tanh$ function is 0, when the elements are uniformly
  14. distributed over the real numbers.
  16. ```{.python .input}
  17. def rnn(inputs, state, params):
  18.     # Shape of `inputs`: (`num_steps`, `batch_size`, `vocab_size`)
  19.     W_xh, W_hh, b_h, W_hq, b_q = params
  20.     H, = state
  21.     outputs = []
  22.     # Shape of `X`: (`batch_size`, `vocab_size`)
  23.     for X in inputs:
  24.         H = np.tanh(np.dot(X, W_xh) + np.dot(H, W_hh) + b_h)
  25.         Y = np.dot(H, W_hq) + b_q
  26.         outputs.append(Y)
  27.     return np.concatenate(outputs, axis=0), (H,)

```{.python .input}

@tab pytorch

def rnn(inputs, state, params):

  1. # Here `inputs` shape: (`num_steps`, `batch_size`, `vocab_size`)
  2. W_xh, W_hh, b_h, W_hq, b_q = params
  3. H, = state
  4. outputs = []
  5. # Shape of `X`: (`batch_size`, `vocab_size`)
  6. for X in inputs:
  7.     H = torch.tanh(torch.mm(X, W_xh) + torch.mm(H, W_hh) + b_h)
  8.     Y = torch.mm(H, W_hq) + b_q
  9.     outputs.append(Y)
  10. return torch.cat(outputs, dim=0), (H,)
  2. ```{.python .input}
  3. #@tab tensorflow
  4. def rnn(inputs, state, params):
  5.     # Here `inputs` shape: (`num_steps`, `batch_size`, `vocab_size`)
  6.     W_xh, W_hh, b_h, W_hq, b_q = params
  7.     H, = state
  8.     outputs = []
  9.     # Shape of `X`: (`batch_size`, `vocab_size`)
  10.     for X in inputs:
  11.         X = tf.reshape(X,[-1,W_xh.shape[0]])
  12.         H = tf.tanh(tf.matmul(X, W_xh) + tf.matmul(H, W_hh) + b_h)
  13.         Y = tf.matmul(H, W_hq) + b_q
  14.         outputs.append(Y)
  15.     return d2l.concat(outputs, axis=0), (H,)

With all the needed functions being defined, next we create a class to wrap these functions and store parameters for an RNN model implemented from scratch.

```{.python .input} class RNNModelScratch: #@save “””An RNN Model implemented from scratch.””” def init(self, vocab_size, num_hiddens, device, get_params, init_state, forward_fn): self.vocab_size, self.num_hiddens = vocab_size, num_hiddens self.params = get_params(vocab_size, num_hiddens, device) self.init_state, self.forward_fn = init_state, forward_fn

  1. def __call__(self, X, state):
  2.     X = npx.one_hot(X.T, self.vocab_size)
  3.     return self.forward_fn(X, state, self.params)
  5. def begin_state(self, batch_size, ctx):
  6.     return self.init_state(batch_size, self.num_hiddens, ctx)
  2. ```{.python .input}
  3. #@tab pytorch
  4. class RNNModelScratch: #@save
  5.     """A RNN Model implemented from scratch."""
  6.     def __init__(self, vocab_size, num_hiddens, device,
  7.                  get_params, init_state, forward_fn):
  8.         self.vocab_size, self.num_hiddens = vocab_size, num_hiddens
  9.         self.params = get_params(vocab_size, num_hiddens, device)
  10.         self.init_state, self.forward_fn = init_state, forward_fn
  12.     def __call__(self, X, state):
  13.         X = F.one_hot(X.T, self.vocab_size).type(torch.float32)
  14.         return self.forward_fn(X, state, self.params)
  16.     def begin_state(self, batch_size, device):
  17.         return self.init_state(batch_size, self.num_hiddens, device)

```{.python .input}

@tab tensorflow

class RNNModelScratch: #@save “””A RNN Model implemented from scratch.””” def init(self, vocab_size, num_hiddens, init_state, forward_fn): self.vocab_size, self.num_hiddens = vocab_size, num_hiddens self.init_state, self.forward_fn = init_state, forward_fn

  1. def __call__(self, X, state, params):
  2.     X = tf.one_hot(tf.transpose(X), self.vocab_size)
  3.     X = tf.cast(X, tf.float32)
  4.     return self.forward_fn(X, state, params)
  6. def begin_state(self, batch_size):
  7.     return self.init_state(batch_size, self.num_hiddens)
  2. Let us check whether the outputs have the correct shapes, e.g., to ensure that the dimensionality of the hidden state remains unchanged.
  4. ```{.python .input}
  5. #@tab mxnet
  6. num_hiddens = 512
  7. model = RNNModelScratch(len(vocab), num_hiddens, d2l.try_gpu(), get_params,
  8.                         init_rnn_state, rnn)
  9. state = model.begin_state(X.shape[0], d2l.try_gpu())
  10. Y, new_state = model(X.as_in_context(d2l.try_gpu()), state)
  11. Y.shape, len(new_state), new_state[0].shape

```{.python .input}

@tab pytorch

num_hiddens = 512 model = RNNModelScratch(len(vocab), num_hiddens, d2l.try_gpu(), get_params, init_rnn_state, rnn) state = model.begin_state(X.shape[0], d2l.try_gpu()) Y, new_state = model(X.to(d2l.try_gpu()), state) Y.shape, len(new_state), new_state[0].shape

  2. ```{.python .input}
  3. #@tab tensorflow
  4. # defining tensorflow training strategy
  5. device_name = d2l.try_gpu()._device_name
  6. strategy = tf.distribute.OneDeviceStrategy(device_name)
  8. num_hiddens = 512
  9. with strategy.scope():
  10.     model = RNNModelScratch(len(vocab), num_hiddens, 
  11.                             init_rnn_state, rnn)
  12. state = model.begin_state(X.shape[0])
  13. params = get_params(len(vocab), num_hiddens)
  14. Y, new_state = model(X, state, params)
  15. Y.shape, len(new_state), new_state[0].shape

We can see that the output shape is (number of time steps $\times$ batch size, vocabulary size), while the hidden state shape remains the same, i.e., (batch size, number of hidden units).

Prediction

Let us first define the prediction function to generate new characters following the user-provided prefix, which is a string containing several characters. When looping through these beginning characters in prefix, we keep passing the hidden state to the next time step without generating any output. This is called the warm-up period, during which the model updates itself (e.g., update the hidden state) but does not make predictions. After the warm-up period, the hidden state is generally better than its initialized value at the beginning. So we generate the predicted characters and emit them.

``{.python .input} def predict_ch8(prefix, num_preds, model, vocab, device): #@save """Generate new characters following theprefix.""" state = model.begin_state(batch_size=1, ctx=device) outputs = [vocab[prefix[0]]] get_input = lambda: d2l.reshape( d2l.tensor([outputs[-1]], ctx=device), (1, 1)) for y in prefix[1:]: # Warm-up period _, state = model(get_input(), state) outputs.append(vocab[y]) for _ in range(num_preds): # Predictnum_preds` steps y, state = model(get_input(), state) outputs.append(int(y.argmax(axis=1).reshape(1))) return ‘’.join([vocab.idx_to_token[i] for i in outputs])

  2. ```{.python .input}
  3. #@tab pytorch
  4. def predict_ch8(prefix, num_preds, model, vocab, device):  #@save
  5.     """Generate new characters following the `prefix`."""
  6.     state = model.begin_state(batch_size=1, device=device)
  7.     outputs = [vocab[prefix[0]]]
  8.     get_input = lambda: d2l.reshape(d2l.tensor(
  9.         [outputs[-1]], device=device), (1, 1))
  10.     for y in prefix[1:]:  # Warm-up period
  11.         _, state = model(get_input(), state)
  12.         outputs.append(vocab[y])
  13.     for _ in range(num_preds):  # Predict `num_preds` steps
  14.         y, state = model(get_input(), state)
  15.         outputs.append(int(y.argmax(dim=1).reshape(1)))
  16.     return ''.join([vocab.idx_to_token[i] for i in outputs])

```{.python .input}

@tab tensorflow

def predictch8(prefix, num_preds, model, vocab, params): #@save “””Generate new characters following the prefix.””” state = model.begin_state(batch_size=1) outputs = [vocab[prefix[0]]] get_input = lambda: d2l.reshape(d2l.tensor([outputs[-1]]), (1, 1)).numpy() for y in prefix[1:]: # Warm-up period , state = model(getinput(), state, params) outputs.append(vocab[y]) for in range(num_preds): # Predict num_preds steps y, state = model(get_input(), state, params) outputs.append(int(y.numpy().argmax(axis=1).reshape(1))) return ‘’.join([vocab.idx_to_token[i] for i in outputs])

  2. Now we can test the `predict_ch8` function.
  3. We specify the prefix as `time traveller ` and have it generate 10 additional characters.
  4. Given that we have not trained the network,
  5. it will generate nonsensical predictions.
  7. ```{.python .input}
  8. #@tab mxnet,pytorch
  9. predict_ch8('time traveller ', 10, model, vocab, d2l.try_gpu())

```{.python .input}

@tab tensorflow

predict_ch8(‘time traveller ‘, 10, model, vocab, params)

  2. ## Gradient Clipping
  4. For a sequence of length $T$,
  5. we compute the gradients over these $T$ time steps in an iteration, which results in a chain of matrix-products with length  $\mathcal{O}(T)$ during backpropagation.
  6. As mentioned in :numref:`sec_numerical_stability`, it might result in numerical instability, e.g., the gradients may either explode or vanish, when $T$ is large. Therefore, RNN models often need extra help to stabilize the training.
  8. Generally speaking,
  9. when solving an optimization problem,
  10. we take update steps for the model parameter,
  11. say in the vector form
  12. $\mathbf{x}$,
  13. in the direction of the negative gradient $\mathbf{g}$ on a minibatch.
  14. For example,
  15. with $\eta > 0$ as the learning rate,
  16. in one iteration we update
  17. $\mathbf{x}$
  18. as $\mathbf{x} - \eta \mathbf{g}$.
  19. Let us further assume that the objective function $f$
  20. is well behaved, say, *Lipschitz continuous* with constant $L$.
  21. That is to say,
  22. for any $\mathbf{x}$ and $\mathbf{y}$ we have
  24. $$|f(\mathbf{x}) - f(\mathbf{y})| \leq L \|\mathbf{x} - \mathbf{y}\|.$$
  26. In this case we can safely assume that if we update the parameter vector by $\eta \mathbf{g}$, then
  28. $$|f(\mathbf{x}) - f(\mathbf{x} - \eta\mathbf{g})| \leq L \eta\|\mathbf{g}\|,$$
  30. which means that
  31. we will not observe a change by more than $L \eta \|\mathbf{g}\|$. This is both a curse and a blessing.
  32. On the curse side,
  33. it limits the speed of making progress;
  34. whereas on the blessing side,
  35. it limits the extent to which things can go wrong if we move in the wrong direction.
  37. Sometimes the gradients can be quite large and the optimization algorithm may fail to converge. We could address this by reducing the learning rate $\eta$. But what if we only *rarely* get large gradients? In this case such an approach may appear entirely unwarranted. One popular alternative is to clip the gradient $\mathbf{g}$ by projecting them back to a ball of a given radius, say $\theta$ via
  39. $$\mathbf{g} \leftarrow \min\left(1, \frac{\theta}{\|\mathbf{g}\|}\right) \mathbf{g}.$$
  41. By doing so we know that the gradient norm never exceeds $\theta$ and that the
  42. updated gradient is entirely aligned with the original direction of $\mathbf{g}$.
  43. It also has the desirable side-effect of limiting the influence any given
  44. minibatch (and within it any given sample) can exert on the parameter vector. This
  45. bestows a certain degree of robustness to the model. Gradient clipping provides
  46. a quick fix to the gradient exploding. While it does not entirely solve the problem, it is one of the many techniques to alleviate it.
  48. Below we define a function to clip the gradients of
  49. a model that is implemented from scratch or a model constructed by the high-level APIs.
  50. Also note that we compute the gradient norm over all the model parameters.
  52. ```{.python .input}
  53. def grad_clipping(model, theta):  #@save
  54.     """Clip the gradient."""
  55.     if isinstance(model, gluon.Block):
  56.         params = [p.data() for p in model.collect_params().values()]
  57.     else:
  58.         params = model.params
  59.     norm = math.sqrt(sum((p.grad ** 2).sum() for p in params))
  60.     if norm > theta:
  61.         for param in params:
  62.             param.grad[:] *= theta / norm

```{.python .input}

@tab pytorch

def grad_clipping(model, theta): #@save “””Clip the gradient.””” if isinstance(model, nn.Module): params = [p for p in model.parameters() if p.requires_grad] else: params = model.params norm = torch.sqrt(sum(torch.sum((p.grad * 2)) for p in params)) if norm > theta: for param in params: param.grad[:] = theta / norm

  2. ```{.python .input}
  3. #@tab tensorflow
  4. def grad_clipping(grads, theta): #@save
  5.     """Clip the gradient."""
  6.     theta = tf.constant(theta, dtype=tf.float32)
  7.     norm = tf.math.sqrt(sum((tf.reduce_sum(grad ** 2)).numpy()
  8.                         for grad in grads))
  9.     norm = tf.cast(norm, tf.float32)
  10.     new_grad = []
  11.     if tf.greater(norm, theta):
  12.         for grad in grads:
  13.             new_grad.append(grad * theta / norm)
  14.     else:
  15.         for grad in grads:
  16.             new_grad.append(grad)
  17.     return new_grad

Training

Before training the model, let us define a function to train the model in one epoch. It differs from how we train the model of :numref:sec_softmax_scratch in three places:

  1. Different sampling methods for sequential data (random sampling and sequential partitioning) will result in differences in the initialization of hidden states.
  2. We clip the gradients before updating the model parameters. This ensures that the model does not diverge even when gradients blow up at some point during the training process.
  3. We use perplexity to evaluate the model. As discussed in :numref:subsec_perplexity, this ensures that sequences of different length are comparable.

Specifically, when sequential partitioning is used, we initialize the hidden state only at the beginning of each epoch. Since the $i^\mathrm{th}$ subsequence example in the next minibatch is adjacent to the current $i^\mathrm{th}$ subsequence example, the hidden state at the end of the current minibatch will be used to initialize the hidden state at the beginning of the next minibatch. In this way, historical information of the sequence stored in the hidden state might flow over adjacent subsequences within an epoch. However, the computation of the hidden state at any point depends on all the previous minibatches in the same epoch, which complicates the gradient computation. To reduce computational cost, we detach the gradient before processing any minibatch so that the gradient computation of the hidden state is always limited to the time steps in one minibatch.

When using the random sampling, we need to re-initialize the hidden state for each iteration since each example is sampled with a random position. Same as the train_epoch_ch3 function in :numref:sec_softmax_scratch, updater is a general function to update the model parameters. It can be either the d2l.sgd function implemented from scratch or the built-in optimization function in a deep learning framework.

```{.python .input} def train_epoch_ch8(model, train_iter, loss, updater, device, #@save use_random_iter): “””Train a model within one epoch (defined in Chapter 8).””” state, timer = None, d2l.Timer() metric = d2l.Accumulator(2) # Sum of training loss, no. of tokens for X, Y in train_iter: if state is None or use_random_iter:

  1.         # Initialize `state` when either it is the first iteration or
  2.         # using random sampling
  3.         state = model.begin_state(batch_size=X.shape[0], ctx=device)
  4.     else:
  5.         for s in state:
  6.             s.detach()
  7.     y = Y.T.reshape(-1)
  8.     X, y = X.as_in_ctx(device), y.as_in_ctx(device)
  9.     with autograd.record():
  10.         y_hat, state = model(X, state)
  11.         l = loss(y_hat, y).mean()
  12.     l.backward()
  13.     grad_clipping(model, 1)
  14.     updater(batch_size=1)  # Since the `mean` function has been invoked
  15.     metric.add(l * d2l.size(y), d2l.size(y))
  16. return math.exp(metric[0] / metric[1]), metric[1] / timer.stop()
  2. ```{.python .input}
  3. #@tab pytorch
  4. def train_epoch_ch8(model, train_iter, loss, updater, device,  #@save
  5.                     use_random_iter):
  6.     """Train a model within one epoch (defined in Chapter 8)."""
  7.     state, timer = None, d2l.Timer()
  8.     metric = d2l.Accumulator(2)  # Sum of training loss, no. of tokens
  9.     for X, Y in train_iter:
  10.         if state is None or use_random_iter:
  11.             # Initialize `state` when either it is the first iteration or
  12.             # using random sampling
  13.             state = model.begin_state(batch_size=X.shape[0], device=device)
  14.         else:
  15.             if isinstance(model, nn.Module) and not isinstance(state, tuple):
  16.                 # `state` is a tensor for `nn.GRU`
  17.                 state.detach_()
  18.             else:
  19.                 # `state` is a tuple of tensors for `nn.LSTM` and
  20.                 # for our custom scratch implementation 
  21.                 for s in state:
  22.                     s.detach_()
  23.         y = Y.T.reshape(-1)
  24.         X, y = X.to(device), y.to(device)
  25.         y_hat, state = model(X, state)
  26.         l = loss(y_hat, y.long()).mean()
  27.         if isinstance(updater, torch.optim.Optimizer):
  28.             updater.zero_grad()
  29.             l.backward()
  30.             grad_clipping(model, 1)
  31.             updater.step()
  32.         else:
  33.             l.backward()
  34.             grad_clipping(model, 1)
  35.             # Since the `mean` function has been invoked
  36.             updater(batch_size=1)
  37.         metric.add(l * d2l.size(y), d2l.size(y))
  38.     return math.exp(metric[0] / metric[1]), metric[1] / timer.stop()

```{.python .input}

@tab tensorflow

def train_epoch_ch8(model, train_iter, loss, updater, #@save params, use_random_iter): “””Train a model within one epoch (defined in Chapter 8).””” state, timer = None, d2l.Timer() metric = d2l.Accumulator(2) # Sum of training loss, no. of tokens for X, Y in train_iter: if state is None or use_random_iter:

  1.         # Initialize `state` when either it is the first iteration or
  2.         # using random sampling
  3.         state = model.begin_state(batch_size=X.shape[0])
  4.     with tf.GradientTape(persistent=True) as g:
  5.         g.watch(params)
  6.         y_hat, state= model(X, state, params)
  7.         y = d2l.reshape(Y, (-1))
  8.         l = loss(y, y_hat)
  9.     grads = g.gradient(l, params)
  10.     grads = grad_clipping(grads, 1)
  11.     updater.apply_gradients(zip(grads, params))
  13.     # Keras loss by default returns the average loss in a batch
  14.     # l_sum = l * float(d2l.size(y)) if isinstance(
  15.     #     loss, tf.keras.losses.Loss) else tf.reduce_sum(l)
  16.     metric.add(l * d2l.size(y), d2l.size(y))
  17. return math.exp(metric[0] / metric[1]), metric[1] / timer.stop()
  2. The training function supports
  3. an RNN model implemented
  4. either from scratch
  5. or using high-level APIs.
  7. ```{.python .input}
  8. def train_ch8(model, train_iter, vocab, lr, num_epochs, device,  #@save
  9.               use_random_iter=False):
  10.     """Train a model (defined in Chapter 8)."""
  11.     loss = gluon.loss.SoftmaxCrossEntropyLoss()
  12.     animator = d2l.Animator(xlabel='epoch', ylabel='perplexity',
  13.                             legend=['train'], xlim=[10, num_epochs])
  14.     # Initialize
  15.     if isinstance(model, gluon.Block):
  16.         model.initialize(ctx=device, force_reinit=True,
  17.                          init=init.Normal(0.01))
  18.         trainer = gluon.Trainer(model.collect_params(),
  19.                                 'sgd', {'learning_rate': lr})
  20.         updater = lambda batch_size: trainer.step(batch_size)
  21.     else:
  22.         updater = lambda batch_size: d2l.sgd(model.params, lr, batch_size)
  23.     predict = lambda prefix: predict_ch8(prefix, 50, model, vocab, device)
  24.     # Train and predict
  25.     for epoch in range(num_epochs):
  26.         ppl, speed = train_epoch_ch8(
  27.             model, train_iter, loss, updater, device, use_random_iter)
  28.         if (epoch + 1) % 10 == 0:
  29.             animator.add(epoch + 1, [ppl])
  30.     print(f'perplexity {ppl:.1f}, {speed:.1f} tokens/sec on {str(device)}')
  31.     print(predict('time traveller'))
  32.     print(predict('traveller'))

```{.python .input}

@tab pytorch

@save

def train_ch8(model, train_iter, vocab, lr, num_epochs, device, use_random_iter=False): “””Train a model (defined in Chapter 8).””” loss = nn.CrossEntropyLoss() animator = d2l.Animator(xlabel=’epoch’, ylabel=’perplexity’, legend=[‘train’], xlim=[10, num_epochs])

  1. # Initialize
  2. if isinstance(model, nn.Module):
  3.     updater = torch.optim.SGD(model.parameters(), lr)
  4. else:
  5.     updater = lambda batch_size: d2l.sgd(model.params, lr, batch_size)
  6. predict = lambda prefix: predict_ch8(prefix, 50, model, vocab, device)
  7. # Train and predict
  8. for epoch in range(num_epochs):
  9.     ppl, speed = train_epoch_ch8(
  10.         model, train_iter, loss, updater, device, use_random_iter)
  11.     if (epoch + 1) % 10 == 0:
  12.         print(predict('time traveller'))
  13.         animator.add(epoch + 1, [ppl])
  14. print(f'perplexity {ppl:.1f}, {speed:.1f} tokens/sec on {str(device)}')
  15. print(predict('time traveller'))
  16. print(predict('traveller'))
  2. ```{.python .input}
  3. #@tab tensorflow
  4. #@save
  5. def train_ch8(model, train_iter, vocab, num_hiddens, lr, num_epochs, strategy,
  6.               use_random_iter=False):
  7.     """Train a model (defined in Chapter 8)."""
  8.     with strategy.scope():
  9.         params = get_params(len(vocab), num_hiddens)
  10.         loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
  11.         updater = tf.keras.optimizers.SGD(lr)
  12.     animator = d2l.Animator(xlabel='epoch', ylabel='perplexity',
  13.                             legend=['train'], xlim=[10, num_epochs])
  14.     predict = lambda prefix: predict_ch8(prefix, 50, model, vocab, params)
  15.     # Train and predict
  16.     for epoch in range(num_epochs):
  17.         ppl, speed = train_epoch_ch8(
  18.              model, train_iter, loss, updater, params, use_random_iter)
  19.         if (epoch + 1) % 10 == 0:
  20.             print(predict('time traveller'))
  21.             animator.add(epoch + 1, [ppl])
  22.     device = d2l.try_gpu()._device_name
  23.     print(f'perplexity {ppl:.1f}, {speed:.1f} tokens/sec on {str(device)}')
  24.     print(predict('time traveller'))
  25.     print(predict('traveller'))

Now we can train the RNN model. Since we only use 10000 tokens in the dataset, the model needs more epochs to converge better.

```{.python .input}

@tab mxnet,pytorch

num_epochs, lr = 500, 1 train_ch8(model, train_iter, vocab, lr, num_epochs, d2l.try_gpu())

  2. ```{.python .input}
  3. #@tab tensorflow
  4. num_epochs, lr = 500, 1
  5. train_ch8(model, train_iter, vocab, num_hiddens, lr, num_epochs, strategy)

Finally, let us check the results of using the random sampling method.

```{.python .input}

@tab mxnet,pytorch

train_ch8(model, train_iter, vocab, lr, num_epochs, d2l.try_gpu(), use_random_iter=True)

  2. ```{.python .input}
  3. #@tab tensorflow
  4. params = get_params(len(vocab_random_iter), num_hiddens)
  5. train_ch8(model, train_random_iter, vocab_random_iter, num_hiddens, lr,
  6.           num_epochs, strategy, use_random_iter=True)

While implementing the above RNN model from scratch is instructive, it is not convenient. In the next section we will see how to improve the RNN model, such as how to make it easier to implement and make it run faster.

Summary

  • We can train an RNN-based character-level language model to generate text following the user-provided text prefix.
  • A simple RNN language model consists of input encoding, RNN modeling, and output generation.
  • RNN models need state initialization for training, though random sampling and sequential partitioning use different ways.
  • When using sequential partitioning, we need to detach the gradient to reduce computational cost.
  • A warm-up period allows a model to update itself (e.g., obtain a better hidden state than its initialized value) before making any prediction.
  • Gradient clipping prevents gradient explosion, but it cannot fix vanishing gradients.

Exercises

  1. Show that one-hot encoding is equivalent to picking a different embedding for each object.
  2. Adjust the hyperparameters (e.g., number of epochs, number of hidden units, number of time steps in a minibatch, and learning rate) to improve the perplexity.
    • How low can you go?
    • Replace random sampling with sequential partitioning. Does this lead to better performance?
    • Replace one-hot encoding with learnable embeddings. Does this lead to better performance?
    • How well will it work on other books by H. G. Wells, e.g., The War of the Worlds?
  3. Modify the prediction function such as to use sampling rather than picking the most likely next character.
    • What happens?
    • Bias the model towards more likely outputs, e.g., by sampling from $q(xt \mid x{t-1}, \ldots, x1) \propto P(x_t \mid x{t-1}, \ldots, x_1)^\alpha$ for $\alpha > 1$.
  4. Run the code in this section without clipping the gradient. What happens?
  5. Change sequential partitioning so that it does not separate hidden states from the computational graph. Does the running time change? How about the perplexity?
  6. Replace the activation function used in this section with ReLU and repeat the experiments in this section. Do we still need gradient clipping? Why?

:begin_tab:mxnet Discussions :end_tab:

:begin_tab:pytorch Discussions :end_tab:

:begin_tab:tensorflow Discussions :end_tab: