Sentiment Analysis: Using Recurrent Neural Networks

:label:sec_sentiment_rnn

Like word similarity and analogy tasks, we can also apply pretrained word vectors to sentiment analysis. Since the IMDb review dataset in :numref:sec_sentiment is not very big, using text representations that were pretrained on large-scale corpora may reduce overfitting of the model. As a specific example illustrated in :numref:fig_nlp-map-sa-rnn, we will represent each token using the pretrained GloVe model, and feed these token representations into a multilayer bidirectional RNN to obtain the text sequence representation, which will be transformed into sentiment analysis outputs :cite:Maas.Daly.Pham.ea.2011. For the same downstream application, we will consider a different architectural choice later.

This section feeds pretrained GloVe to an RNN-based architecture for sentiment analysis. :label:fig_nlp-map-sa-rnn

```{.python .input} from d2l import mxnet as d2l from mxnet import gluon, init, np, npx from mxnet.gluon import nn, rnn npx.set_np()

batch_size = 64 train_iter, test_iter, vocab = d2l.load_data_imdb(batch_size)

  2. ```{.python .input}
  3. #@tab pytorch
  4. from d2l import torch as d2l
  5. import torch
  6. from torch import nn
  8. batch_size = 64
  9. train_iter, test_iter, vocab = d2l.load_data_imdb(batch_size)

Representing Single Text with RNNs

In text classifications tasks, such as sentiment analysis, a varying-length text sequence will be transformed into fixed-length categories. In the following BiRNN class, while each token of a text sequence gets its individual pretrained GloVe representation via the embedding layer (self.embedding), the entire sequence is encoded by a bidirectional RNN (self.encoder). More concretely, the hidden states (at the last layer) of the bidirectional LSTM at both the initial and final time steps are concatenated as the representation of the text sequence. This single text representation is then transformed into output categories by a fully connected layer (self.decoder) with two outputs (“positive” and “negative”).

```{.python .input} class BiRNN(nn.Block): def init(self, vocabsize, embedsize, num_hiddens, num_layers, **kwargs): super(BiRNN, self).__init(**kwargs) self.embedding = nn.Embedding(vocab_size, embed_size)

  1.     # Set `bidirectional` to True to get a bidirectional RNN
  2.     self.encoder = rnn.LSTM(num_hiddens, num_layers=num_layers,
  3.                             bidirectional=True, input_size=embed_size)
  4.     self.decoder = nn.Dense(2)
  6. def forward(self, inputs):
  7.     # The shape of `inputs` is (batch size, no. of time steps). Because
  8.     # LSTM requires its input's first dimension to be the temporal
  9.     # dimension, the input is transposed before obtaining token
  10.     # representations. The output shape is (no. of time steps, batch size,
  11.     # word vector dimension)
  12.     embeddings = self.embedding(inputs.T)
  13.     # Returns hidden states of the last hidden layer at different time
  14.     # steps. The shape of `outputs` is (no. of time steps, batch size,
  15.     # 2 * no. of hidden units)
  16.     outputs = self.encoder(embeddings)
  17.     # Concatenate the hidden states at the initial and final time steps as
  18.     # the input of the fully connected layer. Its shape is (batch size,
  19.     # 4 * no. of hidden units)
  20.     encoding = np.concatenate((outputs[0], outputs[-1]), axis=1)
  21.     outs = self.decoder(encoding)
  22.     return outs
  2. ```{.python .input}
  3. #@tab pytorch
  4. class BiRNN(nn.Module):
  5.     def __init__(self, vocab_size, embed_size, num_hiddens,
  6.                  num_layers, **kwargs):
  7.         super(BiRNN, self).__init__(**kwargs)
  8.         self.embedding = nn.Embedding(vocab_size, embed_size)
  9.         # Set `bidirectional` to True to get a bidirectional RNN
  10.         self.encoder = nn.LSTM(embed_size, num_hiddens, num_layers=num_layers,
  11.                                 bidirectional=True)
  12.         self.decoder = nn.Linear(4 * num_hiddens, 2)
  14.     def forward(self, inputs):
  15.         # The shape of `inputs` is (batch size, no. of time steps). Because
  16.         # LSTM requires its input's first dimension to be the temporal
  17.         # dimension, the input is transposed before obtaining token
  18.         # representations. The output shape is (no. of time steps, batch size,
  19.         # word vector dimension)
  20.         embeddings = self.embedding(inputs.T)
  21.         self.encoder.flatten_parameters()
  22.         # Returns hidden states of the last hidden layer at different time
  23.         # steps. The shape of `outputs` is (no. of time steps, batch size,
  24.         # 2 * no. of hidden units)
  25.         outputs, _ = self.encoder(embeddings)
  26.         # Concatenate the hidden states of the initial time step and final
  27.         # time step to use as the input of the fully connected layer. Its
  28.         # shape is (batch size, 4 * no. of hidden units)
  29.         encoding = torch.cat((outputs[0], outputs[-1]), dim=1)
  30.         # Concatenate the hidden states at the initial and final time steps as
  31.         # the input of the fully connected layer. Its shape is (batch size,
  32.         # 4 * no. of hidden units)
  33.         outs = self.decoder(encoding)
  34.         return outs

Let’s construct a bidirectional RNN with two hidden layers to represent single text for sentiment analysis.

```{.python .input}

@tab all

embed_size, num_hiddens, num_layers, devices = 100, 100, 2, d2l.try_all_gpus() net = BiRNN(len(vocab), embed_size, num_hiddens, num_layers)

  2. ```{.python .input}
  3. net.initialize(init.Xavier(), ctx=devices)

```{.python .input}

@tab pytorch

def initweights(m): if type(m) == nn.Linear: nn.init.xavier_uniform(m.weight) if type(m) == nn.LSTM: for param in m.flat_weights_names: if “weight” in param: nn.init.xavier_uniform(m._parameters[param]) net.apply(init_weights);

  2. ## Loading Pretrained Word Vectors
  4. Below we load the pretrained 100-dimensional (needs to be consistent with `embed_size`) GloVe embeddings for tokens in the vocabulary.
  6. ```{.python .input}
  7. #@tab all
  8. glove_embedding = d2l.TokenEmbedding('glove.6b.100d')

Print the shape of the vectors for all the tokens in the vocabulary.

```{.python .input}

@tab all

embeds = glove_embedding[vocab.idx_to_token] embeds.shape

  2. We use these pretrained
  3. word vectors
  4. to represent tokens in the reviews
  5. and will not update
  6. these vectors during training.
  8. ```{.python .input}
  9. net.embedding.weight.set_data(embeds)
  10. net.embedding.collect_params().setattr('grad_req', 'null')

```{.python .input}

@tab pytorch

net.embedding.weight.data.copy_(embeds) net.embedding.weight.requires_grad = False

  2. ## Training and Evaluating the Model
  4. Now we can train the bidirectional RNN for sentiment analysis.
  6. ```{.python .input}
  7. lr, num_epochs = 0.01, 5
  8. trainer = gluon.Trainer(net.collect_params(), 'adam', {'learning_rate': lr})
  9. loss = gluon.loss.SoftmaxCrossEntropyLoss()
  10. d2l.train_ch13(net, train_iter, test_iter, loss, trainer, num_epochs, devices)

```{.python .input}

@tab pytorch

lr, num_epochs = 0.01, 5 trainer = torch.optim.Adam(net.parameters(), lr=lr) loss = nn.CrossEntropyLoss(reduction=”none”) d2l.train_ch13(net, train_iter, test_iter, loss, trainer, num_epochs, devices)

  2. We define the following function to predict the sentiment of a text sequence using the trained model `net`.
  4. ```{.python .input}
  5. #@save
  6. def predict_sentiment(net, vocab, sequence):
  7.     """Predict the sentiment of a text sequence."""
  8.     sequence = np.array(vocab[sequence.split()], ctx=d2l.try_gpu())
  9.     label = np.argmax(net(sequence.reshape(1, -1)), axis=1)
  10.     return 'positive' if label == 1 else 'negative'

```{.python .input}

@tab pytorch

@save

def predict_sentiment(net, vocab, sequence): “””Predict the sentiment of a text sequence.””” sequence = torch.tensor(vocab[sequence.split()], device=d2l.try_gpu()) label = torch.argmax(net(sequence.reshape(1, -1)), dim=1) return ‘positive’ if label == 1 else ‘negative’

  2. Finally, let's use the trained model to predict the sentiment for two simple sentences.
  4. ```{.python .input}
  5. #@tab all
  6. predict_sentiment(net, vocab, 'this movie is so great')

```{.python .input}

@tab all

predict_sentiment(net, vocab, ‘this movie is so bad’) ```

Summary

  • Pretrained word vectors can represent individual tokens in a text sequence.
  • Bidirectional RNNs can represent a text sequence, such as via the concatenation of its hidden states at the initial and final time steps. This single text representation can be transformed into categories using a fully connected layer.

Exercises

  1. Increase the number of epochs. Can you improve the training and testing accuracies? How about tuning other hyperparameters?
  2. Use larger pretrained word vectors, such as 300-dimensional GloVe embeddings. Does it improve classification accuracy?
  3. Can we improve the classification accuracy by using the spaCy tokenization? You need to install spaCy (pip install spacy) and install the English package (python -m spacy download en). In the code, first, import spaCy (import spacy). Then, load the spaCy English package (spacy_en = spacy.load('en')). Finally, define the function def tokenizer(text): return [tok.text for tok in spacy_en.tokenizer(text)] and replace the original tokenizer function. Note the different forms of phrase tokens in GloVe and spaCy. For example, the phrase token “new york” takes the form of “new-york” in GloVe and the form of “new york” after the spaCy tokenization.

:begin_tab:mxnet Discussions :end_tab:

:begin_tab:pytorch Discussions :end_tab: