1. A Neural Probabilistic Language Model

本文算是训练语言模型的经典之作，Bengio 将神经网络引入语言模型的训练中，并得到了词向量这个副产物。词向量对后面深度学习在自然语言处理方面有很大的贡献，也是获取词的语义特征的有效方法
其主要架构为三层神经网络，如下图所示
NNLM的pytorch实现 - 图1
现在的任务是输入 wt−n+1,…,wt−1 这前 n-1 个单词，然后预测出下一个单词 wt
数学符号说明：

  • C(i)：单词 w 对应的词向量，其中 i 为词 w 在整个词汇表中的索引
  • C：词向量，大小为 |V|×m 的矩阵
  • |V|：词汇表的大小，即预料库中去重后的单词个数
  • m：词向量的维度，一般是 50 到 200
  • H：隐藏层的 weight
  • d：隐藏层的 bias
  • U：输出层的 weight
  • b：输出层的 bias
  • W：输入层到输出层的 weight
  • h：隐藏层神经元个数

计算流程：

  1. 首先将输入的 n-1 个单词索引转为词向量，然后将这 n-1 个词向量进行 concat，形成一个 (n-1)*w 的向量，用 X 表示
  2. 将 X 送入隐藏层进行计算，hiddenout=tanh(d+X∗H)
  3. 输出层共有 |V| 个节点，每个节点 yi 表示预测下一个单词 i 的概率，y 的计算公式为 y=b+X∗W+hiddenout∗U

2.代码实现（PyTorch）

  1. import torch
  2. import torch.nn as nn
  3. import torch.optim as optim
  4. import torch.utils.data as Data
  6. dtype = torch.FloatTensor
  8. sentences = [ "i like dog", "i love coffee", "i hate milk"]
  10. word_list = " ".join(sentences).split() # ['i', 'like', 'dog', 'dog', 'i', 'love', 'coffee', 'i', 'hate', 'milk']
  11. word_list = list(set(word_list)) # ['i', 'like', 'dog', 'love', 'coffee', 'hate', 'milk']
  12. word2idx = {w: i for i, w in enumerate(word_list)} # {'i':0, 'like':1, 'dog':2, 'love':3, 'coffee':4, 'hate':5, 'milk':6}
  13. idx2word = {i: w for i, w in enumerate(word_list)} # {0:'i', 1:'like', 2:'dog', 3:'love', 4:'coffee', 5:'hate', 6:'milk'}
  14. VOCAB_SIZE = len(word_list) # number of Vocabulary, just like |V|, in this task n_class=7
  16. # NNLM(Neural Network Language Model) Parameter
  17. SEQ_LEN = len(sentences[0].split())-1 # n-1 in paper, look back n_step words and predict next word. In this task n_step=2
  18. HIDDEN_SIZE = 2 # h in paper
  19. EMBED_SIZE = 2 # m in paper, word embedding dim
  21. def make_batch(sentences):
  22.   input_batch = []
  23.   target_batch = []
  25.   for sen in sentences:
  26.     word = sen.split()
  27.     input = [word2idx[n] for n in word[:-1]] # [0, 1], [0, 3], [0, 5]
  28.     target = word2idx[word[-1]] # 2, 4, 6
  30.     input_batch.append(input) # [[0, 1], [0, 3], [0, 5]]
  31.     target_batch.append(target) # [2, 4, 6]
  33.   return input_batch, target_batch
  35. input_batch, target_batch = make_batch(sentences)
  36. input_batch = torch.LongTensor(input_batch)
  37. target_batch = torch.LongTensor(target_batch)
  39. dataset = Data.TensorDataset(input_batch, target_batch)
  40. loader = Data.DataLoader(dataset=dataset, batch_size=16, shuffle=True)
  42. class NNLM(nn.Module):
  43.   def __init__(self,vocab_size,embed_size,hidden_size,seq_len):
  44.     super(NNLM, self).__init__()
  45.     self.C = nn.Embedding(vocab_size, embed_size)
  46.     self.H = nn.Parameter(torch.randn(seq_len*embed_size, hidden_size).type(dtype))
  47.     self.W = nn.Parameter(torch.randn(seq_len*embed_size, vocab_size).type(dtype))
  48.     self.d = nn.Parameter(torch.randn(hidden_size).type(dtype))
  49.     self.U = nn.Parameter(torch.randn(hidden_size, vocab_size).type(dtype))
  50.     self.b = nn.Parameter(torch.randn(vocab_size).type(dtype))
  52.   def forward(self, X):
  53.     '''
  54.     X: [batch_size, n_step]
  55.     '''
  56.     X = self.C(X) # [batch_size, n_step] => [batch_size, seq_len, embed_size]
  57.     X = X.view(-1,X.size(1)*X.size(2)) # [batch_size, seq_len*embed_size]
  58.     hidden_out = torch.tanh(self.d + torch.mm(X, self.H)) # [batch_size, hidden_size]
  59.     output = self.b + torch.mm(X, self.W) + torch.mm(hidden_out, self.U) # [batch_size, vocab_size]
  60.     return output
  61. model = NNLM(VOCAB_SIZE,EMBED_SIZE,HIDDEN_SIZE,SEQ_LEN)
  62. criterion = nn.CrossEntropyLoss()
  63. optimizer = optim.Adam(model.parameters(), lr=1e-3)
  65. # Training
  66. for epoch in range(5000):
  67.   for batch_x, batch_y in loader:
  68.     optimizer.zero_grad()
  69.     output = model(batch_x)
  71.     # output : [batch_size, n_class], batch_y : [batch_size] (LongTensor, not one-hot)
  72.     loss = criterion(output, batch_y)
  73.     if (epoch + 1)%1000 == 0:
  74.         print('Epoch:', '%04d' % (epoch + 1), 'cost =', '{:.6f}'.format(loss))
  76.     loss.backward()
  77.     optimizer.step()
  79. # Predict
  80. predict = model(input_batch).data.max(1, keepdim=True)[1]
  82. # Test
  83. print([sen.split()[:SEQ_LEN] for sen in sentences], '->', [idx2word[n.item()] for n in predict.squeeze()])