torchtext语言翻译

原文：https://pytorch.org/tutorials/beginner/torchtext_translation_tutorial.html

本教程介绍了如何使用torchtext预处理包含英语和德语句子的著名数据集的数据，并使用它来训练序列到序列模型，并能将德语句子翻译成英语。

它基于 PyTorch 社区成员 Ben Trevett 的本教程，并获得 Ben 的许可。 我们通过删除一些旧代码来更新教程。

在本教程结束时，您将可以将句子预处理为张量以用于 NLP 建模，并可以使用torch.utils.data.DataLoader来训练和验证模型。

数据处理

torchtext具有工具，可用于创建可以轻松迭代的数据集，以创建语言翻译模型。 在此示例中，我们展示了如何对原始文本句子进行标记，构建词汇表以及将标记数字化为张量。

注意：本教程中的分词需要 Spacy 我们使用 Spacy 是因为它为英语以外的其他语言的分词提供了强大的支持。 torchtext提供了basic_english标记器，并支持其他英语标记器（例如 Moses），但对于语言翻译（需要多种语言），Spacy 是您的最佳选择。

要运行本教程，请先使用pip或conda安装spacy。 接下来，下载英语和德语 Spacy 分词器的原始数据：

  1. python -m spacy download en
  2. python -m spacy download de
  1. import torchtext
  2. import torch
  3. from torchtext.data.utils import get_tokenizer
  4. from collections import Counter
  5. from torchtext.vocab import Vocab
  6. from torchtext.utils import download_from_url, extract_archive
  7. import io
  9. url_base = 'https://raw.githubusercontent.com/multi30k/dataset/master/data/task1/raw/'
  10. train_urls = ('train.de.gz', 'train.en.gz')
  11. val_urls = ('val.de.gz', 'val.en.gz')
  12. test_urls = ('test_2016_flickr.de.gz', 'test_2016_flickr.en.gz')
  14. train_filepaths = [extract_archive(download_from_url(url_base + url))[0] for url in train_urls]
  15. val_filepaths = [extract_archive(download_from_url(url_base + url))[0] for url in val_urls]
  16. test_filepaths = [extract_archive(download_from_url(url_base + url))[0] for url in test_urls]
  18. de_tokenizer = get_tokenizer('spacy', language='de')
  19. en_tokenizer = get_tokenizer('spacy', language='en')
  21. def build_vocab(filepath, tokenizer):
  22.   counter = Counter()
  23.   with io.open(filepath, encoding="utf8") as f:
  24.     for string_ in f:
  25.       counter.update(tokenizer(string_))
  26.   return Vocab(counter, specials=['<unk>', '<pad>', '<bos>', '<eos>'])
  28. de_vocab = build_vocab(train_filepaths[0], de_tokenizer)
  29. en_vocab = build_vocab(train_filepaths[1], en_tokenizer)
  31. def data_process(filepaths):
  32.   raw_de_iter = iter(io.open(filepaths[0], encoding="utf8"))
  33.   raw_en_iter = iter(io.open(filepaths[1], encoding="utf8"))
  34.   data = []
  35.   for (raw_de, raw_en) in zip(raw_de_iter, raw_en_iter):
  36.     de_tensor_ = torch.tensor([de_vocab[token] for token in de_tokenizer(raw_de)],
  37.                             dtype=torch.long)
  38.     en_tensor_ = torch.tensor([en_vocab[token] for token in en_tokenizer(raw_en)],
  39.                             dtype=torch.long)
  40.     data.append((de_tensor_, en_tensor_))
  41.   return data
  43. train_data = data_process(train_filepaths)
  44. val_data = data_process(val_filepaths)
  45. test_data = data_process(test_filepaths)

DataLoader

我们将使用的最后torch个特定函数是DataLoader，它易于使用，因为它将数据作为第一个参数。 具体来说，正如文档所说：DataLoader结合了一个数据集和一个采样器，并在给定的数据集上提供了可迭代的。 DataLoader支持映射样式和可迭代样式的数据集，具有单进程或多进程加载，自定义加载顺序以及可选的自动批量（归类）和内存固定。

请注意collate_fn（可选），它将合并样本列表以形成张量的小批量。 在从映射样式数据集中使用批量加载时使用。

  1. import torch
  3. device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
  5. BATCH_SIZE = 128
  6. PAD_IDX = de_vocab['<pad>']
  7. BOS_IDX = de_vocab['<bos>']
  8. EOS_IDX = de_vocab['<eos>']
  10. from torch.nn.utils.rnn import pad_sequence
  11. from torch.utils.data import DataLoader
  13. def generate_batch(data_batch):
  14.   de_batch, en_batch = [], []
  15.   for (de_item, en_item) in data_batch:
  16.     de_batch.append(torch.cat([torch.tensor([BOS_IDX]), de_item, torch.tensor([EOS_IDX])], dim=0))
  17.     en_batch.append(torch.cat([torch.tensor([BOS_IDX]), en_item, torch.tensor([EOS_IDX])], dim=0))
  18.   de_batch = pad_sequence(de_batch, padding_value=PAD_IDX)
  19.   en_batch = pad_sequence(en_batch, padding_value=PAD_IDX)
  20.   return de_batch, en_batch
  22. train_iter = DataLoader(train_data, batch_size=BATCH_SIZE,
  23.                         shuffle=True, collate_fn=generate_batch)
  24. valid_iter = DataLoader(val_data, batch_size=BATCH_SIZE,
  25.                         shuffle=True, collate_fn=generate_batch)
  26. test_iter = DataLoader(test_data, batch_size=BATCH_SIZE,
  27.                        shuffle=True, collate_fn=generate_batch)

定义我们的nn.Module和Optimizer

这大部分是从torchtext角度出发的：构建了数据集并定义了迭代器，本教程的其余部分仅将模型定义为nn.Module以及Optimizer，然后对其进行训练。

具体来说，我们的模型遵循此处描述的架构（您可以在这里找到注释更多的版本。

注意：此模型只是可用于语言翻译的示例模型； 我们选择它是因为它是任务的标准模型，而不是因为它是用于翻译的推荐模型。 如您所知，目前最先进的模型基于“转换器”； 您可以看到 PyTorch 的实现Transformer层的功能； 特别是，以下模型中使用的“注意”与转换器模型中存在的多头自我注意不同。

  1. import random
  2. from typing import Tuple
  4. import torch.nn as nn
  5. import torch.optim as optim
  6. import torch.nn.functional as F
  7. from torch import Tensor
  9. class Encoder(nn.Module):
  10.     def __init__(self,
  11.                  input_dim: int,
  12.                  emb_dim: int,
  13.                  enc_hid_dim: int,
  14.                  dec_hid_dim: int,
  15.                  dropout: float):
  16.         super().__init__()
  18.         self.input_dim = input_dim
  19.         self.emb_dim = emb_dim
  20.         self.enc_hid_dim = enc_hid_dim
  21.         self.dec_hid_dim = dec_hid_dim
  22.         self.dropout = dropout
  24.         self.embedding = nn.Embedding(input_dim, emb_dim)
  26.         self.rnn = nn.GRU(emb_dim, enc_hid_dim, bidirectional = True)
  28.         self.fc = nn.Linear(enc_hid_dim * 2, dec_hid_dim)
  30.         self.dropout = nn.Dropout(dropout)
  32.     def forward(self,
  33.                 src: Tensor) -> Tuple[Tensor]:
  35.         embedded = self.dropout(self.embedding(src))
  37.         outputs, hidden = self.rnn(embedded)
  39.         hidden = torch.tanh(self.fc(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim = 1)))
  41.         return outputs, hidden
  43. class Attention(nn.Module):
  44.     def __init__(self,
  45.                  enc_hid_dim: int,
  46.                  dec_hid_dim: int,
  47.                  attn_dim: int):
  48.         super().__init__()
  50.         self.enc_hid_dim = enc_hid_dim
  51.         self.dec_hid_dim = dec_hid_dim
  53.         self.attn_in = (enc_hid_dim * 2) + dec_hid_dim
  55.         self.attn = nn.Linear(self.attn_in, attn_dim)
  57.     def forward(self,
  58.                 decoder_hidden: Tensor,
  59.                 encoder_outputs: Tensor) -> Tensor:
  61.         src_len = encoder_outputs.shape[0]
  63.         repeated_decoder_hidden = decoder_hidden.unsqueeze(1).repeat(1, src_len, 1)
  65.         encoder_outputs = encoder_outputs.permute(1, 0, 2)
  67.         energy = torch.tanh(self.attn(torch.cat((
  68.             repeated_decoder_hidden,
  69.             encoder_outputs),
  70.             dim = 2)))
  72.         attention = torch.sum(energy, dim=2)
  74.         return F.softmax(attention, dim=1)
  76. class Decoder(nn.Module):
  77.     def __init__(self,
  78.                  output_dim: int,
  79.                  emb_dim: int,
  80.                  enc_hid_dim: int,
  81.                  dec_hid_dim: int,
  82.                  dropout: int,
  83.                  attention: nn.Module):
  84.         super().__init__()
  86.         self.emb_dim = emb_dim
  87.         self.enc_hid_dim = enc_hid_dim
  88.         self.dec_hid_dim = dec_hid_dim
  89.         self.output_dim = output_dim
  90.         self.dropout = dropout
  91.         self.attention = attention
  93.         self.embedding = nn.Embedding(output_dim, emb_dim)
  95.         self.rnn = nn.GRU((enc_hid_dim * 2) + emb_dim, dec_hid_dim)
  97.         self.out = nn.Linear(self.attention.attn_in + emb_dim, output_dim)
  99.         self.dropout = nn.Dropout(dropout)
  101.     def _weighted_encoder_rep(self,
  102.                               decoder_hidden: Tensor,
  103.                               encoder_outputs: Tensor) -> Tensor:
  105.         a = self.attention(decoder_hidden, encoder_outputs)
  107.         a = a.unsqueeze(1)
  109.         encoder_outputs = encoder_outputs.permute(1, 0, 2)
  111.         weighted_encoder_rep = torch.bmm(a, encoder_outputs)
  113.         weighted_encoder_rep = weighted_encoder_rep.permute(1, 0, 2)
  115.         return weighted_encoder_rep
  117.     def forward(self,
  118.                 input: Tensor,
  119.                 decoder_hidden: Tensor,
  120.                 encoder_outputs: Tensor) -> Tuple[Tensor]:
  122.         input = input.unsqueeze(0)
  124.         embedded = self.dropout(self.embedding(input))
  126.         weighted_encoder_rep = self._weighted_encoder_rep(decoder_hidden,
  127.                                                           encoder_outputs)
  129.         rnn_input = torch.cat((embedded, weighted_encoder_rep), dim = 2)
  131.         output, decoder_hidden = self.rnn(rnn_input, decoder_hidden.unsqueeze(0))
  133.         embedded = embedded.squeeze(0)
  134.         output = output.squeeze(0)
  135.         weighted_encoder_rep = weighted_encoder_rep.squeeze(0)
  137.         output = self.out(torch.cat((output,
  138.                                      weighted_encoder_rep,
  139.                                      embedded), dim = 1))
  141.         return output, decoder_hidden.squeeze(0)
  143. class Seq2Seq(nn.Module):
  144.     def __init__(self,
  145.                  encoder: nn.Module,
  146.                  decoder: nn.Module,
  147.                  device: torch.device):
  148.         super().__init__()
  150.         self.encoder = encoder
  151.         self.decoder = decoder
  152.         self.device = device
  154.     def forward(self,
  155.                 src: Tensor,
  156.                 trg: Tensor,
  157.                 teacher_forcing_ratio: float = 0.5) -> Tensor:
  159.         batch_size = src.shape[1]
  160.         max_len = trg.shape[0]
  161.         trg_vocab_size = self.decoder.output_dim
  163.         outputs = torch.zeros(max_len, batch_size, trg_vocab_size).to(self.device)
  165.         encoder_outputs, hidden = self.encoder(src)
  167.         # first input to the decoder is the <sos> token
  168.         output = trg[0,:]
  170.         for t in range(1, max_len):
  171.             output, hidden = self.decoder(output, hidden, encoder_outputs)
  172.             outputs[t] = output
  173.             teacher_force = random.random() < teacher_forcing_ratio
  174.             top1 = output.max(1)[1]
  175.             output = (trg[t] if teacher_force else top1)
  177.         return outputs
  179. INPUT_DIM = len(de_vocab)
  180. OUTPUT_DIM = len(en_vocab)
  181. # ENC_EMB_DIM = 256
  182. # DEC_EMB_DIM = 256
  183. # ENC_HID_DIM = 512
  184. # DEC_HID_DIM = 512
  185. # ATTN_DIM = 64
  186. # ENC_DROPOUT = 0.5
  187. # DEC_DROPOUT = 0.5
  189. ENC_EMB_DIM = 32
  190. DEC_EMB_DIM = 32
  191. ENC_HID_DIM = 64
  192. DEC_HID_DIM = 64
  193. ATTN_DIM = 8
  194. ENC_DROPOUT = 0.5
  195. DEC_DROPOUT = 0.5
  197. enc = Encoder(INPUT_DIM, ENC_EMB_DIM, ENC_HID_DIM, DEC_HID_DIM, ENC_DROPOUT)
  199. attn = Attention(ENC_HID_DIM, DEC_HID_DIM, ATTN_DIM)
  201. dec = Decoder(OUTPUT_DIM, DEC_EMB_DIM, ENC_HID_DIM, DEC_HID_DIM, DEC_DROPOUT, attn)
  203. model = Seq2Seq(enc, dec, device).to(device)
  205. def init_weights(m: nn.Module):
  206.     for name, param in m.named_parameters():
  207.         if 'weight' in name:
  208.             nn.init.normal_(param.data, mean=0, std=0.01)
  209.         else:
  210.             nn.init.constant_(param.data, 0)
  212. model.apply(init_weights)
  214. optimizer = optim.Adam(model.parameters())
  216. def count_parameters(model: nn.Module):
  217.     return sum(p.numel() for p in model.parameters() if p.requires_grad)
  219. print(f'The model has {count_parameters(model):,} trainable parameters')

出：

  1. The model has 3,491,552 trainable parameters

注意：特别是对语言翻译模型的表现进行评分时，我们必须告诉nn.CrossEntropyLoss函数忽略仅填充目标的索引。

  1. PAD_IDX = en_vocab.stoi['<pad>']
  3. criterion = nn.CrossEntropyLoss(ignore_index=PAD_IDX)

最后，我们可以训练和评估该模型：

  1. import math
  2. import time
  4. def train(model: nn.Module,
  5.           iterator: torch.utils.data.DataLoader,
  6.           optimizer: optim.Optimizer,
  7.           criterion: nn.Module,
  8.           clip: float):
  10.     model.train()
  12.     epoch_loss = 0
  14.     for _, (src, trg) in enumerate(iterator):
  15.         src, trg = src.to(device), trg.to(device)
  17.         optimizer.zero_grad()
  19.         output = model(src, trg)
  21.         output = output[1:].view(-1, output.shape[-1])
  22.         trg = trg[1:].view(-1)
  24.         loss = criterion(output, trg)
  26.         loss.backward()
  28.         torch.nn.utils.clip_grad_norm_(model.parameters(), clip)
  30.         optimizer.step()
  32.         epoch_loss += loss.item()
  34.     return epoch_loss / len(iterator)
  36. def evaluate(model: nn.Module,
  37.              iterator: torch.utils.data.DataLoader,
  38.              criterion: nn.Module):
  40.     model.eval()
  42.     epoch_loss = 0
  44.     with torch.no_grad():
  46.         for _, (src, trg) in enumerate(iterator):
  47.             src, trg = src.to(device), trg.to(device)
  49.             output = model(src, trg, 0) #turn off teacher forcing
  51.             output = output[1:].view(-1, output.shape[-1])
  52.             trg = trg[1:].view(-1)
  54.             loss = criterion(output, trg)
  56.             epoch_loss += loss.item()
  58.     return epoch_loss / len(iterator)
  60. def epoch_time(start_time: int,
  61.                end_time: int):
  62.     elapsed_time = end_time - start_time
  63.     elapsed_mins = int(elapsed_time / 60)
  64.     elapsed_secs = int(elapsed_time - (elapsed_mins * 60))
  65.     return elapsed_mins, elapsed_secs
  67. N_EPOCHS = 10
  68. CLIP = 1
  70. best_valid_loss = float('inf')
  72. for epoch in range(N_EPOCHS):
  74.     start_time = time.time()
  76.     train_loss = train(model, train_iter, optimizer, criterion, CLIP)
  77.     valid_loss = evaluate(model, valid_iter, criterion)
  79.     end_time = time.time()
  81.     epoch_mins, epoch_secs = epoch_time(start_time, end_time)
  83.     print(f'Epoch: {epoch+1:02} | Time: {epoch_mins}m {epoch_secs}s')
  84.     print(f'\tTrain Loss: {train_loss:.3f} | Train PPL: {math.exp(train_loss):7.3f}')
  85.     print(f'\t Val. Loss: {valid_loss:.3f} |  Val. PPL: {math.exp(valid_loss):7.3f}')
  87. test_loss = evaluate(model, test_iter, criterion)
  89. print(f'| Test Loss: {test_loss:.3f} | Test PPL: {math.exp(test_loss):7.3f} |')

出：

  1. Epoch: 01 | Time: 0m 59s
  2.         Train Loss: 5.790 | Train PPL: 327.039
  3.          Val. Loss: 5.250 |  Val. PPL: 190.532
  4. Epoch: 02 | Time: 0m 59s
  5.         Train Loss: 4.762 | Train PPL: 116.990
  6.          Val. Loss: 5.037 |  Val. PPL: 153.939
  7. Epoch: 03 | Time: 0m 59s
  8.         Train Loss: 4.527 | Train PPL:  92.475
  9.          Val. Loss: 4.924 |  Val. PPL: 137.525
  10. Epoch: 04 | Time: 0m 59s
  11.         Train Loss: 4.344 | Train PPL:  76.977
  12.          Val. Loss: 4.801 |  Val. PPL: 121.673
  13. Epoch: 05 | Time: 0m 59s
  14.         Train Loss: 4.210 | Train PPL:  67.356
  15.          Val. Loss: 4.758 |  Val. PPL: 116.536
  16. Epoch: 06 | Time: 0m 59s
  17.         Train Loss: 4.125 | Train PPL:  61.875
  18.          Val. Loss: 4.691 |  Val. PPL: 109.004
  19. Epoch: 07 | Time: 0m 59s
  20.         Train Loss: 4.043 | Train PPL:  56.979
  21.          Val. Loss: 4.639 |  Val. PPL: 103.446
  22. Epoch: 08 | Time: 0m 59s
  23.         Train Loss: 3.947 | Train PPL:  51.771
  24.          Val. Loss: 4.589 |  Val. PPL:  98.396
  25. Epoch: 09 | Time: 0m 59s
  26.         Train Loss: 3.874 | Train PPL:  48.135
  27.          Val. Loss: 4.514 |  Val. PPL:  91.324
  28. Epoch: 10 | Time: 0m 59s
  29.         Train Loss: 3.785 | Train PPL:  44.021
  30.          Val. Loss: 4.467 |  Val. PPL:  87.126
  31. | Test Loss: 4.433 | Test PPL:  84.168 |

后续步骤

  • 查看其余的 Ben Trevett 的torchtext使用教程。
  • 敬请关注使用其他torchtext函数以及nn.Transformer通过下一个单词预测进行语言建模的教程！

脚本的总运行时间：（10 分钟 13.398 秒）

下载 Python 源码：torchtext_translation_tutorial.py

下载 Jupyter 笔记本：torchtext_translation_tutorial.ipynb

由 Sphinx 画廊生成的画廊