在解决过拟合问题时，虽然简化模型和增大数据集是有效的方法，但是实际中并不常用，简化模型意味着有些特征被丢弃，增大数据集面临数据获取难的问题。权重衰减（weight decay）是一种更为常用的方法。
权重衰减等价于 L2 范数正则化（regularization）
理论知识看教程复习即可。重点用代码实现看一下正则化的效果。

生成数据集

原始数据的函数关系为，p 表示特征向量的维数：
权重衰减 - 图1
为了制造过拟合的现象，减少训练集的大小，增加特征的数量到200个。

  1. import torch
  2. import d2lzh
  4. n_train, n_test, num_inputs = 20, 100, 200
  5. true_w, true_b = torch.ones(num_inputs, 1) * 0.01, 0.05
  8. features = torch.randn((n_train + n_test, num_inputs))
  9. labels = torch.matmul(features, true_w) + true_b
  10. labels += torch.normal(mean=0, std=0.01, size=labels.size())
  11. train_features, test_features = features[:n_train, :], features[n_train:, :]
  12. train_labels, test_labels = labels[:n_train], labels[n_train:]

L2范数惩罚项

下面定义_L_2范数惩罚项。这里只惩罚模型的权重参数。

  1. def l2_penalty(w):
  2.     return (w**2).sum() / 2

训练及测试

与前一节的函数类似

  1. batch_size, num_epochs, lr = 1, 100, 0.003
  2. net, loss = d2lzh.linreg, d2lzh.squared_loss
  4. dataset = torch.utils.data.TensorDataset(train_features, train_labels)
  5. train_iter = torch.utils.data.DataLoader(dataset, batch_size, shuffle=True)
  8. def fit_and_plot(lambd):
  9.     train_ls, test_ls = [], []
  10.     for _ in range(num_epochs):
  11.         for x, y in train_iter:
  12.             l = loss(net(x, w, b), y) + lambd * l2_penalty(w)
  13.             l = l.sum()
  15.             if w.grad is not None:
  16.                 w.grad.data.zero_()
  17.                 b.grad.data.zero_()
  18.             l.backward()
  19.             d2lzh.sgd([w, b], lr, batch_size)
  20.         train_ls.append(loss(net(train_features, w, b), train_labels).mean().item())
  21.         test_ls.append(loss(net(test_features, w, b), test_labels).mean().item())
  22.     d2lzh.semilogy(range(1, num_epochs + 1), train_ls, 'epochs', 'loss',
  23.                    range(1, num_epochs + 1), test_ls, ['train', 'test'])
  24.     print('L2 norm of w:', w.norm().item())

观察过拟合

先看一下不加权重衰减的表现，即 惩罚项系数为0

  1. if __name__ == "__main__":
  2.     fit_and_plot(lambd=0)

尽管训练误差近似为 0，但泛化误差高居不下

  1. # 矩阵范数
  2. L2 norm of w: 13.747507095336914

image.png
再来看一下加上权重衰减后的效果，

  1. # 加上权重衰减
  2. if __name__ == "__main__":
  3.     fit_and_plot(lambd=10)

可以看到，泛化误差已经大大降低了，最终得到的矩阵范数也比较小。

  1. # 矩阵范数
  2. L2 norm of w: 0.02657567523419857

image.png

完整代码如下：

  1. import torch
  2. import d2lzh
  4. n_train, n_test, num_inputs = 20, 100, 200
  5. true_w, true_b = torch.ones(num_inputs, 1) * 0.01, 0.05
  7. features = torch.randn((n_train + n_test, num_inputs))
  8. labels = torch.matmul(features, true_w) + true_b
  9. labels += torch.normal(mean=0, std=0.01, size=labels.size())
  10. train_features, test_features = features[:n_train, :], features[n_train:, :]
  11. train_labels, test_labels = labels[:n_train], labels[n_train:]
  13. w = torch.randn((num_inputs, 1), requires_grad=True)
  14. b = torch.zeros(1, requires_grad=True)
  17. def l2_penalty(w):
  18.     return (w ** 2).sum() / 2
  21. batch_size, num_epochs, lr = 1, 100, 0.003
  22. net, loss = d2lzh.linreg, d2lzh.squared_loss
  24. dataset = torch.utils.data.TensorDataset(train_features, train_labels)
  25. train_iter = torch.utils.data.DataLoader(dataset, batch_size, shuffle=True)
  28. def fit_and_plot(lambd):
  29.     train_ls, test_ls = [], []
  30.     for _ in range(num_epochs):
  31.         for x, y in train_iter:
  32.             l = loss(net(x, w, b), y) + lambd * l2_penalty(w)
  33.             l = l.sum()
  35.             if w.grad is not None:
  36.                 w.grad.data.zero_()
  37.                 b.grad.data.zero_()
  38.             l.backward()
  39.             d2lzh.sgd([w, b], lr, batch_size)
  40.         train_ls.append(loss(net(train_features, w, b), train_labels).mean().item())
  41.         test_ls.append(loss(net(test_features, w, b), test_labels).mean().item())
  42.     d2lzh.semilogy(range(1, num_epochs + 1), train_ls, 'epochs', 'loss',
  43.                    range(1, num_epochs + 1), test_ls, ['train', 'test'])
  44.     print('L2 norm of w:', w.norm().item())
  47. if __name__ == "__main__":
  48.     fit_and_plot(lambd=10)

简洁实现：

权重衰减在定义优化器时就可以设置

  1. def fit_and_plot_pytorch(wd):
  2.     net = nn.Linear(num_inputs, 1)
  3.     nn.init.normal_(net.weight, mean=0, std=1)
  4.     nn.init.normal_(net.bias, mean=0, std=1)
  5.     optimizer_w = torch.optim.SGD(params=[net.weight], lr=lr, weight_decay=wd)
  6.     optimizer_b = torch.optim.SGD(params=[net.bias], lr=lr)
  8.     train_ls, test_ls = [], []
  9.     for _ in range(num_epochs):
  10.         for x, y in train_iter:
  11.             l = loss(net(x), y).mean()
  12.             optimizer_w.zero_grad()
  13.             optimizer_b.zero_grad()
  15.             l.backward()
  17.             optimizer_w.step()
  18.             optimizer_b.step()
  19.         train_ls.append(loss(net(train_features), train_labels).mean().item())
  20.         test_ls.append(loss(net(test_features), test_labels).mean().item())
  21.     d2lzh.semilogy(range(1, num_epochs + 1), train_ls, 'epochs', 'loss',
  22.                    range(1, num_epochs + 1), test_ls, ['train', 'test'])
  23.     print('L2 norm of w:', net.weight.data.norm().item())
  26. if __name__ == "__main__":
  27.     fit_and_plot_pytorch(0)
  28.     fit_and_plot_pytorch(10)

结果跟手动实现差不多

  1. L2 norm of w: 13.253257751464844
  2. L2 norm of w: 0.022790107876062393

image.png
image.png

小结

  • 正则化通过为模型损失函数添加惩罚项使学出的模型参数值较小，是应对过拟合的常用手段。
  • 权重衰减等价于L2L_2_L_2范数正则化，通常会使学到的权重参数的元素较接近0。
  • 权重衰减可以通过优化器中的weight_decay超参数来指定。
  • 可以定义多个优化器实例对不同的模型参数使用不同的迭代方法。