参考来源：
Pytorch 使用 ReduceLROnPlateau 来更新学习率

解析说明

  1. torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, 
  2.                                            mode='min', 
  3.                                            factor=0.1, 
  4.                                            patience=10, 
  5.                                            verbose=False, 
  6.                                            threshold=0.0001, 
  7.                                            threshold_mode='rel', 
  8.                                            cooldown=0, 
  9.                                            min_lr=0, 
  10.                                            eps=1e-08
  11.                                           )

在发现 loss 不再降低或者 acc 不再提高之后，降低学习率。各参数意义如下：

  • **mode**：'min' 模式检测 metric 是否不再减小，'max' 模式检测 metric 是否不再增大；
  • **factor**：触发条件后 lr*=factor；
  • **patience**：不再减小（或增大）的累计次数；
  • **verbose**：触发条件后 print；
  • **threshold**：只关注超过阈值的显著变化；
  • **threshold_mode**：有 rel 和 abs 两种阈值计算模式；
    • **rel** 规则：max 模式下如果超过 best(1+threshold) 为显著，min 模式下如果低于 best(1-threshold) 为显著；
    • **abs** 规则：max模式下如果超过 best+threshold 为显著，min 模式下如果低于 best-threshold 为显著；
  • **cooldown**：触发一次条件后，等待一定 epoch 再进行检测，避免 lr 下降过速；
  • **min_lr**：最小的允许 lr；
  • **eps**：如果新旧 lr 之间的差异小于1e-8，则忽略此次更新。

例子，如图所示的 y 轴为 lr，x 为调整的次序，初始的学习率为 0.0009575。则学习率的方程为：lr = 0.0009575 * (0.35)^x
使用 ReduceLROnPlateau 来更新学习率——Pytorch - 图1

  1. import math 
  2. import matplotlib.pyplot as plt
  3. #%matplotlib inline
  5. x = 0 
  6. o = []
  7. p = []
  8. o.append(0)
  9. p.append(0.0009575)
  10. while(x < 8):
  11.     x += 1
  12.     y = 0.0009575 * math.pow(0.35,x)
  13.     o.append(x)
  14.     p.append(y)
  15.     print('%d:   %.50f' %(x,y))
  17. plt.plot(o,p,c='red',label='test') #分别为x,y轴对应数据,c:color,label
  18. plt.legend(loc='best')  # 显示label,loc为显示位置(best为系统认为最好的位置)
  19. plt.show()

难点

我感觉这里面最难的时这几个参数的选择，第一个是初始的学习率（我目前接触的 miniest 和下面的图像分类貌似都是 0.001，我这里训练调整时才发现自己设置的为 0.0009575，这个值是上一个实验忘更改了，但发现结果不错，第一次运行该代码接近到 0.001 这么小的损失值）,这里面的乘积系数以及判断说多少次没有减少（增加）后决定变换学习率都是难以估计的。我自己的最好方法是先按默认不变的 0.001 来训练一下（结合 tensoarboard ）观察从哪里开始出现问题就可以从这里来确定次数，而乘积系数，个人感觉还是用上面的代码来获取一个较为平滑且变化极小的数字来作为选择。建议在做这种测试时可以把模型先备份一下以免浪费过多的时间！

例子

该例子初始学习率为 0.0009575，乘积项系数为：0.35，在我的例子中 x 变化的条件是：累计 125 次没有减小则 x 加 1；自己训练在第一次 lr 变化后（从 0.0009575 变化到 0.00011729）损失值慢慢取向于 0.001（如第一张图所示），准确率达到 69%；
使用 ReduceLROnPlateau 来更新学习率——Pytorch - 图2

  1. import torch
  2. import torchvision
  3. import torchvision.transforms as transforms
  4. import matplotlib.pyplot as plt
  5. import numpy as np
  6. import torch.nn as nn
  7. import torch.nn.functional as F
  8. import torch.optim as optim
  9. from datetime import datetime
  10. from torch.utils.tensorboard import SummaryWriter
  11. from torch.optim import *
  14. PATH = './cifar_net_tensorboard_net_width_200_and_chang_lr_by_decrease_0_35^x.pth'  # 保存模型地址
  16. transform = transforms.Compose(
  17.     [transforms.ToTensor(),
  18.      transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
  20. trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
  21.                                         download=True, transform=transform)
  22. trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
  23.                                           shuffle=True, num_workers=0)
  25. testset = torchvision.datasets.CIFAR10(root='./data', train=False,
  26.                                        download=True, transform=transform)
  27. testloader = torch.utils.data.DataLoader(testset, batch_size=4,
  28.                                          shuffle=False, num_workers=0)
  30. classes = ('plane', 'car', 'bird', 'cat',
  31.            'deer', 'dog', 'frog', 'horse', 'ship', 'truck')
  34. device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
  36. # Assuming that we are on a CUDA machine, this should print a CUDA device:
  38. print(device)
  40. print("获取一些随机训练数据")
  41. # get some random training images
  42. dataiter = iter(trainloader)
  43. images, labels = dataiter.next()
  46. # functions to show an image
  47. def imshow(img):
  48.     img = img / 2 + 0.5     # unnormalize
  49.     npimg = img.numpy()
  50.     plt.imshow(np.transpose(npimg, (1, 2, 0)))
  51.     plt.show()
  54. # show images
  55. imshow(torchvision.utils.make_grid(images))
  56. # print labels
  57. print(' '.join('%5s' % classes[labels[j]] for j in range(4)))
  58. print("**********************")
  60. # 设置一个tensorborad
  61. # helper function to show an image
  62. # (used in the `plot_classes_preds` function below)
  63. def matplotlib_imshow(img, one_channel=False):
  64.     if one_channel:
  65.         img = img.mean(dim=0)
  66.     img = img / 2 + 0.5     # unnormalize
  67.     npimg = img.cpu().numpy()
  68.     if one_channel:
  69.         plt.imshow(npimg, cmap="Greys")
  70.     else:
  71.         plt.imshow(np.transpose(npimg, (1, 2, 0)))    
  73. # 设置tensorBoard
  74. # default `log_dir` is "runs" - we'll be more specific here
  75. writer = SummaryWriter('runs/train')
  77. # get some random training images
  78. dataiter = iter(trainloader)
  79. images, labels = dataiter.next()
  81. # create grid of images
  82. img_grid = torchvision.utils.make_grid(images)
  84. # show images
  85. # matplotlib_imshow(img_grid, one_channel=True)
  86. imshow(img_grid)
  88. # write to tensorboard
  89. # writer.add_image('imag_classify', img_grid)
  91. # Tracking model training with TensorBoard
  92. # helper functions
  94. def images_to_probs(net, images):
  95.     '''
  96.     Generates predictions and corresponding probabilities from a trained
  97.     network and a list of images
  98.     '''
  99.     output = net(images)
  100.     # convert output probabilities to predicted class
  101.     _, preds_tensor = torch.max(output, 1)
  102.     # preds = np.squeeze(preds_tensor.numpy())
  103.     preds = np.squeeze(preds_tensor.cpu().numpy())
  104.     return preds, [F.softmax(el, dim=0)[i].item() for i, el in zip(preds, output)]
  107. def plot_classes_preds(net, images, labels):
  108.     preds, probs = images_to_probs(net, images)
  109.     # plot the images in the batch, along with predicted and true labels
  110.     fig = plt.figure(figsize=(12, 48))
  111.     for idx in np.arange(4):
  112.         ax = fig.add_subplot(1, 4, idx+1, xticks=[], yticks=[])
  113.         matplotlib_imshow(images[idx], one_channel=True)
  114.         ax.set_title("{0}, {1:.1f}%\n(label: {2})".format(
  115.             classes[preds[idx]],
  116.             probs[idx] * 100.0,
  117.             classes[labels[idx]]),
  118.                     color=("green" if preds[idx]==labels[idx].item() else "red"))
  119.     return fig
  121. #
  123. class Net(nn.Module):
  124.     def __init__(self):
  125.         super(Net, self).__init__()
  126.         self.conv1 = nn.Conv2d(3, 200, 5)
  127.         self.pool = nn.MaxPool2d(2, 2)
  128.         self.conv2 = nn.Conv2d(200, 16, 5)
  129.         self.fc1 = nn.Linear(16 * 5 * 5, 120)
  130.         self.fc2 = nn.Linear(120, 84)
  131.         self.fc3 = nn.Linear(84, 10)
  133.     def forward(self, x):
  134.         x = self.pool(F.relu(self.conv1(x)))
  135.         x = self.pool(F.relu(self.conv2(x)))
  136.         x = x.view(-1, 16 * 5 * 5)
  137.         x = F.relu(self.fc1(x))
  138.         x = F.relu(self.fc2(x))
  139.         x = self.fc3(x)
  140.         return x
  142. net = Net()
  143. # # 把net结构可视化出来
  144. writer.add_graph(net, images)
  145. net.to(device)
  147. try:
  148.     net.load_state_dict(torch.load(PATH))
  149.     print("Modle file load successful !")
  150. except:
  151.     print("no model file,it will creat a new file!")
  153. # 训练
  154. print("训练")
  155. criterion = nn.CrossEntropyLoss()
  156. # optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
  158. #在发现loss不再降低或者acc不再提高之后，降低学习率。
  159. optimizer = torch.optim.SGD(net.parameters(), lr=0.0009575, momentum=0.9)
  160. scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min',factor=0.35,verbose=1,min_lr=0.0001,patience=125)
  162. startTime = datetime.now()
  163. for epoch in range(200):  # loop over the dataset multiple times
  164.     running_loss = 0.0
  165.     for i, data in enumerate(trainloader, 0):
  166.         # get the inputs; data is a list of [inputs, labels]
  167. #         inputs, labels = data
  169.         inputs, labels = data[0].to(device), data[1].to(device)
  171.         # zero the parameter gradients
  172.         optimizer.zero_grad() #将参数的grad值初始化为0
  174.         # forward + backward + optimize
  175.         outputs = net(inputs)
  176.         loss = criterion(outputs, labels) #计算损失
  177.         loss.backward() # 反向传播
  178.         optimizer.step() # 反向传播求梯度
  180.         # print statistics
  181.         running_loss += loss.item()
  182.         if i % 2000 == 1999:    # print every 2000 mini-batches
  183.             now_loss = running_loss / 2000 # 2000mini-batches 的平均损失率
  184.             print('[%d, %5d] loss: %.3f' %
  185.                   (epoch + 1, i + 1, now_loss))
  186.             # now_loss = running_loss / 2000
  188.             scheduler.step(now_loss)
  190.                         # 把数据写入tensorflow
  191.             # ...log the running loss
  192.             writer.add_scalar('image training loss on net width 200 chang_lr_by_decrease',
  193.                             now_loss,
  194.                             epoch * len(trainloader) + i)
  196.             writer.add_scalar('learning rate on net width 200 chang_lr_by_decrease',
  197.                             optimizer.state_dict()['param_groups'][0]['lr'],
  198.                             epoch * len(trainloader) + i)                            
  200.             running_loss = 0.0
  202. torch.save(net.state_dict(), PATH)
  204. print('Finished Training')
  205. print("***************************")
  206. print("***************************")
  207. print("***************************")
  208. print("Time taken:", datetime.now() - startTime)
  209. print("***************************")
  210. print("***************************")
  211. print("***************************")
  215. #获取一些随机测试数据
  216. print("获取一些随机测试数据")
  217. dataiter = iter(testloader)
  218. images, labels = dataiter.next()
  220. # print images
  221. imshow(torchvision.utils.make_grid(images))
  222. print('GroundTruth: ', ' '.join('%5s' % classes[labels[j]] for j in range(4)))
  224. # 恢复模型并测试
  225. net = Net()
  226. net.load_state_dict(torch.load(PATH))
  228. outputs = net(images)
  230. _, predicted = torch.max(outputs, 1)
  232. print('Predicted: ', ' '.join('%5s' % classes[predicted[j]]
  233.                               for j in range(4)))
  235. print("**********************")
  236. print("输出训练得到的准确度")
  237. # 输出训练得到的准确度
  238. correct = 0
  239. total = 0
  240. with torch.no_grad():
  241.     for data in testloader:
  242.         images, labels = data
  243.         outputs = net(images)
  244.         _, predicted = torch.max(outputs.data, 1)
  245.         total += labels.size(0)
  246.         correct += (predicted == labels).sum().item()
  248. print('Accuracy of the network on the 10000 test images: %d %%' % (
  249.     100 * correct / total))
  251. class_correct = list(0. for i in range(10))
  252. class_total = list(0. for i in range(10))
  253. with torch.no_grad():
  254.     for data in testloader:
  255.         images, labels = data
  256.         outputs = net(images)
  257.         _, predicted = torch.max(outputs, 1)
  258.         c = (predicted == labels).squeeze()
  259.         for i in range(4):
  260.             label = labels[i]
  261.             class_correct[label] += c[i].item()
  262.             class_total[label] += 1
  264. for i in range(10):
  265.     print('Accuracy of %5s : %2d %%' % (
  266.         classes[i], 100 * class_correct[i] / class_total[i]))