20180114205444652.png

1. PyTorch实现里面都设置 bias=False ?

因为使用了Batch Normalization，而其对隐藏层 ResNet - 图4 有去均值的操作，所以这里的常数项 ResNet - 图5可以消去，因为Batch Normalization有一个操作ResNet - 图6，所以上面ResNet - 图7的数值效果是能由ResNet - 图8所替代的，因此我们在使用Batch Norm的时候，可以忽略各隐藏层的常数项 ResNet - 图9 。这样在使用梯度下降算法时，只用对 ResNet - 图10 ， ResNet - 图11和 ResNet - 图12 进行迭代更新

2. ResNet结构为什么可以解决深度网络退化问题?

因为ResNet更加容易拟合恒等映射，因为ResNet的结构使得网络具有与学习恒等映射的能力，同时也具有学习其他映射的能力。因此ResNet的结构要优于传统的卷积网络（plain networks）结构。

3. ResNet的网络结构

ResNet网络是参考了VGG19网络，在其基础上进行了修改，并通过短路机制加入了残差单元，如下图所示。变化主要体现在ResNet直接使用 stride=2 的卷积做下采样，并且用 global average pool 层替换了全连接层。ResNet的一个重要设计原则是：当feature map大小降低一半时，feature map的数量增加一倍，这保持了网络层的复杂度。从下图中可以看到，ResNet相比普通网络每两层间增加了短路机制，这就形成了残差学习，其中虚线表示feature map数量发生了改变。下图展示的34-layer的ResNet。从最上面的表中可以看到，对于18-layer和34-layer的ResNet，其进行的两层间的残差学习，当网络更深时，其进行的是三层间的残差学习，三层卷积核分别是1x1，3x3和1x1，一个值得注意的是隐含层的feature map数量是比较小的，并且是输出feature map数量的1/4。

20200525143938770.png

4. ResNet-50 网络结构

image-20200805123213120.png

参考文档

ResNet 介绍

https://www.cnblogs.com/long5683/p/12957042.html
https://www.bilibili.com/video/av91981420

ResNet 作者现场演讲

https://zhuanlan.zhihu.com/p/54072011

ResNet 论文详解

https://zhuanlan.zhihu.com/p/462190341

ResNet 要解决的是什么问题

https://www.cnblogs.com/shine-lee/p/12363488.html

ResNet 为什么要1*1卷积

https://zhuanlan.zhihu.com/p/364527122

ResNet 的每个block 里，11，33，1*1卷积这样交替堆叠有什么好处和作用？

https://www.zhihu.com/question/384884504

偏执 bias

https://www.cnblogs.com/wanghui-garcia/p/10775860.html
https://blog.csdn.net/qq_22764813/article/details/97940302

ResNet 网络结构

https://zhoujiahuan.github.io/2020/06/11/ResNet-networks/

调参经验

https://www.cnblogs.com/mfryf/p/11393653.html

代码实现

  1. import torch.nn as nn
  2. import torch
  3. from torch.nn import functional as F
  6. class ResNetBasicBlock(nn.Module):
  7.     def __init__(self, in_channels, out_channels, stride):
  8.         super(ResNetBasicBlock, self).__init__()
  9.         self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=3, stride=stride, padding=1)
  10.         self.bn1 = nn.BatchNorm2d(out_channels)
  11.         self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, stride=stride, padding=1)
  12.         self.bn2 = nn.BatchNorm2d(out_channels)
  14.     def forward(self, x):
  15.         output = self.conv1(x)
  16.         output = F.relu(self.bn1(output))
  17.         output = self.conv2(output)
  18.         output = self.bn2(output)
  19.         return F.relu(x + output)
  22. class ResNetDownBlock(nn.Module):
  23.     def __init__(self, in_channels, out_channels, stride):
  24.         super(ResNetDownBlock, self).__init__()
  25.         self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=3, stride=stride[0], padding=1)
  26.         self.bn1 = nn.BatchNorm2d(out_channels)
  27.         self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, stride=stride[1], padding=1)
  28.         self.bn2 = nn.BatchNorm2d(out_channels)
  29.         self.extra = nn.Sequential(
  30.             nn.Conv2d(in_channels, out_channels, kernel_size=1, stride=stride[0], padding=0),
  31.             nn.BatchNorm2d(out_channels)
  32.         )
  34.     def forward(self, x):
  35.         extra_x = self.extra(x)
  37.         output = self.conv1(x)
  38.         output = F.relu(self.bn1(output))
  39.         output = self.conv2(output)
  40.         output = self.bn2(output)
  41.         return F.relu(extra_x + output)
  44. class ResNet18(nn.Module):
  45.     """ ResNet18 """
  46.     def __init__(self, out_channels):
  47.         super(ResNet18, self).__init__()
  48.         self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, bias=False)
  49.         self.bn1 = nn.BatchNorm2d(64)
  50.         self.maxPool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
  52.         self.layer1 = nn.Sequential(ResNetBasicBlock(64, 64, 1), ResNetBasicBlock(64, 64, 1))
  53.         self.layer2 = nn.Sequential(ResNetDownBlock(64, 128, [2, 1]), ResNetBasicBlock(128, 128, 1))
  54.         self.layer3 = nn.Sequential(ResNetDownBlock(128, 256, [2, 1]), ResNetBasicBlock(256, 256, 1))
  55.         self.layer4 = nn.Sequential(ResNetDownBlock(256, 512, [2, 1]), ResNetBasicBlock(512, 512, 1))
  56.         self.avgPool = nn.AdaptiveAvgPool2d(output_size=(1, 1))
  57.         self.fc = nn.Linear(512, out_channels)
  59.     def forward(self, x):
  60.         out = self.conv1(x)
  61.         out = self.bn1(out)
  62.         out = self.maxPool(F.relu(out))
  64.         out = self.layer1(out)
  65.         out = self.layer2(out)
  66.         out = self.layer3(out)
  67.         out = self.layer4(out)
  68.         out = self.avgPool(out)
  69.         out = torch.flatten(out, 1)
  70.         out = self.fc(out)
  71.         return out
  1. import torch.nn as nn
  2. import torch
  5. class Bottleneck(nn.Module):
  6.     def __init__(self, in_channels, middle_channels, out_channels, stride):
  7.         super(Bottleneck, self).__init__()
  9.         self.bottleneck = nn.Sequential(
  10.             nn.Conv2d(in_channels, middle_channels, kernel_size=1, stride=1, bias=False),
  11.             nn.BatchNorm2d(middle_channels),
  12.             nn.ReLU(),
  13.             nn.Conv2d(middle_channels, middle_channels, kernel_size=3, stride=stride, padding=1, bias=False),
  14.             nn.BatchNorm2d(middle_channels),
  15.             nn.ReLU(),
  16.             nn.Conv2d(middle_channels, out_channels, kernel_size=1, stride=1, bias=False),
  17.             nn.BatchNorm2d(out_channels)
  18.         )
  20.         self.down = False
  21.         if in_channels != out_channels:
  22.             self.down = True
  23.             self.downSample = nn.Sequential(
  24.                 nn.Conv2d(in_channels, out_channels, kernel_size=1, stride=stride, bias=False),
  25.                 nn.BatchNorm2d(out_channels)
  26.             )
  27.         self.relu = nn.ReLU()
  29.     def forward(self, x):
  30.         old_x = x
  31.         output = self.bottleneck(x)
  32.         if self.down:
  33.             old_x = self.downSample(old_x)
  35.         output += old_x
  36.         output = self.relu(output)
  37.         return output
  40. class ResNet101(nn.Module):
  41.     """ ResNet101 """
  43.     def __init__(self, out_channels):
  44.         super(ResNet101, self).__init__()
  46.         self.conv1 = nn.Sequential(
  47.             nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, bias=False),
  48.             nn.BatchNorm2d(64),
  49.             nn.ReLU(),
  50.             nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
  51.         )
  53.         self.layer1 = self.make_layer(in_channels=64, middle_channels=64, out_channels=256, stride=1, count=3)
  54.         self.layer2 = self.make_layer(in_channels=256, middle_channels=128, out_channels=512, stride=2, count=4)
  55.         self.layer3 = self.make_layer(in_channels=512, middle_channels=256, out_channels=1024, stride=2, count=23)
  56.         self.layer4 = self.make_layer(in_channels=1024, middle_channels=512, out_channels=2048, stride=2, count=3)
  57.         self.avgPool = nn.AdaptiveAvgPool2d(output_size=(1, 1))
  58.         self.fc = nn.Linear(2048, out_channels)
  60.     def make_layer(self, in_channels, middle_channels, out_channels, stride, count):
  61.         layers = []
  62.         for i in range(count):
  63.             if i == 0:
  64.                 layers.append(Bottleneck(in_channels, middle_channels, out_channels, stride))
  65.             else:
  66.                 layers.append(Bottleneck(out_channels, middle_channels, out_channels, 1))
  67.         return nn.Sequential(*layers)
  69.     def forward(self, x):
  70.         out = self.conv1(x)
  71.         out = self.layer1(out)
  72.         out = self.layer2(out)
  73.         out = self.layer3(out)
  74.         out = self.layer4(out)
  75.         out = self.avgPool(out)
  76.         out = torch.flatten(out, 1)
  77.         out = self.fc(out)
  78.         return out