《深度学习之Pytorch实战计算机视觉》阅读笔记 第8章：图像风格迁移实战

1. 图像风格迁移

首先选取一幅图像作为基准图像，又称为内容图像，然后选取另一幅或者图像作为希望获取相应风格的图像，称其为风格图像。图像风格迁移就是保证内容图像的内容完整性的前提下，将风格图像的风格融入内容图像中，使内容图像的原始风格最后发生转变，最后的输出的图像呈现的是内容图像和风格图像之间的理想融合。

2. 图像内容损失

1）图像的内容损失

  1. # 使用均方误差作为损失函数
  2. import torch
  3. from torch import nn
  5. class Content_loss(torch.nn.Module):
  6.     def __init__(self,weight,target):
  7.         super(Content_loss,self).__init__()
  8.         self.weight=weight
  9.         self.target=target.detach()*weight
  10.         # detach()对提取到的内容进行锁定，不需要进行梯度
  11.         self.loss_fn=torch.nn.MSELoss()
  13.     def forward(self,input):
  14.         self.loss=self.loss_fn(input*self.weight,self.target)
  15.         return input
  17.     def backward(self):
  18.         self.loss.backward(retain_graph=True)
  19.         return self.loss

2）图像的风格损失

  1. # 使用均方误差作为损失函数
  2. class Style_loss(torch.nn.Module):
  3.     def __init__(self,weight,target):
  4.         super(Style_loss,self).__init__()
  5.         self.weight=weight
  6.         self.target=target.detach()*weight
  7.         self.loss_fn=torch.nn.MSELoss()
  8.         # 格拉姆矩阵，矩阵的内积运算，放大“风格”
  9.         self.gram=Gram_matrix()
  11.     def forward(self,input):
  12.         self.Gram=self.gram(input.clone())
  13.         self.Gram.mul_(self.weight)
  14.         self.loss=self.loss_fn(self.Gram,self.target)
  15.         return input
  17.     def backward(self):
  18.         self.loss.backward(retain_graph=True)
  19.         return self.loss
  21.  class Gram_matrix(torch.nn.Module):
  22.     def forward(self,input):
  23.         a,b,c,d=input.size()
  24.         feature=input.view(a*b,c*d)
  25.         gram=torch.mm(feature,feature.t())
  26.         return gram.div(a*b*c*d)

3. 模型搭建

1）特征提取

  1. cnn = models.vgg16(pretrained=True).features
  2. cnn = cnn.cuda()
  3. # 分别指定提取内容特征和风格特征的卷积层
  4. content_layer = ['Conv_3']
  5. style_layer = ['Conv_1', 'Conv_2', 'Conv_3', 'Conv_4']

2）模型迁移

  1. index = 1
  2. for layer in list(model)[:8]:
  3.     if isinstance(layer, torch.nn.Conv2d):
  4.         name = 'Conv_' + str(index)
  5.         new_model.add_module(name, layer)
  6.         if name in content_layer:
  7.             target = new_model(content_image).clone()
  8.             content_loss = Content_loss(content_weight, target)
  9.             new_model.add_module('content_loss_' + str(index), content_loss)
  10.             content_losses.append(content_loss)
  12.         if name in style_layer:
  13.             target = new_model(style_image).clone()
  14.             target = gram(target)
  15.             style_loss = Style_loss(style_weight, target)
  16.             new_model.add_module('style_loss_' + str(index), style_loss)
  17.             style_losses.append(style_loss)
  19.     if isinstance(layer, torch.nn.ReLU):
  20.         name = 'Relu_' + str(index)
  21.         new_model.add_module(name, layer)
  22.         index += 1
  24.     if isinstance(layer, torch.nn.MaxPool2d):
  25.         name = 'MaxPool_' + str(index)
  26.         new_model.add_module(name, layer)

此时，模型输出结果：
20190828094519.png

3）参数优化

  1. input_img = content_image.clone()
  2. parameter = torch.nn.Parameter(input_img.data)
  3. optimizer = torch.optim.LBFGS([parameter])

4. 训练

  1. epoch_n = 300
  2. epoch = [0]
  3. while epoch[0] <= epoch_n:
  5.     def closure():
  6.         optimizer.zero_grad()
  7.         style_score = 0
  8.         content_score = 0
  9.         parameter.data.clamp_(0, 1)
  10.         new_model(parameter)
  12.         for sl in style_losses:
  13.             style_score += sl.backward()
  15.         for cl in content_losses:
  16.             content_score += cl.backward()
  18.         epoch[0] += 1
  19.         if epoch[0] % 50 == 0:
  20.             print('Epoch:{} Style Loss:{:4f} Content Loss:{:4f}'.format(
  21.                 epoch[0], style_score.item(), content_score.item()))
  23.         return style_score + content_score
  25.     optimizer.step(closure)