深度残差收缩网络其实是一种通用的特征学习方法，是深度残差网络ResNet、注意力机制和软阈值化的集成，可以用于图像分类。本文采用TensorFlow 1.0和TFLearn 0.3.2，编写了图像分类的程序，采用的图像数据为CIFAR-10。CIFAR-10是一个非常常用的图像数据集，包含10个类别的图像。可以在这个网址找到具体介绍：https://www.cs.toronto.edu/~kriz/cifar.html
    1819738-20190927164458588-147256357.png

    参照ResNet代码（https://github.com/tflearn/tflearn/blob/master/examples/images/residual_network_cifar10.py），所编写的深度残差收缩网络的代码如下： 

    1. #!/usr/bin/env python3
    2. # -*- coding: utf-8 -*-
    3. """
    4. Created on Mon Dec 23 21:23:09 2019
    6. M. Zhao, S. Zhong, X. Fu, B. Tang, M. Pecht, Deep Residual Shrinkage Networks for Fault Diagnosis,
    7. IEEE Transactions on Industrial Informatics, 2019, DOI: 10.1109/TII.2019.2943898
    9. @author: super_9527
    10. """
    12. from __future__ import division, print_function, absolute_import
    14. import tflearn
    15. import numpy as np
    16. import tensorflow as tf
    17. from tflearn.layers.conv import conv_2d
    19. # Data loading
    20. from tflearn.datasets import cifar10
    21. (X, Y), (testX, testY) = cifar10.load_data()
    23. # Add noise
    24. X = X + np.random.random((50000, 32, 32, 3))*0.1
    25. testX = testX + np.random.random((10000, 32, 32, 3))*0.1
    27. # Transform labels to one-hot format
    28. Y = tflearn.data_utils.to_categorical(Y,10)
    29. testY = tflearn.data_utils.to_categorical(testY,10)
    31. def residual_shrinkage_block(incoming, nb_blocks, out_channels, downsample=False,
    32.                    downsample_strides=2, activation='relu', batch_norm=True,
    33.                    bias=True, weights_init='variance_scaling',
    34.                    bias_init='zeros', regularizer='L2', weight_decay=0.0001,
    35.                    trainable=True, restore=True, reuse=False, scope=None,
    36.                    name="ResidualBlock"):
    38.     # residual shrinkage blocks with channel-wise thresholds
    40.     residual = incoming
    41.     in_channels = incoming.get_shape().as_list()[-1]
    43.     # Variable Scope fix for older TF
    44.     try:
    45.         vscope = tf.variable_scope(scope, default_name=name, values=[incoming],
    46.                                    reuse=reuse)
    47.     except Exception:
    48.         vscope = tf.variable_op_scope([incoming], scope, name, reuse=reuse)
    50.     with vscope as scope:
    51.         name = scope.name #TODO
    53.         for i in range(nb_blocks):
    55.             identity = residual
    57.             if not downsample:
    58.                 downsample_strides = 1
    60.             if batch_norm:
    61.                 residual = tflearn.batch_normalization(residual)
    62.             residual = tflearn.activation(residual, activation)
    63.             residual = conv_2d(residual, out_channels, 3,
    64.                              downsample_strides, 'same', 'linear',
    65.                              bias, weights_init, bias_init,
    66.                              regularizer, weight_decay, trainable,
    67.                              restore)
    69.             if batch_norm:
    70.                 residual = tflearn.batch_normalization(residual)
    71.             residual = tflearn.activation(residual, activation)
    72.             residual = conv_2d(residual, out_channels, 3, 1, 'same',
    73.                              'linear', bias, weights_init,
    74.                              bias_init, regularizer, weight_decay,
    75.                              trainable, restore)
    77.             # get thresholds and apply thresholding
    78.             abs_mean = tf.reduce_mean(tf.reduce_mean(tf.abs(residual),axis=2,keep_dims=True),axis=1,keep_dims=True)
    79.             scales = tflearn.fully_connected(abs_mean, out_channels//4, activation='linear',regularizer='L2',weight_decay=0.0001,weights_init='variance_scaling')
    80.             scales = tflearn.batch_normalization(scales)
    81.             scales = tflearn.activation(scales, 'relu')
    82.             scales = tflearn.fully_connected(scales, out_channels, activation='linear',regularizer='L2',weight_decay=0.0001,weights_init='variance_scaling')
    83.             scales = tf.expand_dims(tf.expand_dims(scales,axis=1),axis=1)
    84.             thres = tf.multiply(abs_mean,tflearn.activations.sigmoid(scales))
    85.             # soft thresholding
    86.             residual = tf.multiply(tf.sign(residual), tf.maximum(tf.abs(residual)-thres,0))
    89.             # Downsampling
    90.             if downsample_strides > 1:
    91.                 identity = tflearn.avg_pool_2d(identity, 1,
    92.                                                downsample_strides)
    94.             # Projection to new dimension
    95.             if in_channels != out_channels:
    96.                 if (out_channels - in_channels) % 2 == 0:
    97.                     ch = (out_channels - in_channels)//2
    98.                     identity = tf.pad(identity,
    99.                                       [[0, 0], [0, 0], [0, 0], [ch, ch]])
    100.                 else:
    101.                     ch = (out_channels - in_channels)//2
    102.                     identity = tf.pad(identity,
    103.                                       [[0, 0], [0, 0], [0, 0], [ch, ch+1]])
    104.                 in_channels = out_channels
    106.             residual = residual + identity
    108.     return residual
    111. # Real-time data preprocessing
    112. img_prep = tflearn.ImagePreprocessing()
    113. img_prep.add_featurewise_zero_center(per_channel=True)
    115. # Real-time data augmentation
    116. img_aug = tflearn.ImageAugmentation()
    117. img_aug.add_random_flip_leftright()
    118. img_aug.add_random_crop([32, 32], padding=4)
    120. # Building Deep Residual Shrinkage Network
    121. net = tflearn.input_data(shape=[None, 32, 32, 3],
    122.                          data_preprocessing=img_prep,
    123.                          data_augmentation=img_aug)
    124. net = tflearn.conv_2d(net, 16, 3, regularizer='L2', weight_decay=0.0001)
    125. net = residual_shrinkage_block(net, 1, 16)
    126. net = residual_shrinkage_block(net, 1, 32, downsample=True)
    127. net = residual_shrinkage_block(net, 1, 32, downsample=True)
    128. net = tflearn.batch_normalization(net)
    129. net = tflearn.activation(net, 'relu')
    130. net = tflearn.global_avg_pool(net)
    131. # Regression
    132. net = tflearn.fully_connected(net, 10, activation='softmax')
    133. mom = tflearn.Momentum(0.1, lr_decay=0.1, decay_step=20000, staircase=True)
    134. net = tflearn.regression(net, optimizer=mom, loss='categorical_crossentropy')
    135. # Training
    136. model = tflearn.DNN(net, checkpoint_path='model_cifar10',
    137.                     max_checkpoints=10, tensorboard_verbose=0,
    138.                     clip_gradients=0.)
    140. model.fit(X, Y, n_epoch=100, snapshot_epoch=False, snapshot_step=500,
    141.           show_metric=True, batch_size=100, shuffle=True, run_id='model_cifar10')
    143. training_acc = model.evaluate(X, Y)[0]
    144. validation_acc = model.evaluate(testX, testY)[0]

    上面的代码构建了一个小型的深度残差收缩网络，只含有3个基本残差收缩模块，其他的超参数也未进行优化。如果为了追求更高的准确率的话，可以适当增加深度，增加训练迭代次数，以及适当调整超参数。

    前五篇的内容：

    深度残差收缩网络：（一）背景知识 https://www.cnblogs.com/yc-9527/p/11598844.html

    深度残差收缩网络：（二）整体思路 https://www.cnblogs.com/yc-9527/p/11601322.html

    深度残差收缩网络：（三）网络结构 https://www.cnblogs.com/yc-9527/p/11603320.html

    深度残差收缩网络：（四）注意力机制下的阈值设置 https://www.cnblogs.com/yc-9527/p/11604082.html

    深度残差收缩网络：（五）实验验证 https://www.cnblogs.com/yc-9527/p/11610073.html

    原文的链接：

    M. Zhao, S. Zhong, X. Fu, B. Tang, and M. Pecht, “Deep Residual Shrinkage Networks for Fault Diagnosis,” IEEE Transactions on Industrial Informatics, 2019, DOI: 10.1109/TII.2019.2943898

    https://ieeexplore.ieee.org/document/8850096