Overall architecture

下面这张图的做法其实就是三个网络，层数递减，输入的分辨率也递减。最后的结果再concat到一起，过一层吧。然后看结果。中间没有任何的交织
Triple-UNet.jpgTriple-UNet2.jpg

What happen?

  1. 怎么对不同的数据都切成patch，然后最后还要组合在一起，然后concat其他俩网络的feature
    a. 每次只输入一个数据，保证分块后的patch最后经过各种操作后还可以组合在一起
    对于这个问题，研究了monai slide_window_inference
    fall_back_tuple() Typically used when user_provided is a tuple of window size provided by the user, default is defined by data, this function returns an updated user_provided with its non-positive
    components replaced by the corresponding components from default.
    dense_patch_slice() Store all slices in list, return all the slice(0:x) in 3 dimension.
    unravel_slice 就是类似返回当前sw_batch_size个索引，每个索引第一部分是在当前的batch中的位置，第二个None(channel)，第三四五就是对应的维度切片。
    importance_map默认是全为1，计算过的位置就会加一个map
  2. 现在的一个新问题是，按照monai unet最后输出的是一个2通道的，那么最后咋把三个结果concat呢？
    a. monai UNet通道数最少得有(a, b)
  3. 数据输入设想，是每个数据都crop成(512, 512, 128)大小的数据。这样最浅层的网络只会降采样一次（按照monai UNet的设计，bottom层不会有降低和提高分辨率的操作）
  4. monai UNet网络结构的设计，所有的block都是会过两次卷积。
    a. encoder部分，就先经过步长为2的卷积，下采样，同时还会两倍通道数。然后再经过步长为1的卷积正常操作，这里也不改变通道的数量。
    b. decoder部分（这里和下图里的不太一样）是先concat然后再ConvTranse, 之后再正常卷积。
    c. 因为原文是第一次不降采样之后每次输入block之前都会降采样。这里是最后一个block不降采样，前面每次都降采样。
  5. S层(16, 32) M层(16, 32, 64) L层(16, 32, 64, 128, 256)
  6. 看一下别的模型怎么解决多输入这个问题的。
    a. 还没看，但是用自己的方法解决了。还是可以看看别的方法
  7. 多监督，把其余几个网络的结果进行深监督，然后最后一层赋予更大的权重（jiadong）
    a. 目前的效果不好
  8. 当前的验证集效果波动非常大，How to solve this
    a. coslr，确实写了，尝试降低下限试试
    b. 数据shuffle一下，训练的时候确实是shuffle了
    c. batch_size 太小了，用gn来解决
    d. 关于patch_size 必须是2的幂这件事
    e. 增加BCE+DiceLoss
    zx’ words:
    1. 用group normalization z所有的都用gn, 除了最后的聚合层
    2. 最下层concat一下
    3. 第三个维度强行resize成一个统一大小，把有label的地方crop出来再resize（确定crop是不是出了问题）

Results part

1. number of parameters

Model parameters
UNet( 4.8M
VNet(Standard) 45.6 M
TripleUNet(2,3,5) 5.1M


2. The experiments result record

1. UNet

id max epoch best val dice(epoch) test dice source
1 500 0.8284(312) 0.8255 .
2 with stable lr 0.6917 
  1. roi_size = (256, 256, 64)     # window size of validation
  2. sw_batch_size = 4           # the batch size to run window slices
  3. val_num = 30
  5. data = PulVessel_lightning(in_dir, batch_size=1, num_workers=4, val_num=val_num,
  6.     cache=True, cache_rate=1, predict_num=1)
  8. max_epochs=500,
  10. self._model = UNet(
  11.             spatial_dims=3,
  12.             in_channels=1,
  13.             out_channels=2,
  14.             channels=(16, 32, 64, 128, 256),
  15.             strides=(2, 2, 2, 2),
  16.             num_res_units=2,
  17.             norm=Norm.BATCH,
  18.         )
  20. self.loss_function = DiceLoss(to_onehot_y=True, sigmoid=True, squared_pred=False, include_background=False)
  21. self.post_pred = AsDiscrete(argmax=True, to_onehot=2)
  22. self.post_label = AsDiscrete(to_onehot=2)
  23. self.dice_metric = DiceMetric(include_background=False, reduction="mean", get_not_nans=False)
  25. def configure_optimizers(self):
  26.   optimizer = torch.optim.Adam(self._model.parameters(), 0.0001)
  27.   lr_scheduler = CosineAnnealingLR(optimizer, eta_min=0.0001/100, T_max=500,
  28.                                             last_epoch=-1, verbose=True)
  29.   return [optimizer], [lr_scheduler]
  32. self.train_transforms = Compose(
  33.             [
  34.                 LoadImaged(keys=["vol", "seg"]),
  35.                 EnsureChannelFirstd(keys=["vol", "seg"]),
  36.                 ScaleIntensityRanged(keys=['vol'], a_min=-900.0, a_max=200,
  37.                                      b_min=0, b_max=1, clip=True),
  38.                 # RandSpatialCropd(keys=["vol", "seg"], roi_size=(1000, 1000, 48), random_size=False),
  39.                 RandCropByPosNegLabeld(keys=["vol", "seg"], label_key='seg', spatial_size=[256, 256, 64], pos=1,
  40.                                        neg=1, num_samples=1),
  41.                 RandFlipd(keys=["vol", "seg"], prob=0.5, spatial_axis=1),
  42.                 RandFlipd(keys=["vol", "seg"], prob=0.5, spatial_axis=2),
  43.                 ToTensord(keys=["vol", "seg"]),
  44.             ]
  45.         )
  47.         self.val_transforms = Compose(
  48.             [
  49.                 LoadImaged(keys=["vol", "seg"]),
  50.                 EnsureChannelFirstd(keys=["vol", "seg"]),
  51.                 ScaleIntensityRanged(keys=['vol'], a_min=-900.0, a_max=200,
  52.                                      b_min=0, b_max=1, clip=True),
  53.                 # RandSpatialCropd(keys=["vol", "seg"], roi_size=(1000, 1000, 48), random_size=False),
  54.                 # RandFlipd(keys=["vol", "seg"], prob=0.5, spatial_axis=1),
  55.                 # RandFlipd(keys=["vol", "seg"], prob=0.5, spatial_axis=2),
  56.                 ToTensord(keys=["vol", "seg"]),
  57.             ]
  58.         )
  60.         self.test_transforms = Compose(
  61.             [
  62.                 LoadImaged(keys=["vol", "seg"]),
  63.                 EnsureChannelFirstd(keys=["vol", "seg"]),
  64.                 ScaleIntensityRanged(keys=['vol'], a_min=-900.0, a_max=200,
  65.                                      b_min=0, b_max=1, clip=True),
  66.                 # RandSpatialCropd(keys=["vol", "seg"], roi_size=(1000, 1000, 48), random_size=False),
  67.                 # RandFlipd(keys=["vol", "seg"], prob=0.5, spatial_axis=1),
  68.                 # RandFlipd(keys=["vol", "seg"], prob=0.5, spatial_axis=2),
  69.                 ToTensord(keys=["vol", "seg"]),
  70.             ]
  71.         )
  72. Acurrent epoch: 499 current mean dice: 0.8048
  73. best mean dice: 0.8284 at epoch: 312
  74. test：The mean prediction loss is 0.17454605525539768 The best performance is 0.1278761625289917
  75. the event folder:/hpc/data/home/bme/liujy3/code/PulVessel_Lightning/UNet/logs/default/version_0

2. VNet

id max epoch best val dice(epoch) test dice source
1 500 0.8027(294) 0.78 VNet1 
  1. self._model = VNet(
  2.             spatial_dims=3,
  3.             in_channels=1,
  4.             out_channels=2,
  5.         ) # standard VNet
  7. self.loss_function = DiceLoss(to_onehot_y=True, sigmoid=True, squared_pred=False, include_background=False)
  8. self.post_pred = AsDiscrete(argmax=True, to_onehot=2)
  9. self.post_label = AsDiscrete(to_onehot=2)
  10. self.dice_metric = DiceMetric(include_background=False, reduction="mean", get_not_nans=False)
  12. def configure_optimizers(self):
  13.     optimizer = torch.optim.Adam(self._model.parameters(), 0.0001)
  14.     lr_scheduler = CosineAnnealingLR(optimizer, eta_min=0.0001/100, T_max=500,
  15.                                      last_epoch=-1, verbose=True)
  16.     return [optimizer], [lr_scheduler]
  18. roi_size = (256, 256, 64)     # window size of validation
  19. sw_batch_size = 4
  20. val_num = 30
  22. data = PulVessel_lightning(in_dir, batch_size=1, num_workers=4, val_num=val_num,
  23.     cache=True, cache_rate=1, predict_num=1)   # for bme cluster
  25. self.train_transforms = Compose(
  26.             [
  27.                 LoadImaged(keys=["vol", "seg"]),
  28.                 EnsureChannelFirstd(keys=["vol", "seg"]),
  29.                 ScaleIntensityRanged(keys=['vol'], a_min=-900.0, a_max=200,
  30.                                      b_min=0, b_max=1, clip=True),
  31.                 # RandSpatialCropd(keys=["vol", "seg"], roi_size=(1000, 1000, 48), random_size=False),
  32.                 RandCropByPosNegLabeld(keys=["vol", "seg"], label_key='seg', spatial_size=[256, 256, 64], pos=1,
  33.                                        neg=1, num_samples=1),
  34.                 RandFlipd(keys=["vol", "seg"], prob=0.5, spatial_axis=1),
  35.                 RandFlipd(keys=["vol", "seg"], prob=0.5, spatial_axis=2),
  36.                 ToTensord(keys=["vol", "seg"]),
  37.             ]
  38.         )
  40.         self.val_transforms = Compose(
  41.             [
  42.                 LoadImaged(keys=["vol", "seg"]),
  43.                 EnsureChannelFirstd(keys=["vol", "seg"]),
  44.                 ScaleIntensityRanged(keys=['vol'], a_min=-900.0, a_max=200,
  45.                                      b_min=0, b_max=1, clip=True),
  46.                 # RandSpatialCropd(keys=["vol", "seg"], roi_size=(1000, 1000, 48), random_size=False),
  47.                 # RandFlipd(keys=["vol", "seg"], prob=0.5, spatial_axis=1),
  48.                 # RandFlipd(keys=["vol", "seg"], prob=0.5, spatial_axis=2),
  49.                 ToTensord(keys=["vol", "seg"]),
  50.             ]
  51.         )
  53.         self.test_transforms = Compose(
  54.             [
  55.                 LoadImaged(keys=["vol", "seg"]),
  56.                 EnsureChannelFirstd(keys=["vol", "seg"]),
  57.                 ScaleIntensityRanged(keys=['vol'], a_min=-900.0, a_max=200,
  58.                                      b_min=0, b_max=1, clip=True),
  59.                 # RandSpatialCropd(keys=["vol", "seg"], roi_size=(1000, 1000, 48), random_size=False),
  60.                 # RandFlipd(keys=["vol", "seg"], prob=0.5, spatial_axis=1),
  61.                 # RandFlipd(keys=["vol", "seg"], prob=0.5, spatial_axis=2),
  62.                 ToTensord(keys=["vol", "seg"]),
  63.             ]
  64.         )
  65. Best loss is 0.2252 at epoch 294
  66. dice:0.8027
  68. test: The mean prediction loss is 0.22441603291419246 The best performance is 0.15201616287231445
  70. the event floder:/hpc/data/home/bme/liujy3/code/PulVessel_Lightning/VNet/logs/default/version_6

3. TripleUNet( sdata->lpath, ldata->spath)

id max epoch best val dice(epoch) test dice source
1 350 0.5070(284) 0.57 TripleU
2 293 0.6308(272) 0.63(0.72best)

  1. ```python roi_size = (-1, -1, 64) # window size of validation sw_batch_size = 4 # the batch size to run window slices val_num = 30 # the number of the validation data

data = PulVessel_lightning(in_dir, batch_size=1, num_workers=4, val_num=val_num, cache=True, cache_rate=1, predict_num=1) # for hp # for bme clust

model

self._modelS = UNet( spatial_dims=3, in_channels=1, out_channels=2, channels=(16, 32), strides=(2,), num_res_units=2, norm=Norm.BATCH, ) self._modelM = UNet( spatial_dims=3, in_channels=1, out_channels=2, channels=(16, 32, 64), strides=(2, 2,), num_res_units=2, norm=Norm.BATCH, ) self._modelL = UNet( spatial_dims=3, in_channels=1, out_channels=2, channels=(16, 32, 64, 128, 256), strides=(2, 2, 2, 2), num_res_units=2, norm=Norm.BATCH, ) self.final_layer = Convolution( spatial_dims=3, in_channels=6, out_channels=2, strides=1, kernel_size=3, act=Act.PRELU, norm=Norm.BATCH, dropout=0.0, bias=True, conv_only=True, is_transposed=True, )

data

output_s = self._modelS(x) output_m = fake_slide_window(x, 1 / 2, self.sw_batch_size, self._modelM) output_l = fake_slide_window(x, 1 / 4, self.sw_batch_size, self._modelL)

self.train_transforms = Compose( [ LoadImaged(keys=[“vol”, “seg”]), EnsureChannelFirstd(keys=[“vol”, “seg”]), ScaleIntensityRanged(keys=[‘vol’], a_min=-900.0, a_max=200, b_min=0, b_max=1, clip=True), RandSpatialCropd(keys=[“vol”, “seg”], roi_size=(512, 512, 128), random_size=False),

  1.             # RandCropByPosNegLabeld(keys=["vol", "seg"], label_key='seg', spatial_size=[512, 512, 128], pos=1,
  2.             #                        neg=1, num_samples=4),
  3.             RandFlipd(keys=["vol", "seg"], prob=0.5, spatial_axis=1),
  4.             RandFlipd(keys=["vol", "seg"], prob=0.5, spatial_axis=2),
  5.             ToTensord(keys=["vol", "seg"]),
  6.         ]
  7.     )

self.val_transforms = Compose( [ LoadImaged(keys=[“vol”, “seg”]), EnsureChannelFirstd(keys=[“vol”, “seg”]), ScaleIntensityRanged(keys=[‘vol’], a_min=-900.0, a_max=200, b_min=0, b_max=1, clip=True),

  1.     # RandSpatialCropd(keys=["vol", "seg"], roi_size=(1000, 1000, 48), random_size=False),
  2.     # RandFlipd(keys=["vol", "seg"], prob=0.5, spatial_axis=1),
  3.     # RandFlipd(keys=["vol", "seg"], prob=0.5, spatial_axis=2),
  4.     ToTensord(keys=["vol", "seg"]),
  5. ]

)

self.test_transforms = Compose( [ LoadImaged(keys=[“vol”, “seg”]), EnsureChannelFirstd(keys=[“vol”, “seg”]), ScaleIntensityRanged(keys=[‘vol’], a_min=-900.0, a_max=200, b_min=0, b_max=1, clip=True),

  1.     # RandSpatialCropd(keys=["vol", "seg"], roi_size=(1000, 1000, 48), random_size=False),
  2.     # RandFlipd(keys=["vol", "seg"], prob=0.5, spatial_axis=1),
  3.     # RandFlipd(keys=["vol", "seg"], prob=0.5, spatial_axis=2),
  4.     ToTensord(keys=["vol", "seg"]),
  5. ]

)

  1. ![image.png](https://cdn.nlark.com/yuque/0/2022/png/1699458/1647849175596-f373525d-84ad-4a63-9dda-275213824fc1.png#clientId=u8c32a884-48f6-4&crop=0&crop=0&crop=1&crop=1&from=paste&height=399&id=u0d2a1534&margin=%5Bobject%20Object%5D&name=image.png&originHeight=399&originWidth=1119&originalType=binary&ratio=1&rotation=0&showTitle=false&size=64880&status=done&style=none&taskId=uf801cff3-03db-4d5b-a67c-9ffc0d85e92&title=&width=1119)
  3. 2. 加入了多监督。loss =0.4* loss_t +0.2* loss_l +0.2* loss_m +0.2* loss_s
  5. dir:/hpc/data/home/bme/liujy3/code/TripNewUnet/TripleTest/logs/default/version2（and version1)<br />![image.png](https://cdn.nlark.com/yuque/0/2022/png/1699458/1648199016065-267bcc4f-48e5-4397-80dd-2dc4717a6386.png#clientId=uac0acc4d-4ebc-4&crop=0&crop=0&crop=1&crop=1&from=paste&height=387&id=u1951fbe5&margin=%5Bobject%20Object%5D&name=image.png&originHeight=387&originWidth=1243&originalType=binary&ratio=1&rotation=0&showTitle=false&size=71816&status=done&style=none&taskId=u547aa280-985d-4d17-99d8-1f23f510729&title=&width=1243)
  7. <a name="dILOG"></a>
  8. ### 4. TripleUNet
  9. | id | max epoch | best val dice(epoch) | test dice | source |
  10. | --- | --- | --- | --- | --- |
  11. | 1 | 263 | 0.8639(216) | 0.8528 | [TripleU](https://www.yuque.com/tuoyu-hy28g/gar920/clgl92?inner=ASMqk) |
  12. | 2 | 237 | 0.8678(184) | 0.8513  |  |
  13. | 3 |  | 0.8572(245) |  |  |
  14. | 4 |  | 0.8679(216) |  |  |
  16. 1.
  17. ```python
  18. s_channel=(16, 32),
  19. m_channel=(16, 32, 64),
  20. l_channel=(16, 32, 64, 128, 256),
  21. kernel_size: Union[Sequence[int], int] = 3,
  22. up_kernel_size: Union[Sequence[int], int] = 3,
  23. num_res_units: int = 2,
  25. #model
  26. # s_network
  27. self.down_s_1 = self._get_down_layer(1, self.s_channel[0], 2, norm=(Norm.GROUP, {'num_groups': 4}))
  28. self.bottom_s = self._get_bottom_layer(self.s_channel[0], self.s_channel[1], norm=(Norm.GROUP, {'num_groups': 8}))
  29. self.up_s_1 = self._get_up_layer(self.s_channel[0] + self.s_channel[1], 2, 2, norm=(Norm.GROUP, {'num_groups': 1}),
  30.                                  is_top=True)
  32. # m_network
  33. self.down_m_1 = self._get_down_layer(1, self.m_channel[0], 2, norm=(Norm.GROUP, {'num_groups': 4}))  # 1 -> 16
  34. self.down_m_2 = self._get_down_layer(self.m_channel[0], self.m_channel[1], 2,
  35.                                      norm=(Norm.GROUP, {'num_groups': 8}))  # 16 -> 32
  36. self.bottom_m = self._get_bottom_layer(self.m_channel[1], self.m_channel[2],
  37.                                        norm=(Norm.GROUP, {'num_groups': 16}))  # 32 -> 64
  38. self.up_m_1 = self._get_up_layer(self.m_channel[1] + self.m_channel[2], self.m_channel[0], 2,
  39.                                  norm=(Norm.GROUP, {'num_groups': 8}))  # 32+64 -> 16
  40. self.up_m_2 = self._get_up_layer(self.m_channel[0] * 2, 2, 2, norm=(Norm.GROUP, {'num_groups': 1}),
  41.                                  is_top=True)  # 16+16 -> 2
  43. # l_network
  44. self.down_l_1 = self._get_down_layer(1, self.l_channel[0], 2, norm=(Norm.GROUP, {'num_groups': 4}))  # 1 -> 16
  45. self.down_l_2 = self._get_down_layer(self.l_channel[0], self.l_channel[1], 2,
  46.                                      norm=(Norm.GROUP, {'num_groups': 8}))  # 16 -> 32
  47. self.down_l_3 = self._get_down_layer(self.l_channel[1], self.l_channel[2], 2,
  48.                                      norm=(Norm.GROUP, {'num_groups': 16}))  # 32 -> 64
  49. self.down_l_4 = self._get_down_layer(self.l_channel[2], self.l_channel[3], 2,
  50.                                      norm=(Norm.GROUP, {'num_groups': 32}))  # 64-128
  51. self.bottom_l = self._get_bottom_layer(self.l_channel[3], self.l_channel[4],
  52.                                        norm=(Norm.GROUP, {'num_groups': 32}))  # 128-256
  53. self.up_l_1 = self._get_up_layer(self.l_channel[3] + self.l_channel[4], self.l_channel[2], 2,
  54.                                  norm=(Norm.GROUP, {'num_groups': 16}))  # 256+128 - 64
  55. self.up_l_2 = self._get_up_layer(self.l_channel[2] * 2, self.l_channel[1], 2,
  56.                                  norm=(Norm.GROUP, {'num_groups': 8}))  # 64+64 - 32
  57. self.up_l_3 = self._get_up_layer(self.l_channel[1] * 2, self.l_channel[0], 2,
  58.                                  norm=(Norm.GROUP, {'num_groups': 4}))  # 32+32 - 16
  59. self.up_l_4 = self._get_up_layer(self.l_channel[0] * 2, 2, 2, norm=(Norm.GROUP, {'num_groups': 1}),
  60.                                  is_top=True)  # 16+16 - 2
  62. self.final_layer = Convolution(
  63.     spatial_dims=3,
  64.     in_channels=6,
  65.     out_channels=2,
  66.     strides=1,
  67.     kernel_size=3,
  68.     act=Act.PRELU,
  69.     norm=Norm.INSTANCE,
  70.     dropout=0.0,
  71.     bias=True,
  72.     conv_only=True,
  73.     is_transposed=True,
  74. )

loss =0.4 loss_t +0.2 loss_l +0.2 loss_m +0.2 loss_s
其余跟之前一样，就是加了GN，效果好了很多。下一个实验：证明GN的作用，再在新代码上用BN试试。
/hpc/data/home/bme/liujy3/code/mTU/mTU1/logs/default/version_0

2.
将1中的normalization改为了BN，测试看是不是GN的效果提升，但是并不是，效果仍然和上面一样好。现在就是在找原因。

3.
现在为了找出是不是fake slide window 里面每次的batchsize为1的问题，我已经改过一次了。但是效果仍然一样。
将自己写的网络实例化，然后传入fake slide window，但是效果一样

  1. import torch
  2. import torch.nn as nn
  3. from typing import Any, Sequence, Tuple, Union, Optional
  4. from monai.inferers.utils import _get_scan_interval
  5. from monai.data.utils import dense_patch_slices, compute_importance_map, get_valid_patch_size
  6. from monai.utils import BlendMode
  7. from monai.data.utils import dense_patch_slices
  8. from monai.networks.blocks.convolutions import Convolution, ResidualUnit
  9. from monai.networks.layers.factories import Act, Norm
  12. class mTriUNet(torch.nn.Module):
  13.     def __init__(
  14.         self,
  15.         spatial_dims: int = 3,
  16.         s_channel=(16, 32),
  17.         m_channel=(16, 32, 64),
  18.         l_channel=(16, 32, 64, 128, 256),
  19.         kernel_size: Union[Sequence[int], int] = 3,
  20.         up_kernel_size: Union[Sequence[int], int] = 3,
  21.         num_res_units: int = 2,
  22.         act: Union[Tuple, str] = Act.PRELU,
  23.         norm: Union[Tuple, str] = Norm.BATCH,
  24.         dropout: float = 0.0,
  25.         bias: bool = True,
  26.         dimensions: Optional[int] = None,
  27.     ):
  28.         super().__init__()
  30.         self.dimensions = spatial_dims
  31.         self.kernel_size = kernel_size
  32.         self.up_kernel_size = up_kernel_size
  33.         self.num_res_units = num_res_units
  34.         self.act = act
  35.         self.norm = norm
  36.         self.dropout = dropout
  37.         self.bias = bias
  39.         self.s_channel = s_channel
  40.         self.m_channel = m_channel
  41.         self.l_channel = l_channel
  43.         # s_network
  44.         self.down_s_1 = self._get_down_layer(1, self.s_channel[0], 2, norm=self.norm)
  45.         self.bottom_s = self._get_bottom_layer(self.s_channel[0], self.s_channel[1], norm=self.norm)
  46.         self.up_s_1 = self._get_up_layer(self.s_channel[0] + self.s_channel[1], 2, 2, norm=self.norm,
  47.                                          is_top=True)
  49.         # m_network
  50.         self.down_m_1 = self._get_down_layer(1, self.m_channel[0], 2, norm=self.norm)  # 1 -> 16
  51.         self.down_m_2 = self._get_down_layer(self.m_channel[0], self.m_channel[1], 2,
  52.                                              norm=self.norm)  # 16 -> 32
  53.         self.bottom_m = self._get_bottom_layer(self.m_channel[1], self.m_channel[2],
  54.                                                norm=self.norm)  # 32 -> 64
  55.         self.up_m_1 = self._get_up_layer(self.m_channel[1] + self.m_channel[2], self.m_channel[0], 2,
  56.                                          norm=self.norm)  # 32+64 -> 16
  57.         self.up_m_2 = self._get_up_layer(self.m_channel[0] * 2, 2, 2, norm=self.norm,
  58.                                          is_top=True)  # 16+16 -> 2
  60.         # l_network
  61.         self.down_l_1 = self._get_down_layer(1, self.l_channel[0], 2, norm=self.norm)  # 1 -> 16
  62.         self.down_l_2 = self._get_down_layer(self.l_channel[0], self.l_channel[1], 2,
  63.                                              norm=self.norm)  # 16 -> 32
  64.         self.down_l_3 = self._get_down_layer(self.l_channel[1], self.l_channel[2], 2,
  65.                                              norm=self.norm)  # 32 -> 64
  66.         self.down_l_4 = self._get_down_layer(self.l_channel[2], self.l_channel[3], 2,
  67.                                              norm=self.norm)  # 64-128
  68.         self.bottom_l = self._get_bottom_layer(self.l_channel[3], self.l_channel[4],
  69.                                                norm=self.norm)  # 128-256
  70.         self.up_l_1 = self._get_up_layer(self.l_channel[3] + self.l_channel[4], self.l_channel[2], 2,
  71.                                          norm=self.norm)  # 256+128 - 64
  72.         self.up_l_2 = self._get_up_layer(self.l_channel[2] * 2, self.l_channel[1], 2,
  73.                                          norm=self.norm)  # 64+64 - 32
  74.         self.up_l_3 = self._get_up_layer(self.l_channel[1] * 2, self.l_channel[0], 2,
  75.                                          norm=self.norm)  # 32+32 - 16
  76.         self.up_l_4 = self._get_up_layer(self.l_channel[0] * 2, 2, 2, norm=self.norm,
  77.                                          is_top=True)  # 16+16 - 2
  79.         self.final_layer = Convolution(
  80.             spatial_dims=3,
  81.             in_channels=6,
  82.             out_channels=2,
  83.             strides=1,
  84.             kernel_size=3,
  85.             act=Act.PRELU,
  86.             norm=Norm.INSTANCE,
  87.             dropout=0.0,
  88.             bias=True,
  89.             conv_only=True,
  90.             is_transposed=True,
  91.         )
  93.         def _get_down_layer(
  94.             self,
  95.             in_channels: int,
  96.             out_channels: int,
  97.             strides: int,
  98.             norm: Union[Tuple, str] = Norm.INSTANCE,
  99.             is_top=False
  100.         ) -> nn.Module:
  101.             """
  102.             Args:
  103.             in_channels: number of input channels.
  104.             out_channels: number of output channels.
  105.             strides: convolution stride.
  106.             is_top: True if this is the top block.
  107.             """
  108.             mod: nn.Module
  109.             if self.num_res_units > 0:
  110.                 mod = ResidualUnit(
  111.                     self.dimensions,
  112.                     in_channels,
  113.                     out_channels,
  114.                     strides=strides,
  115.                     kernel_size=self.kernel_size,
  116.                     subunits=self.num_res_units,
  117.                     act=self.act,
  118.                     norm=norm,
  119.                     dropout=self.dropout,
  120.                     bias=self.bias,
  121.                 )
  122.                 return mod
  123.             mod = Convolution(
  124.                 self.dimensions,
  125.                 in_channels,
  126.                 out_channels,
  127.                 strides=strides,
  128.                 kernel_size=self.kernel_size,
  129.                 act=self.act,
  130.                 norm=norm,
  131.                 dropout=self.dropout,
  132.                 bias=self.bias,
  133.             )
  134.             return mod
  136.         def _get_bottom_layer(
  137.             self,
  138.             in_channels: int,
  139.             out_channels: int,
  140.             norm: Union[Tuple, str] = Norm.INSTANCE
  141.         ) -> nn.Module:
  142.             """
  143.             Args:
  144.             in_channels: number of input channels.
  145.             out_channels: number of output channels.
  146.             """
  147.             return self._get_down_layer(in_channels, out_channels, 1, norm, False)
  149.         def _get_up_layer(
  150.             self,
  151.             in_channels: int,
  152.             out_channels: int,
  153.             strides: int,
  154.             norm: Union[Tuple, str] = Norm.INSTANCE,
  155.             is_top: bool = False
  156.         ) -> nn.Module:
  157.             """
  158.             Args:
  159.             in_channels: number of input channels.
  160.             out_channels: number of output channels.
  161.             strides: convolution stride.
  162.             is_top: True if this is the top block.
  163.             """
  164.             conv: Union[Convolution, nn.Sequential]
  166.             conv = Convolution(
  167.                 self.dimensions,
  168.                 in_channels,
  169.                 out_channels,
  170.                 strides=strides,
  171.                 kernel_size=self.up_kernel_size,
  172.                 act=self.act,
  173.                 norm=norm,
  174.                 dropout=self.dropout,
  175.                 bias=self.bias,
  176.                 conv_only=is_top and self.num_res_units == 0,
  177.                 is_transposed=True,
  178.             )
  180.             if self.num_res_units > 0:
  181.                 ru = ResidualUnit(
  182.                     self.dimensions,
  183.                     out_channels,
  184.                     out_channels,
  185.                     strides=1,
  186.                     kernel_size=self.kernel_size,
  187.                     subunits=1,
  188.                     act=self.act,
  189.                     norm=norm,
  190.                     dropout=self.dropout,
  191.                     bias=self.bias,
  192.                     last_conv_only=is_top,
  193.                 )
  194.                 conv = nn.Sequential(conv, ru)
  196.                 return conv
  198.             def fake_slide_window(
  199.                 self,
  200.                 inputs: torch.Tensor,
  201.                 ratio,
  202.                 sw_batch_size: int,
  203.                 mode: Union[BlendMode, str] = BlendMode.CONSTANT,
  204.                 sigma_scale: Union[Sequence[float], float] = 0.125,
  205.                 device: Union[torch.device, str, None] = None,
  206.                 sw_device: Union[torch.device, str, None] = None,
  207.                 *args: Any,
  208.                 **kwargs: Any,
  209.             ) -> torch.Tensor:
  212.                 num_spatial_dims = len(inputs.shape) - 2
  214.                 if device is None:
  215.                     device = inputs.device
  216.                     if sw_device is None:
  217.                         sw_device = inputs.device
  219.                         image_size = list(inputs.shape[2:])
  220.                         batch_size = inputs.shape[0]
  221.                         roi_size = []
  222.                         for i in image_size:
  223.                             i = i * ratio
  224.                             roi_size.append(int(i))
  226.                             # =0?
  227.                             overlap = 0
  228.                             scan_interval = _get_scan_interval(image_size, roi_size, num_spatial_dims, overlap)
  230.                             # Store all slices in list
  231.                             slices = dense_patch_slices(image_size, roi_size, scan_interval)
  232.                             num_win = len(slices)  # number of windows per image
  233.                             total_slices = num_win * batch_size  # total number of windows
  235.                             # Create window-level importance map
  236.                             importance_map = compute_importance_map(
  237.                                 get_valid_patch_size(image_size, roi_size), mode=mode, sigma_scale=sigma_scale, device=device
  238.                             )
  240.                             # Perform predictions
  241.                             output_image, count_map = torch.tensor(0.0, device=device), torch.tensor(0.0, device=device)
  242.                             _initialized = False
  243.                             for slice_g in range(0, total_slices, sw_batch_size):
  244.                                 slice_range = range(slice_g, min(slice_g + sw_batch_size, total_slices))
  245.                                 unravel_slice = [
  246.                                     [slice(int(idx / num_win), int(idx / num_win) + 1), slice(None)] + list(slices[idx % num_win])
  247.                                     for idx in slice_range
  248.                                 ]
  249.                                 window_data = torch.cat([inputs[win_slice] for win_slice in unravel_slice]).to(sw_device)
  250.                                 if ratio == 1/2:
  251.                                     seg_prob = self.m_network(window_data).to(device)  # batched patch segmentation
  252.                                 elif ratio == 1/4:
  253.                                     seg_prob = self.s_network(window_data).to(device)
  255.                                     if not _initialized:  # init. buffer at the first iteration
  256.                                         output_classes = seg_prob.shape[1]
  257.                                         output_shape = [batch_size, output_classes] + list(image_size)
  258.                                         # allocate memory to store the full output and the count for overlapping parts
  259.                                         output_image = torch.zeros(output_shape, dtype=torch.float32, device=device)
  260.                                         count_map = torch.zeros(output_shape, dtype=torch.float32, device=device)
  261.                                         _initialized = True
  263.                                         # store the result in the proper location of the full output. Apply weights from importance map.
  264.                                         for idx, original_idx in zip(slice_range, unravel_slice):
  265.                                             output_image[original_idx] += importance_map * seg_prob[idx - slice_g]
  266.                                             count_map[original_idx] += importance_map
  268.                                             # account for any overlapping sections
  269.                                             output_image = output_image / count_map
  271.                                             return output_image
  273.                                         def s_network(self, x_s):
  274.                                             down_out_s_1 = self.down_s_1(x_s)
  275.                                             bottom_out_s = self.bottom_s(down_out_s_1)
  276.                                             up_out_s_1 = self.up_s_1(torch.cat([down_out_s_1, bottom_out_s], dim=1))
  277.                                             return up_out_s_1
  279.                                         def m_network(self, x_m):
  280.                                             down_out_m_1 = self.down_m_1(x_m)
  281.                                             down_out_m_2 = self.down_m_2(down_out_m_1)
  282.                                             bottom_out_m = self.bottom_m(down_out_m_2)
  283.                                             up_out_m_1 = self.up_m_1(torch.cat([down_out_m_2, bottom_out_m], dim=1))
  284.                                             up_out_m_2 = self.up_m_2(torch.cat([down_out_m_1, up_out_m_1], dim=1))
  285.                                             return up_out_m_2
  287.                                         def l_network(self, x_l):
  288.                                             down_out_l_1 = self.down_l_1(x_l)
  289.                                             down_out_l_2 = self.down_l_2(down_out_l_1)
  290.                                             down_out_l_3 = self.down_l_3(down_out_l_2)
  291.                                             down_out_l_4 = self.down_l_4(down_out_l_3)
  292.                                             bottom_out_l = self.bottom_l(down_out_l_4)
  293.                                             up_out_l_1 = self.up_l_1(torch.cat([down_out_l_4, bottom_out_l], dim=1))
  294.                                             up_out_l_2 = self.up_l_2(torch.cat([down_out_l_3, up_out_l_1], dim=1))
  295.                                             up_out_l_3 = self.up_l_3(torch.cat([down_out_l_2, up_out_l_2], dim=1))
  296.                                             up_out_l_4 = self.up_l_4(torch.cat([down_out_l_1, up_out_l_3], dim=1))
  297.                                             return up_out_l_4
  299.                                         def forward(self, x):
  300.                                             # input data
  301.                                             # three-path
  302.                                             # s (16, 32)
  303.                                             output_s = self.fake_slide_window(x, 1/4, 1)
  305.                                             # m (16, 32, 64)
  306.                                             output_m = self.fake_slide_window(x, 1/2, 1)
  308.                                             # l (16, 32, 64, 128, 256)
  309.                                             output_l = self.l_network(x)
  311.                                             # concat
  312.                                             output = torch.cat((output_l, output_m, output_s), 1)
  313.                                             # final layer + softmax?
  314.                                             output = self.final_layer(output)
  316.                                             return output, output_l, output_m, output_s

4.
我现在猜测是我的网络的问题，然后我用了原来的UNet和bn，然后用我新的切patch的方法，但是他妈的结果还是一样，bullshit。

  1. import torch
  2. import torch.nn as nn
  3. import pytorch_lightning as pl
  4. from typing import Any, Callable, List, Sequence, Tuple, Union
  5. from monai.networks.layers.factories import Act
  6. from monai.utils import BlendMode
  7. from monai.inferers.utils import _get_scan_interval
  8. from monai.data.utils import dense_patch_slices, compute_importance_map, get_valid_patch_size
  9. from monai.networks.layers import Norm
  10. from monai.networks.blocks.convolutions import Convolution, ResidualUnit
  11. from monai.networks.nets import UNet
  14. def recover_divisible_patch(inputs, image_size, slice_range, unravel_slice):
  15.     # only for non-overlap
  17.     device = inputs.device
  18.     batch_size = 1
  19.     output_classes = inputs.shape[1]
  20.     output_shape = [batch_size, output_classes] + list(image_size)
  21.     # allocate memory to store the full output and the count for overlapping parts
  22.     output_image = torch.zeros(output_shape, dtype=torch.float32, device=device)
  24.     # store the result in the proper location of the full output. Apply weights from importance map.
  25.     for idx, original_idx in zip(slice_range, unravel_slice):
  26.         output_image[original_idx] += inputs[idx]
  28.         return output_image
  31.     def divisible_patch(
  32.         inputs: torch.Tensor,
  33.         ratio,
  34.         device: Union[torch.device, str, None] = None,
  35.         *args: Any,
  36.         **kwargs: Any,
  37.     ):
  39.         num_spatial_dims = len(inputs.shape) - 2
  41.         if device is None:
  42.             device = inputs.device
  44.             image_size = list(inputs.shape[2:])
  45.             batch_size = inputs.shape[0]
  46.             roi_size = []
  47.             for i in image_size:
  48.                 i = i * ratio
  49.                 roi_size.append(int(i))
  51.                 overlap = 0
  52.                 scan_interval = _get_scan_interval(image_size, roi_size, num_spatial_dims, overlap)
  54.                 # Store all slices in list
  55.                 slices = dense_patch_slices(image_size, roi_size, scan_interval)
  56.                 num_win = len(slices)  # number of windows per image
  57.                 total_slices = num_win * batch_size  # total number of windows
  59.                 slice_range = range(0, total_slices)
  61.                 unravel_slice = [
  62.                     [slice(int(idx / num_win), int(idx / num_win) + 1), slice(None)] + list(slices[idx % num_win])
  63.                     for idx in slice_range
  64.                 ]
  66.                 # This is end of cut data
  67.                 window_data = torch.cat([inputs[win_slice] for win_slice in unravel_slice]).to(device)
  69.                 return window_data, image_size, slice_range, unravel_slice
  72.             class mswTripleUNet(torch.nn.Module):
  73.                 def __init__(self):
  74.                     super().__init__()
  75.                     self._modelS = UNet(
  76.                         spatial_dims=3,
  77.                         in_channels=1,
  78.                         out_channels=2,
  79.                         channels=(16, 32),
  80.                         strides=(2,),
  81.                         num_res_units=2,
  82.                         norm=Norm.BATCH,
  83.                     )
  84.                     self._modelM = UNet(
  85.                         spatial_dims=3,
  86.                         in_channels=1,
  87.                         out_channels=2,
  88.                         channels=(16, 32, 64),
  89.                         strides=(2, 2,),
  90.                         num_res_units=2,
  91.                         norm=Norm.BATCH,
  92.                     )
  93.                     self._modelL = UNet(
  94.                         spatial_dims=3,
  95.                         in_channels=1,
  96.                         out_channels=2,
  97.                         channels=(16, 32, 64, 128, 256),
  98.                         strides=(2, 2, 2, 2),
  99.                         num_res_units=2,
  100.                         norm=Norm.BATCH,
  101.                     )
  102.                     self.final_layer = Convolution(
  103.                         spatial_dims=3,
  104.                         in_channels=6,
  105.                         out_channels=2,
  106.                         strides=1,
  107.                         kernel_size=3,
  108.                         act=Act.PRELU,
  109.                         norm=Norm.BATCH,
  110.                         dropout=0.0,
  111.                         bias=True,
  112.                         conv_only=True,
  113.                         is_transposed=True,
  114.                     )
  115.                     # the batch size to run window slices
  116.                     self.sw_batch_size = 1
  118.                     def forward(self, x):
  119.                         # input data
  120.                         x_l = x
  121.                         x_m, image_size_m, slice_range_m, unravel_slice_m = divisible_patch(x, 1/2)
  122.                         x_s, image_size_s, slice_range_s, unravel_slice_s = divisible_patch(x, 1/4)
  124.                         # split into 3 part
  125.                         output_s = self._modelS(x_s)
  126.                         output_s = recover_divisible_patch(output_s, image_size_s, slice_range_s, unravel_slice_s)
  128.                         output_m = self._modelM(x_m)
  129.                         output_m = recover_divisible_patch(output_m, image_size_m, slice_range_m, unravel_slice_m)
  131.                         output_l = self._modelL(x_l)
  133.                         # concat
  134.                         output = torch.cat((output_s, output_m, output_l), 1)
  135.                         # final layer + softmax?
  136.                         output = self.final_layer(output)
  137.                         return output, output_l, output_m, output_s

5. TripeUNet(543)

id different max epoch best val dice(epoch) test dice source
1 5 4 3 228 0.8613（213） 0.8507
2 最后不叠加loss still 0.8680 335
3 中间融合(add+BN), 叠加loss 231 0.8583 213
4 中间融合(add+GN),叠加loss 250 0.8674 237 0.8552
5 中间融合(cat+GN),叠加loss(TU_ccatgn_new) 253 0.8714 187 0.8586
6 中间融合(cat_gn+GN),叠加loss 227 0.8619 216

第五：cat之后别忘了加normalization，灵感来自attention。

6. for other dataset

id dataset method max epoch best val dice(epoch) test dice source
1 Cornary 中间融合(cat+GN) 337 0.7605 334
2 roi_jy 中间融合(cat+GN) 243 0.8736 89
3
4
5
  1. cornary
  2. jingyang 采用的选择roi的方法，下面是他们的大小

the mean of s1 is 368.1923076923077, the min of s1 is 299, the max of s1 is 434
the mean of s2 is 256.46153846153845, the min of s2 is 199, the max of s2 is 322
the mean of s3 is 251.6769230769231, the min of s3 is 157, the max of s3 is 320
256 128 128
128 64 64
64 32 32
32 16 16
16 8 8

7. bottom重新组合

使中间S和M网络对应的尺寸都缩小一倍，使他们可以组合在一起。

  1. 缩小
    1. 可以使bottom带有stride（之前的bottom_layer都没有stride
    2. 让倒数第二层的步长翻倍（目前采用了这个，简单，对应的up也得翻倍
  2. 重组
    1. 一个问题是加上对batch_size的支持（先就为1这么跑吧 | id | dataset | method | max epoch | best val dice(epoch) | test dice | source | | —- | —- | —- | —- | —- | —- | —- | | 1 | roi_jy | 2a | 260 | 0.8709 255 | | | | 2 |
      |
      | | | | | | 3 | | | | | | | | 4 | | | | | | | | 5 | |
      | | | | |

这个问题就在于最后一维的块太小了，而且采用1b的方法最后的步长太大，很多的信息都没有采集到。
todo: 把roi数据裁大一点。
end: 把margin增大到了16，但是效果还是没有提升，反而val_Dice只有0.54，严重怀疑我的代码有问题。(TU_16Roi-52869)

问题，可能在于最后一层是prelu，考虑设置成sigmoid，这样不用在diceloss里算sigmoid了。至于算dice，一般是在argmax之后算。

8. Multi-Scale(jm)

  1. TU_multiscale. 数据大小分别是(256, 192, 192), (214, 160, 160), (192, 128, 128)。然后大网络->中网络，中网络->小网络。所有数据都是提前先切patch，每个数据都切了不少patch。（问题：这些patch都过一遍吗，还是每次就随机选择呢？应该还是要随机选择，肯定不可能全部过一遍)

    1. as the S the center.<br />v1: three path, and supervised persepctively.<br />v2: three path, crop, simple convlution, and finally supervised by the center part.
id dataset method max epoch patch strategy or number best val dice(epoch) test dice source
1 roi_jy v1
250(randsample) 0.7756(149)
2 roi_jy v2
125 0.7345(107)
3





4





5





9. Tu_trans

id dataset method max epoch patch strategy or number best val dice(epoch) test dice
source
1 roi_jy S2M2L 432
0.7612(387


2 roi_jy L2M2S doing 125



3 roi_jy Lonly2S 149
0.6607(90


4 roi_jy mTUV2L2S 500
0.82 0.77

5







10. 跳连 concat

id dataset method max epoch patch strategy or number best val dice(epoch) test dice
source
1 roi_jy v1 500
0.8410（325


2 roi_jy v2(with background) 500 0 0.8437


3 roi_jy






4 roi_jy






5







如果采用先训练大网络，然后训练其他网络的方法。那么在load进大网络参数后，还更新它吗？如果不更新的话，最后的loss还加权吗？（我觉得需要更新，因为目的是增加多尺度的内容
也可以采用freeze的方法，先freeze部分参数，但是得有初始值，然后