最近，看到很多轻量化工作是基于RepVGG改进而来，决定重新回顾一下RepVGG，并在此记录一些理解与心得。

论文地址：https://arxiv.org/abs/2101.03697

Introduction

RepVGG通过结构重参数化思想，让训练网络的多路结构(多分支模型训练时的优势——性能高)转换为推理网络的单路结构(模型推理时的好处——速度快、省内存），从而达到推理速度快与模型性能高兼备的效果。

Method

根据Resnet与Inception等论文论证，与单路直筒结构相比，增加shortcut与muti-branch可以提高模型的性能，RepVGG在VGG的基础上做了相关实验，从下表中可以发现，Identity branch与1x1 branch能够提高准确率，但同时推理时间也延长了许多。

为了在保证模型精度的基础上，降低模型推理时间，RepVGG采用了结构重构的方法，即在训练时使用复杂结构，而测试时将复杂模型重构成单路直筒型模型，如下图所示。图B中，RepVGG在training时加入了shortcut和1x1 branch，而在test时，RepVGG重构成直筒型VGG模型，如图c。

Re-param for Plain Inference-time Model
如下图所示，我们的目的是将带有BN层的3x3卷积与1x1卷积以及Identity mapping融合成一个3x3卷积，如何融合呢？操作分为两部：1.融合conv与bn；2.合并卷积。

1.Merge BN and Conv together

2. Fuse 3x3conv 1x1conv and identity

假设输入特征尺寸为（1x2x3x3），输出特征尺寸为输入相同，stride=1，那么conv3x3的卷积过程如下图所示，conv的特征尺寸为（2x2x3x3），首先对输入扩边，padding=kernel_size // 2，然后从左上角开始滑动窗口，最后获得右边的输出特征。

conv_1x1的卷积相较于3x3卷积更为简单，只需要将卷积参数与对应的单个输入参数相乘再相加。

为了使conv_1x1可以与conv_3x3线性相加，我们可以将conv_1x1的1x1卷积核扩边成3x3的卷积核，这样形式上就可以变成conv_3x3，同时，结果与conv_1x1一致。

同样，Identity mapping可以看作特殊的conv_1x1，参数是固定的，即输入特征的某一层对应的卷积参数为1，其余均为0。我们可以按照上面介绍的conv_1x1与conv_3x3转化方式，将Identity mapping转化成conv_3x3的形式，如下图所示。

如此，我们就可以将带有BN层的3x3卷积与1x1卷积以及Identity mapping融合成一个3x3卷积。

code

我们看一下代码如何实现RepVGG，首先，我们看到deploy的控制Flag，当depoly为False时，模型处于training状态，结构没有重参化，当deploy为True时，模型处于test状态，需要对结构重参。

结构重参时，需要调用switch_to_deploy()，该函数的作用是调用self.get_equivalent_kernel_bias()获得重构后的kernel, bias， 对重构卷积赋值–self.rbr_reparam = nn.Conv2d，其中Conv2d的参数是kernel，偏移是bias，将参数detach脱离计算图，并删除其余卷积操作内存（否则无法正常运行）。

get_equivalent_kernel_bias(self):逻辑很简单，分为两部：1.调用self._fuse_bn_tensor()将conv与bn融合。需要注意的是，identity没有conv，只有BN操作，所以在self._fuse_bn_tensor()函数中需要构造卷积kernel，kernel_value[i, i % input_dim, 1, 1] = 1，需要注意分组卷积的形式。函数的返回值 return kernel3x3 + self._pad_1x1_to_3x3_tensor(kernel1x1) + kernelid, bias3x3 + bias1x1 + biasid

class RepVGGBlock(nn.Module):'''RepVGGBlock is a basic rep-style block, including training and deploy statusThis code is based on https://github.com/DingXiaoH/RepVGG/blob/main/repvgg.py'''def __init__(self, in_channels, out_channels, kernel_size=3,stride=1, padding=1, dilation=1, groups=1, padding_mode='zeros', deploy=False, use_se=False):super(RepVGGBlock, self).__init__()""" Initialization of the class.Args:in_channels (int): Number of channels in the input imageout_channels (int): Number of channels produced by the convolutionkernel_size (int or tuple): Size of the convolving kernelstride (int or tuple, optional): Stride of the convolution. Default: 1padding (int or tuple, optional): Zero-padding added to both sides ofthe input. Default: 1dilation (int or tuple, optional): Spacing between kernel elements. Default: 1groups (int, optional): Number of blocked connections from inputchannels to output channels. Default: 1padding_mode (string, optional): Default: 'zeros'deploy: Whether to be deploy status or training status. Default: Falseuse_se: Whether to use se. Default: False"""self.deploy = deployself.groups = groupsself.in_channels = in_channelsself.out_channels = out_channelsassert kernel_size == 3assert padding == 1padding_11 = padding - kernel_size // 2self.nonlinearity = nn.ReLU()if use_se:raise NotImplementedError("se block not supported yet")else:self.se = nn.Identity()if deploy:self.rbr_reparam = nn.Conv2d(in_channels=in_channels, out_channels=out_channels, kernel_size=kernel_size, stride=stride,padding=padding, dilation=dilation, groups=groups, bias=True, padding_mode=padding_mode)else:self.rbr_identity = nn.BatchNorm2d(num_features=in_channels) if out_channels == in_channels and stride == 1 else Noneself.rbr_dense = conv_bn(in_channels=in_channels, out_channels=out_channels, kernel_size=kernel_size, stride=stride, padding=padding, groups=groups)self.rbr_1x1 = conv_bn(in_channels=in_channels, out_channels=out_channels, kernel_size=1, stride=stride, padding=padding_11, groups=groups)def forward(self, inputs):'''Forward process'''if hasattr(self, 'rbr_reparam'):return self.nonlinearity(self.se(self.rbr_reparam(inputs)))if self.rbr_identity is None:id_out = 0else:id_out = self.rbr_identity(inputs)return self.nonlinearity(self.se(self.rbr_dense(inputs) + self.rbr_1x1(inputs) + id_out))def get_equivalent_kernel_bias(self):kernel3x3, bias3x3 = self._fuse_bn_tensor(self.rbr_dense)kernel1x1, bias1x1 = self._fuse_bn_tensor(self.rbr_1x1)kernelid, biasid = self._fuse_bn_tensor(self.rbr_identity)return kernel3x3 + self._pad_1x1_to_3x3_tensor(kernel1x1) + kernelid, bias3x3 + bias1x1 + biasiddef _pad_1x1_to_3x3_tensor(self, kernel1x1):if kernel1x1 is None:return 0else:return torch.nn.functional.pad(kernel1x1, [1, 1, 1, 1])def _fuse_bn_tensor(self, branch):if branch is None:return 0, 0if isinstance(branch, nn.Sequential):kernel = branch.conv.weightrunning_mean = branch.bn.running_meanrunning_var = branch.bn.running_vargamma = branch.bn.weightbeta = branch.bn.biaseps = branch.bn.epselse:assert isinstance(branch, nn.BatchNorm2d)if not hasattr(self, 'id_tensor'):input_dim = self.in_channels // self.groupskernel_value = np.zeros((self.in_channels, input_dim, 3, 3), dtype=np.float32)for i in range(self.in_channels):kernel_value[i, i % input_dim, 1, 1] = 1self.id_tensor = torch.from_numpy(kernel_value).to(branch.weight.device)kernel = self.id_tensorrunning_mean = branch.running_meanrunning_var = branch.running_vargamma = branch.weightbeta = branch.biaseps = branch.epsstd = (running_var + eps).sqrt()t = (gamma / std).reshape(-1, 1, 1, 1)return kernel * t, beta - running_mean * gamma / stddef switch_to_deploy(self):if hasattr(self, 'rbr_reparam'):returnkernel, bias = self.get_equivalent_kernel_bias()self.rbr_reparam = nn.Conv2d(in_channels=self.rbr_dense.conv.in_channels, out_channels=self.rbr_dense.conv.out_channels,kernel_size=self.rbr_dense.conv.kernel_size, stride=self.rbr_dense.conv.stride,padding=self.rbr_dense.conv.padding, dilation=self.rbr_dense.conv.dilation, groups=self.rbr_dense.conv.groups, bias=True)self.rbr_reparam.weight.data = kernelself.rbr_reparam.bias.data = biasfor para in self.parameters():para.detach_()self.__delattr__('rbr_dense')self.__delattr__('rbr_1x1')if hasattr(self, 'rbr_identity'):self.__delattr__('rbr_identity')if hasattr(self, 'id_tensor'):self.__delattr__('id_tensor')self.deploy = True

Drawback

在RepVGG中存在一个致命的缺点，我们知道RepVGG是为了轻量化而生，但是结构重参导致模型参数方差较大，从而引起量化误差，实验证明，RepVGG通过训练后量化会将准确率降低20%，与此同时，由于特殊的training与test结构差异，RepVGG很难进行感知量化训练。为了对量化友好，Repopt提出了很好的思路解决这个问题，我们下篇继续介绍Repopt。