Loop filtering using SSIM based CNN

This article comes from the proposal JVET-T0069《SSIM based CNN model for  in-loop filtering》, At present, most of the video coding research based on deep learning is to optimize objective indicators , However, for human vision, a higher objective index sometimes does not mean a higher subjective effect . therefore , The proposal proposes a neural network model CNNLF(convolutional neural network based in-loop filter) For subjective optimization .

brief introduction

CNNLF Used in loop filtering stage , be in deblocking after ,SAO Before , Such as Fig.1 Shown .

The reconstruction frame will first use DF Filter processing , And then pass by CNNLF Model processing , At the end of the day SAO Further treatment . This method works at the block level , adopt RDO Decide whether to use CNNLF Handle , Therefore, it is necessary to pass a flag bit in the code stream to indicate whether CNNLF.

Network structure

The whole network structure is shown in the figure , from 8 A residual unit (3x3 Convolution ) constitute , Except for the last convolution layer generation 3 Dimension output , Other convolution layers are generated 64 Whitman's sign . Finally, there are two SE Block and one 3x3 Convolution connection is used after the network for adaptive mixed features to generate the final reconstructed image . Be careful , The first residual element is not used 1x1 Convolution . Shorting in the network can promote the flow of shallow features .

Residual unit

Pictured above , The residual element consists of a 1x1 Convolution layer and two 3x3 The convolution layer constitutes , The activation function is ReLU, There are two short circuits and one in the unit SE block （squeeze-and-excitation block）.

Short circuit will 1x1 The output sum of convolution 3x3 The output of convolution can be added to learn the residual . In the following formula x It's input ,F(x) Is the entire output ,f() Express 1x1 Convolution operation ,g() It means residual learning .

SE The block structure is shown in the figure below , It processes features into multi-channel forms . It adapts the calibration feature channels by showing the dependencies between the modeled feature channels , It is conducive to model learning and compressed information .

Training

Model USES DIV2K Dataset training , contain 800 Training images ,100 Verification images , All images use VTM10.0 stay AI Configure the next encoding ,QP={22,27,32,37}. For different QP Train only one model . And the training indicators are SSIM, Test indicators use VMAF, because VMAF Discontinuous, so it is not suitable for use in training . The training configuration is shown in the table 1, among loss The function uses SSIM and L1 loss A weighted , Two sets of weight parameters are used here (0.2,0.8) and (0.8,0.2)

experimental result

The experimental test configuration is AI、RA and LDB,QP={22,27,32,37}. Due to the high computational complexity ,RA Configuration only runs the first GOP.YUV Do not use GPU, Decoding will use GPU and CPU Decode each time .

Loss = 0.8 x SSIM + 0.2 x L1

Loss = 0.2 x SSIM + 0.8 x L1

Interested parties, please pay attention to WeChat official account Video Coding