2021.4, The effect ranks No 23（papers with code）

Model principle

thought ： take transformer Introduce the field of anomaly detection , Transform the generation based reconfiguration problem into inpainting problem （ Cover some areas of the image and restore , It can be regarded as a self-monitoring method ）, Use transformer To capture the information of the big receptive field . 

Patch Embeddings and Multihead Feature Self-attention

Patch embeddings

Pictured , Divide the input graph into 16*16 Grid graph of , Then select randomly from the grid 7*7 Size chart , Constitutive tensor [252,49,768]（batchsize：252,7*7:49,16*16*3（RGB passageway ）：768）. It is converted into [252,49,512], The graph output to the network is [252,49,512]（ It includes unmasked blocks and layers added to supplement the positional relationship ）,[252,768]（ Covered blocks ）.

Multihead Feature Self-attention MFSA The long attention mechanism

query,key,value Graphs whose tensors are all inputs [252,49,512],query,key the （MLP） Two fully connected layers （ Set activation function between layers ） Output [252,49,256], Then transform the layout output of the tensor [252,8,49,32],value Through a full connection layer （ Do not set the activation function ） Output [252,49,256], Then transform the layout output of the tensor [252,8,49,64].

key Tensor retransfer [252,8,32,49] And query Tensor multiplication and division Gain tensor [252,8,49,49]. the softmax Return tensor p_attn[252,8,49,49], take p_attn And value Multiply the return tensor value_pro[252,8,49,64].

take value_pro After conversion and full connection, the tensor is output [252,49,512].

Network Architecture and Training

Pass the input tensor through the full connection layer gelu Activation goes through the full connection layer .

--------------------------------------------

adopt MSFA And MLP build transformer, Post output tensor [252,49,512], Averaging , Output through the full connection layer [252,768].

Map to a graph [252,3,16,16], Calculate the loss function , The loss function includes L2 Loss / Loss of structural similarity （structural similarity）/ Gradient amplitude similarity loss （gradient magnitude similarity）

experiment

details ：transformer The training time is relatively long , Sometimes more than 500 individual epoch. When validation sets loss exceed 50 individual epoch There is no obvious decline , Then stop training , And choose the best model to evaluate .

Ablation Experiment ： A long connection ,MSA And MFSA, The length of the window （ The original for 7）.

Reference resources

For anomaly detection Transformer - InTra《Inpainting Transformer for Anomaly Detection》_ I'm the blog of rhubarb -CSDN Blog _transformer Anomaly detection