Labelsmoothing introduction and code implementation

What is? label smoothing？

Label smoothing （Label smoothing）, image L1、L2 and dropout equally , It is a regularization method in the field of machine learning , Usually used for classification problems , The purpose is to prevent the model from overconfident prediction during training , Improve the problem of poor generalization ability .

Use label smoothing Purpose

label smoothing Commonly used for classification tasks , Prevent the model from over fitting in training , Improve the generalization ability of the model .

Use label smoothing

class LabelSmoothingCrossEntropy(nn.Module):
    """ Cross Entropy loss with label smoothing. """
    def __init__(self, smoothing=0.1):
        """ Constructor for the LabelSmoothing module. :param smoothing: label smoothing factor """
        super(LabelSmoothingCrossEntropy, self).__init__()
        assert 0.0 < smoothing < 1.0
        self.smoothing = smoothing
        self.confidence = 1. - smoothing

    def forward(self, x, target):
        """  How to write it 1 """
        # logprobs = F.log_softmax(x, dim=-1)
        # nll_loss = -logprobs.gather(dim=-1, index=target.unsqueeze(1))
        # nll_loss = nll_loss.squeeze(1) #  Get the cross entropy loss 
        # #  Note that this should be understood in combination with the formula , At the same time, pay attention to those who predict correctly , Also have a/K, among a Is the smoothing factor ,K Is the number of categories 
        # smooth_loss = -logprobs.mean(dim=1)
        # loss = self.confidence * nll_loss + self.smoothing * smooth_loss
        """  How to write it 2 """
        y_hat = torch.softmax(x, dim=1)
        #  here cross_loss and nll_loss Equivalent 
        cross_loss = self.cross_entropy(y_hat, target)
        smooth_loss = -torch.log(y_hat).mean(dim=1)
        # smooth_loss You can also use the following method to calculate , Be careful loga + logb = log(ab)
        # smooth_loss = -torch.log(torch.prod(y_hat, dim=1)) / y_hat.shape[1]
        loss = self.confidence * cross_loss + self.smoothing * smooth_loss
        return loss.mean()

    def cross_entropy(self, y_hat, y):
        return - torch.log(y_hat[range(len(y_hat)), y])

Then the loss function you may just start using is ：

lossfunc = nn.CrossEntropyLoss()

You just change it to ：

lossfunc = LabelSmoothingCrossEntropy(smoothing=0.1)

You can use label smoothing in your code .