Preprocessing - outlier detection

1、 effect

It can detect and process abnormal values contained in variable data , Outlier detection logic is a data set of variables （ Similar to column ） Determine according to the set threshold , Filter out the data falling within the detection range of outliers , Then replace the original data according to the disposal method .

2、 Input / output description

Input ： Raw data column

Output ： Data column after removing outliers

3、 Modeling steps

Laida criterion

The data needs to obey the normal distribution , Plus or minus 3∂ Is the probability that 99.7%, So the average distance 3∂ The probability of occurrence of values other than P(|x-u| 3∂) = 0.003, It belongs to a very small probability event . If the data does not obey the normal distribution , It can also be described by how many times the standard deviation away from the average .

IQR distinguish

Four minute spacing (IQR) Is the difference between the upper quartile and the lower quartile . And we passed IQR Of 1.5 Times the standard , Regulations ： exceed （ The top quartile +1.5 times IQR distance , Or the lower quartile -1.5 times IQR distance ） The point of is the outlier .