Basic indicators

Error rate ： The proportion of misjudgment in the test sample

precision ：1- Error rate

Division method of training set and test set ：

1. Set aside method

Aggregate data set D= Training set S+ Test set T

Traditional three seven open and so on , Three test sets , 70% training set . Generally, stratified sampling is also used , Let the proportion of positive and negative samples in the test set and the training set be the same .

The single set aside method is often not reliable , It usually happens many times at random .

2. Cross validation

k Fold cross validation is about all samples D Divide into k Share , Then choose one of them k-1 As a training set , The remaining 1 As a test set , And repeat this process k Time ( Each test set is different ).

When dividing, you can also choose a variety of division methods , commonly k Fold cross validation usually needs to be repeated randomly using different division methods p Time , This is called p Time k Crossover verification .

If there are only m individual , And then take k=m, This is called Keep one , In this way, it will not be affected by the random sample division , Because the training set is relatively close D, So the effect is often more accurate , But it takes a lot of time .

3. Self help law

Neither of the above two methods can be fully utilized D Training ( It takes too much time to leave one method ), To solve this problem , Propose self-help method (bootstrapping).

Given contains m individual The data set of the sample D, There is a sample put back m Time , Every time 1 Samples ,( Note the total number of samples and data sets D The total number of samples is the same !!!), In this way, we can get D',D' As a training set ,D\D' As test set , Such test results are also called “ Out of the bag estimate ”.

The self-help method is smaller in the data set 、 It's hard to divide training effectively / It's useful when testing sets . And it can generate many different training sets from the initial data set , Better for integrated learning .

But there are drawbacks , Changed the distribution of the initial dataset , It introduces estimation bias , So it's better to set aside and cross verify when the data set is sufficient .