Li Hongyi, machine learning 3 Gradient descent

One 、 Source of error

1.1  Under fitting and over fitting

• If the deviation of the model in the training set is too large , That is, under fitting . resolvent ： Redesign the model ; Consider more powers 、 More complex models .

• If the model gets a small error on the training set , But we get a big error in the test set , This means that the model may have a large variance , It's over fitting . resolvent ： Add more data ; Regularization processing .

1.2  Model selection

• Cross validation （Cross Validation）： Divide the training set into two parts , Part of it is a training set , Part as validation set . Train the model with the training set , Then compare... On the validation set , Choose the best model , Then train the best model with all the training sets .

• N- Crossover verification （N-fold Cross Validation）： Divide the training set into N Share , Will this N Training sets train separately , Then find out Average error , choice Average The model with the least error , All training sets will be used to train the model with the minimum average error .

Two 、 gradient descent

• 
  
  ： Loss function （Loss Function）

• 
  
  ： Parameters （parameters）（ Represents a set of parameters , There may be more than one ）

2.1  Adjust the learning rate

2.2  Gradient descent optimization

• SGD(Stochastic Gradient Descent, Stochastic gradient descent )

• Adagrad(Adaptive gradient, Adaptive gradient )
  Learning principles ： Add the squares of the respective historical gradients of each dimension , Then when updating, divide by the historical gradient value, so that the learning rate of each parameter is related to their gradient , Then the learning rate of each parameter is different ： Vulnerable to past gradients , This leads to a rapid decline in the learning rate , The ability to learn more knowledge is getting weaker and weaker , Will stop learning ahead of time .

• RMSProp(root mean square prop, Root mean square )
  Learning principles ∶ The attenuation factor is introduced on the basis of adaptive gradient , When the gradient accumulates , Would be right “ In the past ” And “ Now? ” Make a balance , Adjust the attenuation through super parameters . Suitable for dealing with non-stationary targets （ That is, time related ), about RNN The effect is very good .

• Adam(Adaptive momentum optimization, Adaptive momentum optimization )
  It is the most popular optimization method in deep learning , It combines adaptive gradients
  Handle sparse gradients
  And root mean square
  Good at handling non-stationary targets
  The advantages of , It is suitable for large data sets and high-dimensional space .

2.3  Feature scaling

3、 ... and 、 Limitation of gradient descent

• It is easy to fall into local extremum （local minimal）;

• Stuck is not extreme , But the differential value is 0 The place of （ Stagnation point ）;

• The differential value is close to 0 Just stop , But it's just gentle here , It's not the extreme point .

Four 、 summary