当前位置：网站首页>Watermelon book chapter 2 Notes - Performance Measurement

Watermelon book chapter 2 Notes - Performance Measurement

2022-07-21 22:41:00 【kurok_】

This article only discusses the measurement methods of prediction tasks .

Return to the task

The mean square error is often used in regression tasks :

$E(f;D)=\frac{1}{m}\sum^m_{i=1}(f(x_i)-y_i)^2$

A more general , For data distribution D And the probability density function p(), The mean square error can be expressed as

$E(f;D)=\int_{x\sim D} (f(x)-y)^2p(x)dx$

Classification task

1. Error rate and accuracy

Error rate ： The proportion of the total number of samples in the sample station with wrong classification

precision ：1- Error rate

There is also a form of probability distribution , Similar to the return mission .

2. Precision rate 、 Recall rate 、F1

Real examples (TP)、 False positive example (FP)、 True counter example (TN)、 False counter example (FN)

Precision rate P=TP/(TP+FP), In the positive example of judgment , It's really the proportion of positive examples

Recall rate R=TP/(TP+FN), In all positive examples , The proportion judged to be true

P-R curve , Can roughly judge which model is good , Generally, if A All inclusive B shows A good , But it's rare , You can compare the area , It can also be compared Balance point (BEP), namely P=R The value of time , The bigger the better .

however BEP It's still too simplified , What is commonly used is F1,F1 yes P and R Of Harmonic mean

$\frac1F=\frac12*(\frac1P+\frac1R)$

In some cases, more attention may be paid to precision or recall . You can give F1 Assign different weights , as follows ：

$\frac{1}{F_{\beta}} = \frac{1}{1+\beta^2}(\frac1P+\frac{\beta^2}{R})$

\beta Greater than 1 When the recall rate is high , Less than 1 The hourly accuracy rate accounts for a high proportion .

in addition ：macroF1 Indicators like that , See watermelon book P32.

3.ROC And AUC

True case rate TPR=TP/(TP+FN), In all positive examples , The proportion judged

The false positive rate is FPR=FP/(TN+FP), Among the samples with wrong judgment , The proportion of positive cases

Choose different decision thresholds , You can draw multiple TPR and FPR

With TPR and FPR The graph drawn for the coordinate axis is ROC diagram ,ROC The area of the curvilinear siege is AUC.

AUC The meaning of ？ In form , Consider the sample prediction is the sorting quality .

From the definition of sequencing loss (P35), If the prediction of the positive example is less than that of the negative example , Then write down a penalty , If equal, then remember 0.5 Penalty points .

AUC=1- Sort loss( Understand graphically )