Anybody could tell me the correct way of evaluating the performance in a multi-class classification problem with umbalanced or imbalanced class presence ?

For example in this 3 class problem, I am used to have the following performance measures derived from the confusion matrix:

(this example is attached in results.txt, I dont know why the formatting is so ugly)

True | Estimated Labels
Labels | Normal Suprav Ventri| Totals
-----------------|---------------------|-------
Normal |282094 1534 3991 |287619
Supraventricular| 102 854 378 | 1334
Ventricular | 521 1309 742 | 2572
-----------------|---------------------|-------
Totals |282717 3697 5111 |291525

Balanced Results
-----------------
| Normal || Supravent || Ventricul || TOTALS |
| Se +P || Se +P || Se +P || Acc | Se | +P |
| 98% 78% || 64% 55% || 29% 49% || 64% | 64% | 61% |

Unbalanced Results
-------------------
| Normal || Supravent || Ventricul || TOTALS |
| Se +P || Se +P || Se +P || Acc | Se | +P |
| 98% 100% || 64% 23% || 29% 15% || 97% | 64% | 46% |

(this example is attached in results.txt, I dont know why the formatting is so ugly)

As you can see, the presence of the 3 classes is VERY umbalanced (287619, 1334, 2572), and consequently I noticed that the positive predictive value (+P) is biased by this umbalance. Trying to avoid this, I calculated the “balanced” results, which means scaling all rows in the confusion matrix to sum the same (or balance the class presence), and then calculate the performance. As can be noted, the class sensitivity (Se) is not affected by this row-operations. The totals means the average Se and +P of all classes, and Acc is the trace of the confusion matrix divided by the total amount of examples (291525).

Thanks in advance for any comment.
Mariano.

Dear Mariano,

I don’t really understand the balancing method you describe. Does it mean you scale the confusion matrix values? But that does not alter your classifier.

The path that I’m taking on imbalanced problems is rather to tune the classifier operating point to the situation that counter-weights the large class with ROC analysis.

Check out this knowledge-base example:
http://perclass.com/doc/kb/17.html

Hope it helps,

Pavel

P.S.: you may use code tag to include a piece of code (it’s available as a fast link above the edit window)

pavel - 03 June 2010 02:52 PM

Dear Mariano,

I don’t really understand the balancing method you describe. Does it mean you scale the confusion matrix values? But that does not alter your classifier.

The path that I’m taking on imbalanced problems is rather to tune the classifier operating point to the situation that counter-weights the large class with ROC analysis.

Check out this knowledge-base example:
http://perclass.com/doc/kb/17.html

Hope it helps,

Pavel

P.S.: you may use code tag to include a piece of code (it’s available as a fast link above the edit window)

Thanks Pavel !

I will pleased read the kb article.

Yes, the balanced results means scaling the confusion matrix, or upsample the less represented classes in the test set. The trained classifier is the same, but I can control the training by weighting less represented examples.

BTW, I have another older post in this forum. You probably could have any comment on these.

THanks for the help,
Mariano.

pavel - 03 June 2010 02:52 PM

Dear Mariano,

I don’t really understand the balancing method you describe. Does it mean you scale the confusion matrix values? But that does not alter your classifier.

The path that I’m taking on imbalanced problems is rather to tune the classifier operating point to the situation that counter-weights the large class with ROC analysis.

Check out this knowledge-base example:
http://perclass.com/doc/kb/17.html

Hope it helps,

Pavel

P.S.: you may use code tag to include a piece of code (it’s available as a fast link above the edit window)

So Pavel, the article describe the way of tuning the weights (or priors in prtools qdc mapping object, I am right?) but as long as I know you are not changing the training weights. Because the ROC analysis you do from the trained mapping and test dataset:

r=sdroc(out)

Is this correct ?

Mariano.

Not really. While by setting priors in your probabilistic model you also alter the decision boundary, you do not know how to set the priors to achieve specific performance trade-off. ROC analysis, described in the article, takes a different approach by considering your trained classifier as a black box providing per-class soft outputs. Whatever priors or parameters you set inside, ROC only considers the performances of this black box for different settings of a decision function (output weighting for multi-class discriminants).

The test set is used as a reference to find the desired connection between different sets of weights and test set performances. So yes, you are right that ROC analysis is applied to a trained model on a test set. That’s the only way how to tune classifiers to a specific performance requirements of an application.

Of course, in reality, after you set your operating point, you need to go to an independent test set to really validate how you classification system performs. We do this by writing a custom algorithm which trains a model and performs ROC on parts of the training data and is then tested on a separate test set (see example in sda_pca_clf_roc.m)

Hope it helps,

Pavel