Hi everyone, and thanks for reading.

I am working in an automatic classifier where the classification problem itself has these characteristics:

- Several events (feature vectors) from subjects for training, and several for testing. The amount could be very different from subject to subject.
- 3 classes, very unbalanced. Lets say in term of the class representation 10-2-1

I am working with linear and quadratic classifiers for now. I dealed with problems designing the first classifier specially during the training to overcome the problem of the most represented classes to bias the trainig in its favor, and I am quite happy with the results.

The doubt I have now is regarding the most correct way of evaluating the classifier performance, since the problem is unbalanced in the subject presence, the classes present by subject, and the class presence in the whole test dataset. It seems to me that doing a simple evaluation of the trained classifier in the test dataset, will result in a biased confusion matrix (CM), at least because of the classe unbalance. Maybe the class sensitivity will not be unbalanced, but probably the class positive predictivity. By the moment, I tried some strategies of simulating a balancing of the classes in the CM (all rows sums the same in the CM), and then calculating the performance. Or groping by subjects, calculating CMs by subject and averaging the performance estimates. Then I have three persepctives of the classifier performace, but I guess this could be done more correctly, am I right ?

Thanks in advance for any comment, or just for reading :)
Mariano.

Hi Mariano,

have you considered estimating ROC characteristic for your classifier using the performance measures of interest (probably sensitivity and precision)?
You could then select the operating point to satisfy your needs (e.g. constrain precision to be above certain value) and then evaluate your classifier decisions on an independent test set in terms of really observed sensitivity/precision?

See this example to choose specific measures in ROC analysis:
http://perclass.com/doc/guide/ROC_analysis.html#measures

and this section on selecting operating point by constraints:
http://perclass.com/doc/guide/ROC_analysis.html#optim

You’re mentioning ‘subjects’ - if you problem involves generalization to unseen persons, probably the best way to do training and test splitting is over persons in leave-one-subject-out fashion?

Hope it helps,

Pavel

Thanks for the advice Pavel. I considered the ROC characteristic and I was able of selectign a working point, in terms of sensibility and positive predictive value (P+) (similar to specificity in this multi-class problem). The problem I am trying to describe, is that the P+ performance estimates may be biased by the class representation in the test dataset. For example the test dataset has class A, B and C, and suppose that classsize(Test_dataset) is [10000 500 200]. It is clear than failing a fix amount in each class will degrade more the performance of the less represented class. In the original post I commented some approaches to overcome this, but I am not sure if there is another more correct way, like sampling the test dataset to balance the class representation.

Mariano.