perClass Documentation
version 5.1 (31-May-2017)

kb6: Can ROC analysis be performed for multi-class problems?

Keywords: ROC analysis, operating points, discriminants, multi-class

Problem: How to perform ROC analysis in multi-class problems?

Solution: The ROC analysis is performed by output weighting using the sdroc, similarly to the two-class problem

First, we generate a multi-class dataset and divide it into training and test sets. Second, we train the mixture of Gaussian classifier and execute it on the validation set, storing its soft outputs.

>> a=sddata(gendatm(1000)) 
'Multi-Class Problem' 1000 by 2 sddata, 8 classes: [122  123  110  146  114  138  129  118]
>> [tr,ts]=randsubset(a,0.5); 
>> p=sdlinear(tr); 
>> out=ts*p 
Multi-Class Problem, 498 by 2 sddata, 8 classes: [65  55  55  72  58  59  68  66]

Multi-class ROC is estimated using the sdroc command analogously to the two-class case. By default, the sdroc estimates a sub-optimal ROC using a greedy optimizer:

>> r=sdroc(out) 
>> sddrawroc(r)

Use cursor keys to flip through the available per-class error measures in the ROC plot. The weights for a given operating point may be shown in the plot title by pressing the 'w' key. Press the 's' key (save) on the keyboard to store the selected operating point as default.