Username Remember Me?
Password   forgot password?
   
   
3 class design optimal ROC setup
Posted: 17 June 2016 05:54 PM   [ Ignore ]  
Master
RankRankRank
Total Posts:  51
Joined  2010-05-27

Dear the developer,

I am writing to the 3 class design. I have a 3 class data set with the class 12, 13, and 80. I want to design to set up the optimal ROC to get the highest classification accuracy rate (TPr) on 12 and 13, and as high classification confidence rate (Precision) as possible on 12 and 13. The 12 and 13 are important classes, but confusions between the 12 and 13 are acceptable. Confusions betwteen 80 and 12+13 should be reduced as much as possible (to minimal). could you please advise how to fix the ROC point? I have one of the example below. but could not achieve a good performance. thank you so much!

r=sdroc(out,’ops’,ops,’measures’,{’TPr’,’12’,’precision’,’12’,’TPr’,’13’,’precision’,’13’,’TPr’,’80’,’precision’,’80’})
r2=constrain(r,’TPr(12)’,0.88,’precision(12)’,0.79)
r3=setcurop(r2,’max’,’TPr(80)’)

Profile
 
 
Posted: 20 June 2016 02:39 PM   [ Ignore ]   [ # 1 ]  
Administrator
Avatar
RankRankRankRank
Total Posts:  356
Joined  2008-04-26

Dear Liwei,

one possible way could be to several (or all) of your measures like you do and then do cost-sensitive optimization of the op.point. This means, to create a cost matrix (same size as confusion matrix) and fill in some costs (penalties for specific errors).

For example, if you want to avoid that genuine “12” and “13” examples will be misclassified as “80”, you could use the following specification (I use “fruit” data relabeled into classes 12,13 and 80):

>> rand('state',1); [tr,ts]=randsubset(b,0.5)
'Fruit set' 999 by 2 sddata3 classes'12'(333'13'(333'80'(333
'Fruit set' 1001 by 2 sddata3 classes'12'(334'13'(334'80'(333)

Training model on one part:

>> p=sdkmeans(tr,30)
[class '12' 11 iters.  7 clusters pruned,  23 clusters] [class '13' 10 iters.  12 clusters pruned,  18 clusters] [class '80' 9 iters.  7 clusters pruned,  23 clusters] 
sequential pipeline       2x1 
'1-NN (k-means prototypes)+Decision'
 
1 1-NN (k-means prototypes)    2x3  64 prototypes
 2 Decision                3x1  weighting
3 classes

Estimate ROC, store confusion matrices:

>> r=sdroc(ts,p,'measures',{'TPr','12','precision','12','TPr','13','precision','13','TPr','80','precision','80'},'confmat')
ROC (2000 w-based op.points7 measures), curop108
est
1:TPr(12)=0.992:precision(12)=0.983:TPr(13)=0.934:precision(13)=0.895:TPr(80)=0.886:precision(80)=0.937:mean-error=0.07

>> sdconfmatts.labts*(p*r) )

ans =

 
True      Decisions
 Labels    
|      12      13      80  Totals
-----------------------------------------------
 
12        |    329       3       2   |    334
 13        
|      5     310      19   |    334
 80        
|      3      36     294   |    333
-----------------------------------------------
 
Totals    |    337     349     315   |   1001

To cost-optimize, we create a cost matrix putting higher cost on diagonal (want to preserve accuracies) and 12/13 to 80 misclassification:

>> M=zeros(3); M(1:4:end)=1M(1:2,3)=4

=

     
1     0     4
     0     1     4
     0     0     1

find op.point minimizing loss using our cost matrix:
>> 
r2=setcurop(r,'cost',M)
ROC (2000 w-based op.points7 measures), curop1356
est
1:TPr(12)=0.992:precision(12)=0.973:TPr(13)=0.974:precision(13)=0.825:TPr(80)=0.786:precision(80)=0.987:mean-error=0.09

confusion matrix:
>> 
sdconfmatts.labts*(p*r2) )

ans =

 
True      Decisions
 Labels    
|      12      13      80  Totals
-----------------------------------------------
 
12        |    330       4       0   |    334
 13        
|      4     324       6   |    334
 80        
|      5      69     259   |    333
-----------------------------------------------
 
Totals    |    339     397     265   |   1001

We lowered the errors, although not perfectly.

If we use too high cost, the solution will become useless:

>> M=zeros(3); M(1:4:end)=1M(1:2,3)=10000

=

           
1           0       10000
           0           1       10000
           0           0           1

>> r3=setcurop(r,'cost',M)
ROC (2000 w-based op.points7 measures), curop1154
est
1:TPr(12)=0.142:precision(12)=1.003:TPr(13)=1.004:precision(13)=0.365:TPr(80)=0.056:precision(80)=1.007:mean-error=0.60
>> sdconfmatts.labts*(p*r3) )

ans =

 
True      Decisions
 Labels    
|      12      13      80  Totals
-----------------------------------------------
 
12        |     47     287       0   |    334
 13        
|      0     334       0   |    334
 80        
|      0     315      18   |    333
-----------------------------------------------
 
Totals    |     47     936      18   |   1001

Profile
 
 
Posted: 20 June 2016 02:42 PM   [ Ignore ]   [ # 2 ]  
Master
RankRankRank
Total Posts:  51
Joined  2010-05-27

thank you! it seems very helpful.

Profile
 
 
Posted: 20 June 2016 10:18 PM   [ Ignore ]   [ # 3 ]  
Master
RankRankRank
Total Posts:  51
Joined  2010-05-27

Dear the developer,

thank you. I looked at your example.

1). You use 50% data as the training, 50% data as the ROC estimation. I use 70% data as the training and ROC estimation; then use 30% for testing the classifier performance.
which way is better?

2). If I use 50% data as the training and 50% data as the ROC estimation. Is there any way that I could retrain the classifier using all the 100% data to make the classifier more robust? which operating point should I use if I retrain the classifier using all the 100% data?

Look forward to hearing from you as soon as possible.

Liwei

Profile
 
 
   
 
 
‹‹ sdconfmat issue      Deep learning test ››