Forum Home  >  perClass  >  Technical support  >  Thread
Search  
  
  Advanced Search
Welcome Guest   ·   Login   ·   Register   ·   Member List
Username Remember Me?
Password   forgot password?
   
   
measures in sdcrossval
 
Mathieu
Posted: 05 March 2010 04:14 PM   [ Ignore ]  
Newbie
Rank
Total Posts:  20
Joined  2010-02-18

Hi,

I would like to know the list of ‘measures’ that can be used in sdcrossval. In particular I would like to find a measure equivalent to:

res=sdcrossval(sdknn*sddecide, data, 'measures',{'class-errors'})*getpriors(data)'

Thanks

Mathieu

Profile
 
 
carmen
Posted: 05 March 2010 06:24 PM   [ Ignore ]   [ # 1 ]  
Administrator
Rank
Total Posts:  13
Joined  2008-05-02

Hi Mathieu,

you can find the list of possible measures in PRSD Studio documentation at:
http://perclass.com/doc/guide/ROC_analysis.html#measures

The measure ‘class-errors’ computes the fraction of misclassified samples for each class.
The measure ‘mean-error’ computes the mean of the class errors, using equal priors.
If you want to have the class errors weighted by different priors (you mentioned the ones using getpriors(data)), the fastest way is: 

res=sdcrossval(sdknn*sddecide, data, 'measures',{'class-errors'});
tmp=repmat(getpriors(data),size(res,1),1);
res2=res.*tmp;

is this a good solution?

Carmen

[ Edited: 05 March 2010 06:30 PM by carmen]
Profile
 
 
Mathieu
Posted: 05 March 2010 07:18 PM   [ Ignore ]   [ # 2 ]  
Newbie
Rank
Total Posts:  20
Joined  2010-02-18

Hi Carmen,

Thank you for the quick reply. What you proposed is a good solution, but what I really need is, for every run, the error rate (#error / #data). Since it’s what you get by default by using crossval in prtools, I think it would be useful to have the same measure in prsd studio.

Mathieu

[ Edited: 05 March 2010 07:30 PM by Mathieu]
Profile
 
 
carmen
Posted: 09 March 2010 02:58 PM   [ Ignore ]   [ # 3 ]  
Administrator
Rank
Total Posts:  13
Joined  2008-05-02

Hi Mathieu

in 2.0.9 is possible to use different priors for computation of the mean error. For example:

>> load fruit
>> pd=sdgauss*sddecide
untrained pipeline 2 steps: sdgauss+sdp_decide
>>sdcrossval(pd,a,'folds',2,'measures',{'mean-error',[0.8 0.1 0.1]})
2 folds: [1: ] [2: ] 

ans =

 2-fold rotation

 ind mean (std)  measure
   1 0.10 (0.02) mean error over classes, priors [0.8,0.1,0.1]
   2 0.13 (0.01) mean error over classes, priors [0.3,0.3,0.3]

Note that sdcrossval includes the mean error with equal priors by default

Carmen

Profile
 
 
Mathieu
Posted: 09 March 2010 04:11 PM   [ Ignore ]   [ # 4 ]  
Newbie
Rank
Total Posts:  20
Joined  2010-02-18

Hi Carmen, thank you very much for this upgrade.

In PRSD Studio 2.0.9 is it normal that the function sdtest always return the mean-error in addition to the measure requested in the function, for example:

>> res = sdtest(data, p*sddecide, 'measures',{'mean-error',getpriors(data)})

res =

    0.1690    0.2651

>> res = sdtest(data, p*sddecide, 'measures',{'class-errors'})

res =

    0.0736    0.4566    0.2651

Mathieu

Profile
 
 
carmen
Posted: 09 March 2010 04:36 PM   [ Ignore ]   [ # 5 ]  
Administrator
Rank
Total Posts:  13
Joined  2008-05-02

Hi Mathieu,

yes, the mean error with equal priors is always returned. This is because the code computing the error measures is used also in sdroc. Here we need the mean error to set the default operating point by minimizing that error.
However, it is true that in sdtest is not really necessary to add this measure, when other measures are defined. In the next release, we will consider providing it as default only when no other measures are requested.
thanks,

Carmen

Profile
 
 
   
 
 
‹‹ Use custom feature selection algorithm in a pipeline      Turn off messages in sdcrossval ››