Username Remember Me?
Password   forgot password?
   
   
going deep in a trained classifier
Posted: 10 March 2009 04:49 PM   [ Ignore ]  
Novice
Rank
Total Posts:  14
Joined  2009-02-17

Hi again PRTools team

I want to ask something about the Adaboost implementation of yours. First of I would like to know if it is the same one used by Viola and Jones. If so, is it possible to know which features that are provided are being used? Also how can I tell the weights of the individual week classifiers? Right now I use Adaboost to train classifiers like treec and stumpc. I have a large database so is there a possibility of overfitting when using treec? I hope it’s not too much ;)

Thank you very much

Profile
 
 
Posted: 11 March 2009 07:44 AM   [ Ignore ]   [ # 1 ]  
Administrator
Avatar
RankRankRankRank
Total Posts:  360
Joined  2008-04-26

Hi Thomasson,

I think PRTools implementation of AdaBoost, used with the decision stump, is fairly close to the Viola&Jones;AdaBoost selecting the features and learning the threshold for each. However, there may be some differences, for example in the way the subset if generated in each training step. In PRTools adaboostc, fixed 20% of the training data is taken considering sample weights (see line 81, addaboostc.m). I’m not sure what exact procedure is used in your implementation of Viola&Jones;learning.

Regarding the access to features selected and classifier weights, lets make a small example on 20D artificial problem:

>> a=gendatd(100,20)
Difficult Dataset100 by 20 dataset with 2 classes[57  43]

we train adaboost with stumpc only selecting a feature in each step and learning the threshold
we ask for 10 steps
>> w=adaboostc(a,stumpc,10)
Weighted Voting20 to 2 trained  mapping   --> wvotec

lets look inside the w mapping:
>> 
d=getdata(w)



    
[20x20 mapping]    [1x10 double]

here are stored the trained classifiers:
>> 
d{1}
20 to 20 trained  mapping   
--> stacked

% and here the classifier weights for the eventual voting combiner:
>> 
d{2}

ans 
=

  
Columns 1 through 7

    0.1409    0.0000    0.1517    0.1205   
-0.1478    0.1948    0.0321

  Columns 8 through 10

   
-0.0601    0.0059   -0.0013

% if we look inside the stacked classifierwe find 10 decision treesone for each step:
>> 
d2=getdata(d{1})

d2 

  
Columns 1 through 4

    [20x2 mapping]    [20x2 mapping]    [20x2 mapping]    [20x2 mapping]

  Columns 5 through 8

    [20x2 mapping]    [20x2 mapping]    [20x2 mapping]    [20x2 mapping]

  Columns 9 through 10

    [20x2 mapping]    [20x2 mapping]

inside each tree is a table defining features and thresholds:
>> 
d3=getdata(d2{1})

d3 

    
[3x6 double]    [1]

first columne is feature indexsecond the thresholdThe third and fourth columns give "the next table row" 
to go. (the rows 2 and 3 are terminal for us). The last columns hold estimated class priors
>> d3{1}

ans 
=

    
5.0000   -0.4353    2.0000    3.0000    0.5909    0.4091
    1.0000         0         0         0    0.9000    0.1000
    1.0000         0         0         0    0.9286    0.0714

different treedifferent feature selected:
>> 
d2{5}{1}

ans 
=

    
8.0000   -0.8262    2.0000    3.0000    0.6364    0.3636
    1.0000         0         0         0    0.8889    0.1111
    1.0000         0         0         0    0.9333    0.0667

Regarding the overtraining, I’m not really sure. People sell AdaBoost as the classifier that does not overtrain. I would recommend to see it for yourself on yor problem. Train it and
compare the error estimated on a test set with the apparent error (estimated on the training set). Large difference between both would suggest overtraining.

Hope it helps,

Pavel

[ Edited: 11 March 2009 02:25 PM by pavel]
Profile
 
 
Posted: 11 March 2009 02:23 PM   [ Ignore ]   [ # 2 ]  
Novice
Rank
Total Posts:  14
Joined  2009-02-17

Dear Prtools team

while I was checking what is going on in my classifier I noticed that for training an adaboostc classifier with 200 stupcs the system uses the same features several times. Isn’t it true that ones a feature is used, is excluded from future evaluations? Additionally, training an adaboostc with exactly the same data I get different results. Is there a factor of randomness in the adaboost implementation of prtools?

thanks again for your time

[ Edited: 18 March 2009 01:02 PM by thomasson]
Profile
 
 
Posted: 20 March 2009 08:41 AM   [ Ignore ]   [ # 3 ]  
Administrator
Avatar
RankRankRankRank
Total Posts:  360
Joined  2008-04-26

Dear Thoomasson,

PRTools adaboostc with stumpc base classifier may indeed use the same feature several times as it does not skip already used features. However, each time it attempts to address a bit different classification problem as AdaBoost algorithm focuses on more problematic samples later on during its training. Each stage will, therefore, have a different threshold. That’s maybe the fundamental difference between using AdaBoost as feature selector (Viola & Jones scheme) and as the general classifier fusion strategy (PRTools implementation) you were asking for in your first post.

Regarding the randomness. Yes, there is a random element in AdaBoost in general, because to construct weak learners with high bias and high variance each base classifier is trained only on a subset of the data (20% of the total set in PRTools) - see line 81 in adaboostc.m

You can set the random state before training adaboostc so that the result will be repeatable

>> rand('state',42)
>> 
w1=adaboostc(a,stumpc,10)
Weighted Voting2 to 2 trained  mapping   --> wvotec
>> rand('state',42)
>> 
w2=adaboostc(a,stumpc,10)
Weighted Voting2 to 2 trained  mapping   --> wvotec
>> d1=getdata(w1)

d1 

    
[2x20 mapping]    [1x10 double]

>> d1{2}

ans 
=

  
Columns 1 through 7

   
-0.0000    0.9505    0.2556   -0.1226   -0.0209   -0.3326    0.2841

  Columns 8 through 10

   
-0.2202    0.0000   -0.0000

>> d2=getdata(w2)

d2 

    
[2x20 mapping]    [1x10 double]

>> d2{2}

ans 
=

  
Columns 1 through 7

   
-0.0000    0.9505    0.2556   -0.1226   -0.0209   -0.3326    0.2841

  Columns 8 through 10

   
-0.2202    0.0000   -0.0000

Hope it helps,

Pavel

[ Edited: 20 March 2009 08:43 AM by pavel]
Profile
 
 
Posted: 20 March 2009 01:26 PM   [ Ignore ]   [ # 4 ]  
Novice
Rank
Total Posts:  14
Joined  2009-02-17

Dear Pavel

I was also thinking that it would be that 20% factor you had told me in a previous post. Anyway, one think that troubles me is the size of my feature space. I found some way to reduce it by deleting specific feature. can I do that through prtools? Is there a command where I can say delete feature 2378 6789 9876 etc? I try with featsel by I don’t get any results

Profile
 
 
Posted: 20 March 2009 03:04 PM   [ Ignore ]   [ # 5 ]  
Administrator
Avatar
RankRankRankRank
Total Posts:  360
Joined  2008-04-26

Hi Thomasson,

if you know what features to skip, you can remove them directly from the dataset. You can use, for example, this approach:

lets say we have a dataset with 5 features
>> a=gendatd(100,5)
Difficult Dataset100 by 5 dataset with 2 classes[58  42]

here I remove features 2 and (use setdiff to provide indices of features to preserve):
>> 
b=a(:,setdiff(1:5,[2 4]))
Difficult Dataset100 by 3 dataset with 2 classes[58  42]

If you take this route, you may need to identify your original features with a unique feature indentifier as the indices to the dataset are relative. You can use PRTools “feature labels” for this purpose. You can assign your unique feature identifiers using setfeatlab:

>> a=gendatd(100,5)
Difficult Dataset100 by 5 dataset with 2 classes[54  46]

lets set the feature labels to numerical feature indetifiers (we use 10,20etc to distinguish them from indices you can use any number of string)
>> 
a=setfeatlab(a,10*(1:5)')
Difficult Dataset, 100 by 5 dataset with 2 classes: [54  46]
>> flab=getfeatlab(a) % initial feature labels

flab =

    10
    20
    30
    40
    50

% we want to remove features with identifiers 20 and 40
% lets get the feature indices to preserve:
>> ind=find(~ismember(flab,[20 40]))

ind =

     1
     3
     5

>> b=a(:,ind)
Difficult Dataset, 100 by 3 dataset with 2 classes: [54  46]
>> 
>> getfeatlab(b)

ans =

    10
    30
    50

Does this help you to tag and remove some of the features in your massive dataset?

Best Regards,

Pavel

Profile
 
 
Posted: 20 March 2009 03:16 PM   [ Ignore ]   [ # 6 ]  
Novice
Rank
Total Posts:  14
Joined  2009-02-17

Dear Pavel

Your answer is really helpful. I have a question though. What do you mean that the ‘the indexes to the dataset are relative’? When I create the dataset I use for example a matrix of Nx19125, so if I want to exclude feature 3890 I can use 3890 as an index for the feature right?

Profile
 
 
Posted: 20 March 2009 03:27 PM   [ Ignore ]   [ # 7 ]  
Administrator
Avatar
RankRankRankRank
Total Posts:  360
Joined  2008-04-26

Hi,

yes, index is the same the first time. I was thinking about the situation later on where you have just 19124 features and need to remove the feature originally called 2314. Then you would need to rely on unique feature identifiers instead of indices to the current dataset.

Pavel

Profile
 
 
Posted: 20 March 2009 03:39 PM   [ Ignore ]   [ # 8 ]  
Novice
Rank
Total Posts:  14
Joined  2009-02-17

Thanks a lot

Profile
 
 
Posted: 17 April 2009 03:27 PM   [ Ignore ]   [ # 9 ]  
Novice
Rank
Total Posts:  14
Joined  2009-02-17

Hi again PRTools team

As I continued working on my project it appeared that Adaboostc with stumpc is using almost always the same feature. this is too bad for me. How can I avoid it? can I set the threshold at fixed value? Another thing that troubles me has to do with the weights. I have noticed that negative weights are applied to the features. What does this negative value mean?

Thanks for you time

Profile
 
 
Posted: 02 August 2009 06:19 AM   [ Ignore ]   [ # 10 ]  
Novice
Rank
Total Posts:  2
Joined  2009-07-29

Hi Prtools team

I’m a biomedical engineering student currently working on my final yr project, which involves design and implementation of a sleep apnea diagnosis algorithm (written in matlab with extensive use of PRtools function) using SIMULINK & EMBEDDED MATLAB.

I am having difficulty importing one of the trained mapping dataset file (created by a researcher) as an input to my SIMULINK block (FROM WHAT I KNOW, I THINK SIMULINK ONLY ACCEPTS DATA TYPE DOUBLE), i used the getdata command posted by pavel to extract this mapping file:

>> getdata(nnmap2) %1x2 cell

ans =

[27x27 mapping] [27x2 mapping]

>> nnmap2_1=getdata(nnmap2{1}) %1x1 struct

nnmap2_1 =

rot: [1x27 double]
offset: [1x27 double]
lablist_in: [27x1 double]

>> nnmap2_2=getdata(nnmap2{2}) %1x1 cell

nnmap2_2 =

[1x1 network]

>> getdata(nnmap2_1{1})
??? Cell contents reference from a non-cell array object.

>> getdata(nnmap2_1(1))
??? Undefined function or method ‘getdata’ for input arguments of type
‘struct’.

then i was unable to view anymore details for the trained weights. can anybody please kindly help me identify the individual weights of nnmap2 file? it would be greatly appreciated and helpful to the implementation of the entire algorithm.

thank you very much

Oscar

Profile
 
 
Posted: 02 August 2009 07:43 AM   [ Ignore ]   [ # 11 ]  
Administrator
Avatar
RankRankRankRank
Total Posts:  360
Joined  2008-04-26

HI oscar_xyd,

you should post more details on the mapping (the standard output of Matlab when you type nnmap2 would help). How did you train the mapping? Is is standard PRTools command or your own mapping? Does it depend on 3rd party tools (such as neural network toolbox?)

If your platform supports calling x86 PC DLL library, you could consider deploying your trained classifiers using PRSD Studio. Its components, called libPRSD.dll provides fast execution for a large number of algorithms. More details at: http://prsysdesign.net/index.php/html/software/
If you’re interested in testing this approach, let us know at

To find details of the implementation, look in the m-file implementing the mapping. You can find out which file is it by calling getmapping_file(your_mapping).

Hope it helps,

Pavel

Profile
 
 
Posted: 03 August 2009 11:26 AM   [ Ignore ]   [ # 12 ]  
Novice
Rank
Total Posts:  2
Joined  2009-07-29

Thanks Pavel

Right now im not sure how this nnmap2 was created because it was done by a former researcher at uni, but here are some more details i got:

>> NNMap2
LM Neural Classf, 27 to 2 trained mapping --> sequential

>> struct(NNMap2)

ans =
mapping_file: ‘sequential’
mapping_type: ‘trained’
data: {[27x27 mapping] [27x2 mapping]}
labels: [2x1 double]
size_in: 27
size_out: 2
scale: 1
out_conv: 0
cost: []
name: ‘LM Neural Classf’
user: []
version: {[1x1 struct] ‘11-Aug-2008 02:39:28’}

% Features2: 5000x27 Double,extracted features

>> TestMapNN = Features2*NNMap2
5346 by 2 dataset with 0 classes: []

i don’t think our matlab function depends on any other 3rd party tools, the prtools functions used in the m file are map.m and labeld.m. Our main approach is to re-write the matlab code using embedded matlab so our simulink model can be readily converted into implementable c-code using realtime workshop, and the main problem right now is importing NNMap into simulink environment. would you happen to have any suggestion for this?

the dsp we are using is eZdsp F2812, and im still not sure how to check whether it supports x86 PC DLL library.

sorry if it is too much, thanks again

Oscar

Profile
 
 
Posted: 03 August 2009 12:23 PM   [ Ignore ]   [ # 13 ]  
Administrator
Avatar
RankRankRankRank
Total Posts:  360
Joined  2008-04-26

Hi Oscar,

I see - your platform is a DSP. Then our x86 DLL binary cannot be used.

It seems to me that your NNMap2 is Levenberg-Marquardt neural network (trained using lmnc.m in PRTools but implemented through Neural Network toolbox of Mathworks).  To extract implementation details, you’ll have to dive in the neural network toolbox internals, I’m afraid.

If you encounter difficulties with the extraction of execution info, we may offer help in terms of our consultancy activities. In that case, contact us directly at .

With Best Regards,

Pavel

Profile