Username Remember Me?
Password   forgot password?
   
   
Simple question on calculating probabilities
Posted: 16 October 2014 07:58 AM   [ Ignore ]  
Novice
Rank
Total Posts:  9
Joined  2009-06-02

Hi!

Suppose you trained a classifier and on your testset you get a confusion matrix like:
T\E 0 1
0 90 10
1 5 95

Now you test an object and your classifier (C) gives you the probabilities
P(C=0) = 0.8 and P(C=1) = 0.2
Normally you stop there and say the probabilities for the object (O) are the same as the classifier output and thus P(O=0)=0.8 and P(O=1)=0.2

My question is: in the final assesment shouldn’t you take the probabilities of classification into account?

Then you get (ignoring priors):
P(O=0) = P(O=0|C=0)*P(C=0)+P(O=0|C=1)*P(C=1) = (90/95)*0.8+(10/105)*0.2 (about 0.78)
and
P(O=1) = P(O=1|C=0)*P(C=0)+P(O=1|C=1)*P(C=1) = (5/95)*0.8 + (95/105)*0.2 (about 0.22)

Any ideas?

Regards,

Rob

Profile
 
 
Posted: 16 October 2014 01:17 PM   [ Ignore ]   [ # 1 ]  
Administrator
Avatar
RankRankRankRank
Total Posts:  355
Joined  2008-04-26

Hi Rob,

my first thought is that you’re suggesting to first train a classifier and then update it (its decision rule) based on information estimated from a separate test set (conf.mat). I think it’s an interesting idea and I don’t recall seeing this before. However, you end up with a new classifier (the soft outputs P(C=0) and P(C=1) are possibly already computed using Bayes equation inside the original model, now you’re bringing an additional one on top).
I think that, if you follow this path, you will need yet another test set to validate performance of your final classifier.

Kind Regards,

Pavel

Profile