Hi,

In a tracking system I am training an qdc classifier online. This means I do not have a test set and cannot use things like bootstrapping to measure the classifier performance. Still I want to have an indication of the expected performance. Can I do that by using some distance function of the class distributions ? Like the Bhattacharyya distance ?

Thanks in advance and regards,

Rob Ottenhoff

An easy way is to estimate by meancov the density parameters and generate a testset, e.g.

a = gendath; % training set
w = qdc(a); % classifier
[u,g] = meancov(a); % density parameters (means and cov mat)
x = gauss(1000,u,g); % artificial testset of size 1000
x*w*testc % classification performance

Note that this just tells you how much the distributions used by qdc overlap.
This is only an estimate of the performance of the classifier if the true distributions are
really gaussian. You may use gendatp or gendatk to generate distributions that follow
the class distributions somewhat better if they are not Gaussian.

Bob Duin

BobDuin - 18 February 2010 03:28 PM

An easy way is to estimate by meancov the density parameters and generate a testset, e.g.

a = gendath; % training set
w = qdc(a); % classifier
[u,g] = meancov(a); % density parameters (means and cov mat)
x = gauss(1000,u,g); % artificial testset of size 1000
x*w*testc % classification performance

Note that this just tells you how much the distributions used by qdc overlap.
This is only an estimate of the performance of the classifier if the true distributions are
really gaussian. You may use gendatp or gendatk to generate distributions that follow
the class distributions somewhat better if they are not Gaussian.

Bob Duin

Hi Bob,

Thanks for the answer. Indeed I want to know how much the distributions overlap. But I am tracking ‘real time’. There is no room to generate and evaluate such an artificial test set. I am looking for an analytical solution.

Regards,

Rob

Sorry, I misunderstood you. The only thing I can offer is using all the Mahalanobis distances
that can easily be derived from the class means and inverted covariances that are stored in w.data.
This is good as long as the covariances are about equal.

Best regards,

Bob Duin

Hi,

This is what I did: I created (8-dimensional) datasets with 1500 samples, 200 were used for training, the rest for testing(with cleval). I also calculated the Bhattacharyya-distance of the estimated distributions in the training set. (I used an adapted version to handle near-singular covariance matrix cases). Attached is a plot of the results. It seems I can use a threshold on the distance to ensure a reasonable classifier. Makes that sense?

I now have another question:
I noticed that the estimated covariance matrices of my data are nearly always nearly singular. So I use the pseudo-inverse just like qdc does. But actually it means that there are linear dependencies amongst my features, and that means that I can remove some of them. Is there a (fast) way to select the best feature(s) to remove, using the estimated covariance matrix?

Thanks in advance,

Rob

For this dataset a threshold on the Bhattacharyya distance makes sense. As long as you
use qdc and assume normal distributions it might be good. Whether it applies to all your
problems I cannot say.

W.r.t. feature selection you may consider to user your own routine as a criterion
in feateval and rank the features w.r.t. the determinant of the (reduced) covariance
matrix (using the trace?).

Best regards,

Bob