Login Register

Auto-login on future visits

Forgot your password?


We have more than 10 years of academic experience in theoretical and applied pattern recognition. Because our close involvement with research continues, PR Sys Design provides state-of-the-art solutions to industrial problems.

ROC analysis for hierarchical classifiers (ICPR 2010)

ROC analysis and cost-sensitive optimization for hierarchical classifiers, Pavel Paclik, Carmen Lai, Thomas C.W. Landgrebe, and Robert P.W.Duin, to appear in proceedings of ICPR 2010, Istanbul, Turkey, 23-26 August 2010.


Instead of solving complex pattern recognition problems using a single complicated classifier, it is often beneficial to leverage our prior knowledge and decompose the problem into parts. These may be tackled using specific feature subsets and simpler classifiers resulting in a hierarchical system. In this paper, we propose an efficient and scalable approach for cost-sensitive optimization of a general hierarchical classifier using ROC analysis. This allows the designer to view the hierarchy of trained classifiers as a system, and tune it according to the application needs.

ROC Skeleton - Pattern Recognition Letters

The ROC skeleton for multiclass ROC estimation, Thomas C.W. Landgrebe and Pavel Paclik, Pattern Recognition Letters, Volume 31 , Issue 9 (July 2010), Pages: 949-958.


Multiclass operating characteristics are a generalisation of the two-class receiver operator characteristic. A limitation regarding this generalisation is the computational complexity with increasing numbers of classes. In this paper the ROC skeleton approach is proposed for efficiently estimating the operating characteristic. New operating points are computed from actual training samples, versus an alternative approach involving grid generation, that is prone to redundant calculations, and poor adaptation to certain classifier architectures. An extensive experimentation with a number of datasets and classifiers as a function of the number of calculations reveals the efficiency of this approach. Also notable is how in many cases good performance can be achieved with surprisingly few calculations, but the converse may also apply.
Keywords: ROC analysis; Operating characteristics; Multiclass ROC; Cost sensitive optimisation

ICPR 2008

Variance estimation for two-class and multi-class ROC analysis using operating point averaging.
P. Paclik, C. Lai, J. Novovicova, R.P.W.Duin. Proc. of the 19th Int. Conf. on Pattern Recognition (ICPR2008, Tampa, USA, December 2008), IEEE Press, 2008.

Slides from the SIMPLI'08 (PDF, 1.3MB) presentation in the ICT group at TU Delft.


Receiver Operating Characteristic (ROC) analysis enables fine-tuning of a trained classifier to a desired performance trade-off situation. ROC estimated from a finite test set is, however, insufficient for the sake of classifier comparison as it neglects performance variances. This research presents a practical algorithm for variance estimation at individual operating points of ROC curves or surfaces. It generalizes the threshold averaging of Fawcett et.al. to arbitrary operating point definition including the weighting-based formulation used in multi-class ROC analysis. The statistical test comparing performance differences between operating points of the same curve is illustrated for two-class and multi-class ROC.

Selected publications

more publications...