perClass Documentation
version 5.0 (21-sep-2016)

kb31: Measuring classifier speed

Published on: 25-sep-2014

perClass version used: 4.3 (24-jun-2014)

Problem: How to reliably measure classifier execution speed?

Solution: Use sdexe command to measure only the buffer processing, not overhead of runtime setup.

31.1. Introduction ↩

perClass was designed to provide fast classifier execution in real-time applications, such as on-line sorting or defect detection. Apart of accurate classifier, the major requirement in such applications is fast processing. The classifier has only a limited time to return the results, typically few milliseconds. In this article, we discuss how to reliably measure classifier execution speed.

31.2. Speed measurement in Matlab ↩

Matlab provides tic/toc commands to measure time needed for execution of a block of code.

Let us make a small example. We will train a Parzen classifier on fruit data set:

>> a
'Fruit set' 260 by 2 sddata, 3 classes: 'apple'(100) 'banana'(100) 'stone'(60) 
>> p=sdparzen(a)
.....sequential pipeline       2x1 'Parzen model+Decision'
 1 Parzen model            2x3  260 prototypes, h=0.8
 2 Decision                3x1  weighting, 3 classes

To time execution, we run it on 10000 random samples:

>> data=rand(10000,2);

>> tic; data*p; toc
Elapsed time is 0.095310 seconds.
>> tic; data*p; toc
Elapsed time is 0.092284 seconds.
>> tic; data*p; toc
Elapsed time is 0.093810 seconds.

The tic/toc timing showed that our classifier can process 10000 samples in about 94 milliseconds.

This, however, includes also the time needed to setup large memory buffers and preparing classifier for execution. In our deployed application, we are performing such tasks as well, but only once. What is more important is the time needed to process 10000 samples in a continuous manner.

In perClass, we can measure that using the sdexe command. It returns, as the second output, the time needed only to process the input buffer (without setup and allocation time).

>> [out,t]=sdexe(p,data);
>> t

t =


>> [out,t]=sdexe(p,data);
>> t

t =


>> [out,t]=sdexe(p,data);
>> t

t =


As we can see, the relevant processing time only 54 milliseconds.

31.3. Speed measurement out of Matlab ↩

When we export our classifier for execution using sdexport, we may measure the execution time out-of-Matlab.

>> sdexport(p,'/tmp/out.ppl')
Exporting pipeline..ok
This pipeline requires perClass runtime version 4.0 (11-jul-2013) or higher.

The fastest way is to use the sdrun utility, bundled with perClass in SDK/interfaces/sdrun directory.

When provide with the pipeline file, it only prints pipeline details:

$ ./sdrun /tmp/out.ppl 
Pipeline name: 'Parzen model+Decision'
Minimum required runtime version: 4.0 (11-jul-2013)
Input type: double, dimensionality: 2
Output type: int, dimensionality: 1, decisions
Operating point count: 1, current: 1
Possible decisions: 1:apple, 2:banana, 3:stone

When called with the -t switch, sdrun will perform time measurements. By default on 100000 samples. We may provide a different buffer size as an extra argument:

$ ./sdrun /tmp/out.ppl -t 10000
Measuring execution speed for 'Parzen model+Decision'
double precision, 2D input: ..........
Average speed on 10 rounds of 10000 samples: 56.0 msec.
Total throughput: 178.51 kilo samples/sec.

The timing is performed ten times, only measuring the time needed to process data in an existing buffer. As we can see, also here we get average speed of 56 milliseconds.

31.4. Speed measurement via perClass Runtime API ↩

Finally, you may perform precise speed measurements via sd_Tic and sd_Toc functions of perClass Runtime API.

double elapsed_sec;
/* your code to time */

31.5. Summary ↩

When measuring execution speed of your classifiers, make sure you measure only the time needed to perform processing of input buffer in a tight control loop of your system. This can be easily done via sdexe command in Matlab or sdrun utility on the command line.