In this application, we have a food product stream with possible contamination. Both, the product and contamination can be represented by multiple material types. Product consists of two types of nuts, namely wallnut and pistacia. Foreign objects, appearing in product stream, may be more variable including nut shells, (olive) kernels, wood chops, leaves, or stones.

Application requirement is to remove foreign objects on a conveyor belt. Therefore, it is not sufficient to only classify product/contamination per-pixel but also to identify coordinates of large-enough objets for automated picking.

We used Specim FX17 camera (900-1700nm) to acquire images with relevant object examples. The Specim LUMO software was employed to save example scans.

‍

We load the scans into perClass Mira GUI, define classes of interest and build a statistical classification model. In our specific case, we defined a three-class problem considering a background class, and separate product and foregign object classes. Note, that the last two are complex by definition as they encompass multiple material types. This is a standard approach adopeted in perClass Mira, because the software is designed to deal with multi-modal multi-class problems. In other solutions for industrial hyperspetral imaging, class definitions need to be typically very narrow, each specifically targetting one material subtype.

In this screenshot, we can see an example labeled image

perClass Mira trained models in fully automatic manner. The user may improve and fine-time model performance but is not required to do any programming or have machine learning expertise.

Decisoins of a model, trained in perClass Mira, is shown on the following screen-shots with both the label image (note no labels as this is an independent test data) and per-pixel classifier decisions.

The following screenshot shows object detection result: For both the product and foreign object classe, connected regions larger than 500 pixels are reported, smaller objects are discared. This is a user-defined setting avoiding tiny scraps that cannot be handled by actuators. Note, that both classes of interest are flagged as foreground in the class list (the 'F' flag next to class name). In perClass Mira, each foreground class is handled separately, so touching objects of different classes are properly isolated (see the walnut in the left central part and the leaf fragment behind it: Both get isolated as separate objects).

We have validated the model performance on multiple unseen scans loaded in perClass Mira environment.

To deploy the solution in our custom setup, perClass Mira Runtime DLL is used. Typically, a competent C/C++ programmer can follow included examples and integrate model execution on live data within an hour. In our case, we build a command-line application using Specim Specsensor SDK to communicate with the FX camera and perClass Mira Runtime DLL for data processing.

perClass Mira Runtime accepts directly raw spectral frames arriving from the sensor. In our situation, each FX17 spectral frame contains 640 spatial pixels and 224 spectral bands. Internally, perClass Mira Runtime performs dark/white correction by a user-defined scan saved on disk. The reference information can be changed anytime at runtime to allow re-calibration in the production system. This design is of crucial importance for hassle-free production and maintenance as it allows one to handle illumination changes or light source replacements. Note, that this functionality is not available in some other hyperspectral imaging products on the market where dark/white references are fixed in the design stage and changing them requires re-building the models.

perClass Mira Runtime may provide per-pixel decisions. In our use-case it is, however, more important to extract object information. After each frame is processed, the software provides more details on (possibly multiple) objects found for a given frame. For each object, its coordinates, size and class are returned. This information may be used to coordinate actuator action needed to remove foreign objects. Note, that the perClass Mira Runtime does not require its user to deal with difficult problems such as data buffering and spatial processing of line-scan stream. The user only provides frames to the runtime as they come and reads out the object details. No data copies, double buffers or external machine vision tool-chains are needed.

The perClass Mira Runtime is available on both 64bit Windows and Linux. In order to assess data processing speed, we build a simple benchmark where raw spectral cube is loaded in memory and individual raw frames are passed to the runtime. We let the system process 10000 frames (skipping the first 500 for warm-up) and report the best of five runs. On a Linux workstation with NVIDIA GTX1080, the model spends on average 1.45 ms per spectral frame (i.e. processes 691 frames per second). This covers the entire processing from data correction to multi-class object segmentation with small object removal and coordinate measurement. See the screenshot:

The graphical window shows the content of an internal processing buffer with per-pixel decisions. The crosses highlight detected object positions. Saving the buffer content as a PNG image file from a running application is possible via the public perClass Mira Runtime API call. This allows integrators to easily visualize what the algorithms really do even in headless systems.

Additional functionality, useful in production deployments, is easy switching between computational backends. The options include CPU with multi-core support, NVIDIA GPU using CUDA and also Open CL (for example using integrated intel GPU).  It is also possible to obtain precise time measurements from the runtime API.

The work-flow, described in this application note, was realized with publicly available perClass Mira 1.4  for Windows 64-bit or Linux 64-bit operating systems.

For further information, contact Dr.Pavel Paclik, pavel.paclik@perclass.com, tel: +31(0)648060368.

‍