Multispectral imaging for medical and industrial machine vision systems

What is multispectral imaging?

The term “multispectral” is often misunderstood or misrepresented. This is solely because there is no fixed definition of “multispectral” in physics textbooks. The literal meaning is imaging with more than one spectral band. By this definition, even an RGB camera could fall under the category of multispectral imaging. A dual-band RGB-NIR camera can also be termed as multispectral in the sense that it covers both the visible and NIR spectral bands.

There is an unsaid understanding within the imaging community that imaging with two to fifteen bands can be termed as multispectral. Having said that, there are companies within the imaging business who extend multispectral to systems having up to 25 wavebands. Despite this somewhat murky situation, there is one point which has been well defined in scientific literature and that is this: multispectral imaging can consist of spectral bands which are discretely positioned from each other. They need not be continuous.

For example, a multispectral imaging system could have two spectral wavebands within the visible spectrum, discretely positioned from each other (e.g., one band in the blue region and the second band in the red region), a third band in the NIR region and a fourth in the SWIR region. Hence multispectral imaging uses a subset of targeted wavelengths at chosen locations within a defined spectral range.

It is this characteristic that most clearly distinguishes multispectral imaging from hyperspectral imaging. These two terms are often used interchangeably, which is confusing and incorrect. Hyperspectral imaging techniques use continuous and contiguous ranges of wavelengths (e.g., 400-1100 nm in steps of 1 nm). Hence, there is a clear and defined distinction between the two types of imaging – multispectral and hyperspectral.

Multispectral imaging applications

Multispectral imaging enhances inspection capabilities in several applications such as agriculture, medical and other industrial applications which use machine vision cameras. With increasing world population and foreseeable shortage of natural resources, efficient and quality-controlled agricultural production is the need of the hour.

The majority of farmers today use visual inspection as an assessment method to control crop growth and quality. But what a human eye can perceive is limited and subjective. There are many quality assessments which are not only beyond human vision but also beyond conventional RGB color imaging.

Farms that employ a precision agriculture strategy and intelligent farming methods use multispectral cameras to help achieve high efficiency, significantly reduce labor costs and yield more accurate results. Multispectral images are a very effective tool for evaluating soil productivity and analyzing plant health. Viewing the health of soil and crops with the naked eye is very limited and is reactionary. Multispectral sensor technology allows the farmer to see further than the naked eye.

Apart from estimating crop yield, multispectral imaging can help the farmer to view damaged crops and make the necessary modifications to the crop-growth management. Identifying weeds, disease and pests with multispectral imaging is getting popular as early detection helps to optimize methods and resources for good plant growth. Multispectral imaging can also assist in counting plants and determining farm population densities. Not only does it help to provide data on soil fertility but also has tremendous potential in land management related to the production of crops.

On top of plant-growth-related assistance, multispectral imaging combined with deep learning and artificial intelligence can also help to control and measure crop irrigation and monitor livestock.

For inspection of fruit and vegetable produce, multispectral imaging can provide a combination of visible and non-visible wavelengths to measure and analyze the extrinsic features (such as color, texture, surface damage, shape and size) and intrinsic features (such as dry matter content, ripeness, moisture content, sugar and fat content) at the same time.

Similarly, in meat and fish inspection, multispectral imaging assists in analyzing cuts, fat and bone content, inspection of damage to the skin surface and color of the meat.

Multispectral imaging can add tremendous value in medical applications, especially in surgical tasks. Combining color imaging with NIR bands can help to locate and distinguish between tumors and surrounding tissues. The multispectral setup can be achieved in many ways but a very efficient and cost-effective way which also reduces the system complexities is by using prism-based multi-sensor cameras.

For endoscopic, surgical imaging applications, two or three images of different wavebands can be captured simultaneously. The images would be “fused” such that the elements from the non-visible NIR channels are overlaid on the visible RGB image to provide the surgeon with an “augmented” view of the tissues or blood vessels on which he or she was operating.

Multispectral imaging can also help in pharmaceutical tablet manufacturing and packaging, right from granulation to filling and sealing of tablet strips. Combining RGB color with NIR wavelengths allows not only surface inspection of the package and blister but also assists in imaging through the blister packs. This helps in identification, counting and quality analysis of multiple individual tablets which can be determined simultaneously in any given package.

In industrial applications such as PCB inspection, simultaneous imaging of RGB and NIR wavebands is very helpful to not only inspect surface components such as capacitors, transistors and other components but also the buried copper or gold conducting lines. Especially when it comes to recycling of electronics, precise inspection of precious metals and components is very helpful.

In textile and printing inspection, multispectral cameras can help to reproduce and measure accurate color. Not only can samples be accurately color-matched, but various garment materials such as leather, vinyl, plastics, threads, metals and polyester can be identified accurately.

In general, all applications which require discrete wavebands within the visible spectrum, or that need extending from the visible spectrum to NIR bands or SWIR, would benefit from using multispectral imaging.

Multispectral camera technologies

The first multispectral systems created were either used for scientific imaging from space or for analyzing and digitizing paintings and objects of cultural heritage. The original LANDSAT 1 satellite launched in 1972 was equipped with a four-band multispectral imaging system including visible green and red channels plus two NIR wavebands.

By the time LANDSAT 7 was launched in 1999, the system had expanded to eight multispectral wavebands spanning from visible blue to thermal infrared. These and subsequent multispectral satellites are mostly used for agricultural and environmental analyses ranging from coastal and ocean current observations to vegetation analysis, drought stress, burnt/fire-affected areas, and even cloud cover patterns. These are extremely sophisticated – and expensive – systems, from the optics to the sensors used.

Similarly, advanced multispectral still cameras have been used for many years for applications in art and archaeology. These cameras use up to 18 multispectral wavebands to map and tentatively identify pigments and retouchings on works of art. The images are also used to digitize and/or visually enhance old and faded documents and relics. Conservators can also use multispectral imaging to distinguish original sections from in-paints and to select the proper conservation procedures.

There have been different types of multispectral systems developed over time, based on Fourier transform spectroscopy, liquid crystal tunable filters, wide and narrow band filters, etc. As the various approaches have been refined, they have migrated from ultra high-end satellite and art conservation systems into machine vision cameras offering combinations of resolution, frame rates, and prices that enable them to be used for a wide range of multispectral applications. In this technology guide, we will focus on these camera-based multispectral imaging techniques which are becoming increasingly popular in machine vision applications.

Area scan vs. line scan for multispectral imaging

Among the methods for multispectral imaging explained, there are only a very few that can be used for high speed, industrial applications. In area scan, the multi-sensor prism-based approach is well suited for inspection of mass-produced goods manufactured at high speeds. Other methods for area scan such as pixelated multispectral pixel array (snapshot mosaic) and the filter wheel-based approach are too slow for industrial imaging. Apart from this, the spatial resolution and reconstruction of pixel information using snapshot mosaic cameras is also very challenging.

Filter wheel-based cameras are bulky and consist of several moving parts which reduces the robustness of this approach. Having said that, snapshot mosaic and filter-wheel approaches provide higher numbers of spectral wavebands compared to the multi-sensor prism-based approach. Snapshot mosaic is suitable for agriculture, intelligent farming, and medical imaging applications which do not need good spatial accuracy. Filter wheel-based cameras are particularly well suited for digital archiving of old paintings and classical art. Multi-sensor prism-based cameras work well for precision agriculture, intelligent farming, inline inspection of goods such as fruits, vegetables, meat, sea food and industrial goods, such as food and pharmaceutical packaging, electronics and printed circuit boards.

For multispectral imaging with line scan cameras, there are two main approaches which have good potential. One is using the Pushbroom hyperspectral sensors which allow scaling down from a hyperspectral approach (225 spectral bands) to a multispectral approach (5 spectral bands at a 6.5kHz line rate), which makes the approach usable for industrial medium-speed applications such as inspection of food items, recycling and packed goods.

The multi-sensor prism-based line sensor approach allows very high speeds (up to 77 kHz at 4K pixels) and simultaneous imaging of visible and NIR bands for a combination of up to four spectral bands. The speed makes it possible to use this approach for all high-speed applications based on belt, lane or free-fall sorting.

The third approach – using standard trilinear line sensors with optical filters, reducing the horizontal line resolutions, and achieving 6 to 12 channels – has been trying to make inroads into printing, food, ceramic, and textile inspection for many years but has not been successful due to complicated calibration procedures, low accuracy and difficult to use APIs.

Key considerations when selecting camera technology for multispectral imaging

Ease of setup (system integration) - Using multispectral imaging is much more complex than using standard machine vision cameras. In order to setup and integrate different components of a multispectral imaging system, it is important to have good know-how, not only about the camera but also the calibration procedures involving the light source, nature of object to be inspected, and the bottlenecks arising from data processing and correction of image data. The total system integration may not be as complicated as a hyperspectral system, but this really depends on what the user wants to achieve with the multispectral imaging system.

Speed & Resolution: Industrial inspection procedures require high throughput. The readout architectures and structure of many multispectral systems is limited on speed. The speed depends on number of wavelength channels, the type of multispectral technology used, and the interface. The higher the number of spectral bands, the more difficult it is to capture the amount of light required in high speed applications. Spatial resolution can also be a challenge for multispectral imaging, especially when inspecting small objects. Cameras based on snapshot mosaic sensors use interpolation to estimate the missing spatial information from individual pixel values, but this is not very accurate when it comes to inspecting smaller defect sizes. Each application may require a different trade-off between the possible number of multispectral channels and the achievable speed and resolution.

Number of spectral wavebands: The number of spectral wavebands required for an application really depends on the nature of the object to be inspected, the required inspection accuracy, and what accuracy can be achieved on the image processing side by using additional spectral estimation techniques. In some applications such as red edge detection or NDVI analysis, it is clearly known which bands in the red and NIR regions are required to capture the desired data from the plants. This is also true when it comes to plastics and organic materials where the spectral data is well known. Another example is fluorescence endoscopy where the ICG absorption and fluorescence remission band is known. In such cases, it may not be required to go beyond a limited number of bands. However, there are also applications that involve a mix of different materials to be inspected, or multiple spectral bands are required to accurately identify a specific wavelength band, or spectral color measurement applications based on multispectral imaging. Such applications would need a relatively higher number of spectral wavebands.

Flexibility: Flexible or scalable multispectral systems are mainly preferred for applications where different types of materials are inspected on the same machine. Flexibility allows the user to adjust the multispectral imaging system according to the needs of an application. This flexibility is mainly on the number of spectral wavebands required which does increase or decrease the speed of the imaging system. Flexibility on some systems also means lower robustness as there may be changing or moving parts that need to be replaced (e.g. on a filter-wheel approach, the filter wheel can be easily replaced but it adds a moving component in the system which has an influence on the system robustness). On the other hand, there are cameras that have flexibility while manufacturing them but no flexibility once the product is completed. Multi-sensor prism-based cameras have flexibility during the manufacturing process where the desirable spectral response of the camera can be selected based on hard dichroic coatings and base prism parameters. However, once the prism-sensor assembly is made, it cannot be changed. Snapshot mosaic sensor-based cameras have the same logic. Once the multispectral filter array is fixed on the sensor, it cannot be replaced or modified during an inspection task.

Handling of the multispectral data cube and data-streaming: One of the challenges in multispectral imaging is handling the multispectral data cube. This is far less complex than a hyperspectral data cube which can be several hundreds of spectra per pixel, but it is more complex than handling a traditional RGB camera system. The system architecture must have the capability to handle, filter and interpret the multispectral data correctly. The lesser the number of spectral channels, the less complicated this is. The second challenge can arise from the methods used for data streaming from the camera to the processing station. In the case of multi-streaming, which has the advantage that individual data streams can be controlled independently, the challenge is to manage this on the application software. Dealing with multi-streams requires software architectures that can handle two or more streams at the same time. Software designed for only single streams expects a device to either send single frames or multipart payloads which are all available at the same time. Hence for both single frames and multipart payloads, the user can call in a single function and get the images from one stream. However, there are a few platforms, such as JAI’s eBUS Player, which can open the camera device a second or third time in a read-only mode and work with multiple data streams.

System costs – One driver of decision making is always the cost. Compact, user friendly, mass-produced cameras cost less than highly specialized and bulky systems. The cost is also driven by the inspection task that needs to be carried out. Applications which are end-consumer or close to end-consumer-driven, such as inspection of food items and agriculture, are more price sensitive as compared to applications in research, hi-tech, or scientific imaging. Today high-end hyperspectral imaging systems can start with prices around EUR 20,000 per camera system. Multispectral cameras in mass production should be well below the EUR 10,000 mark in order to be commercially attractive. Multispectral cameras based on multiple cameras are more expensive than other approaches such as multi-sensor prism-based cameras or cameras based on multispectral arrays. It is also important to note that the cost discussion must be weighted and driven in relation to the value that multispectral imaging can offer to solve or simplify an existing imaging problem.

Hyperspectral and the future of multispectral imaging

Traditional spectrometers used to measure the spectral footprint of objects have been tried and tested for many decades. These are extremely accurate but do not serve a larger field of view and point measurement techniques have not gained much momentum in recent years.

Camera-based imaging techniques are the future of spectral imaging. Still, it is a well-known fact that hyperspectral imaging offers tremendous potential in all areas of imaging and image processing. This is because spectral imaging or spectral analysis is the ultimate physical footprint of any material. In the quest to identify this spectral footprint, hyperspectral imaging as a technology will continue to grow. It has, in fact, come a long way from space and satellite imaging to now be available for many machine vision applications. It is making great inroads in laboratory and research applications. But despite the potential of this technology, it has not been able to achieve many breakthroughs in industrial applications. We do see some areas of industrial imaging where hyperspectral imaging is used and we do hear a lot of hype surrounding the possibilities with this technology, however as explained in Chapter 4, it is still very complex, expensive and technically not up to the mark for most of the industrial inspection applications.

From a business perspective, companies want to implement niche technologies but also want to be affordable to the market at the same time in order to gain a large market share. This is not possible today with hyperspectral imaging. Given the limitations of hyperspectral imaging today, multispectral imaging serves as a bridge technology to industrialized mass production inspection of goods in high throughput, high resolution, high robustness, competitive pricing, easy to use and integrate based on a limited number of spectral wavebands.

Learn more about JAI’s multispectral cameras:

Learn more about our multispectral prism-based area scan cameras in the Fusion Series
Learn more about our multispectral prism-based line scan cameras in the Sweep+ Series