Just as machine vision systems have evolved from traditional monochrome cameras to many systems that now utilize full color imaging information, there has also been an evolution from systems that only captured broadband images in the visible spectrum, to those that can utilize targeted spectral bands in both visible and non-visible spectral regions to perform more sophisticated inspection and analysis.
The color output of the cameras used in the machine vision industry today is largely based on Bayer-pattern or trilinear sensor technology. But imaging is moving well beyond conventional color where standard RGB is not enough to carry out inspection tasks. Some applications demand unconventional RGB wavelength bands while others demand a combination of visible and non-visible wavelengths. Others require exclusively non-visible wavelengths such as UV, NIR or SWIR, with no wavebands in the visible spectrum.
Complex metrology and imaging applications are beginning to demand higher numbers of spectral channels or possibilities to select application-specific spectral filtering at high inspection throughputs. With the traditional machine vision industry merging with intricate measurement technologies, consistent, reliable, high-fidelity color and multispectral imaging are playing key roles in industrial quality control.
Multispectral cameras capture image data at specific frequencies across the electromagnetic spectrum. The wavelengths may be separated by filters or using instruments which are sensitive to specific wavelengths, including light from frequencies beyond our visible sight, such as infrared. Spectral imaging also allows for extraction of additional information which the human eye fails to capture.
Read on to learn more about multispectral imaging solutions and how various camera capabilities can be utilized to meet your application requirements.
The term “multispectral” is often misunderstood or misrepresented. This is solely because there is no fixed definition of “multispectral” in physics textbooks. The literal meaning is imaging with more than one spectral band. By this definition, even an RGB camera could fall under the category of multispectral imaging. A dual-band RGB-NIR camera can also be termed as multispectral in the sense that it covers both the visible and NIR spectral bands.
There is an unsaid understanding within the imaging community that imaging with two to fifteen bands can be termed as multispectral. Having said that, there are companies within the imaging business who extend multispectral to systems having up to 25 wavebands. Despite this somewhat murky situation, there is one point which has been well defined in scientific literature and that is this: multispectral imaging can consist of spectral bands which are discretely positioned from each other. They need not be continuous.
For example, a multispectral imaging system could have two spectral wavebands within the visible spectrum, discretely positioned from each other (e.g., one band in the blue region and the second band in the red region), a third band in the NIR region and a fourth in the SWIR region. Hence multispectral imaging uses a subset of targeted wavelengths at chosen locations within a defined spectral range.
It is this characteristic that most clearly distinguishes multispectral imaging from hyperspectral imaging. These two terms are often used interchangeably, which is confusing and incorrect. Hyperspectral imaging techniques use continuous and contiguous ranges of wavelengths (e.g., 400-1100 nm in steps of 1 nm). Hence, there is a clear and defined distinction between the two types of imaging – multispectral and hyperspectral.
Multispectral imaging enhances inspection capabilities in several applications such as agriculture, medical and other industrial applications which use machine vision cameras. With increasing world population and foreseeable shortage of natural resources, efficient and quality-controlled agricultural production is the need of the hour.
The majority of farmers today use visual inspection as an assessment method to control crop growth and quality. But what a human eye can perceive is limited and subjective. There are many quality assessments which are not only beyond human vision but also beyond conventional RGB color imaging.
Farms that employ a precision agriculture strategy and intelligent farming methods use multispectral cameras to help achieve high efficiency, significantly reduce labor costs and yield more accurate results. Multispectral images are a very effective tool for evaluating soil productivity and analyzing plant health. Viewing the health of soil and crops with the naked eye is very limited and is reactionary. Multispectral sensor technology allows the farmer to see further than the naked eye.
Apart from estimating crop yield, multispectral imaging can help the farmer to view damaged crops and make the necessary modifications to the crop-growth management. Identifying weeds, disease and pests with multispectral imaging is getting popular as early detection helps to optimize methods and resources for good plant growth. Multispectral imaging can also assist in counting plants and determining farm population densities. Not only does it help to provide data on soil fertility but also has tremendous potential in land management related to the production of crops.
On top of plant-growth-related assistance, multispectral imaging combined with deep learning and artificial intelligence can also help to control and measure crop irrigation and monitor livestock.
For inspection of fruit and vegetable produce, multispectral imaging can provide a combination of visible and non-visible wavelengths to measure and analyze the extrinsic features (such as color, texture, surface damage, shape and size) and intrinsic features (such as dry matter content, ripeness, moisture content, sugar and fat content) at the same time.
Similarly, in meat and fish inspection, multispectral imaging assists in analyzing cuts, fat and bone content, inspection of damage to the skin surface and color of the meat.
Multispectral imaging can add tremendous value in medical applications, especially in surgical tasks. Combining color imaging with NIR bands can help to locate and distinguish between tumors and surrounding tissues. The multispectral setup can be achieved in many ways but a very efficient and cost-effective way which also reduces the system complexities is by using prism-based multi-sensor cameras.
For endoscopic, surgical imaging applications, two or three images of different wavebands can be captured simultaneously. The images would be “fused” such that the elements from the non-visible NIR channels are overlaid on the visible RGB image to provide the surgeon with an “augmented” view of the tissues or blood vessels on which he or she was operating.
Multispectral imaging can also help in pharmaceutical tablet manufacturing and packaging, right from granulation to filling and sealing of tablet strips. Combining RGB color with NIR wavelengths allows not only surface inspection of the package and blister but also assists in imaging through the blister packs. This helps in identification, counting and quality analysis of multiple individual tablets which can be determined simultaneously in any given package.
In industrial applications such as PCB inspection, simultaneous imaging of RGB and NIR wavebands is very helpful to not only inspect surface components such as capacitors, transistors and other components but also the buried copper or gold conducting lines. Especially when it comes to recycling of electronics, precise inspection of precious metals and components is very helpful.
In textile and printing inspection, multispectral cameras can help to reproduce and measure accurate color. Not only can samples be accurately color-matched, but various garment materials such as leather, vinyl, plastics, threads, metals and polyester can be identified accurately.
In general, all applications which require discrete wavebands within the visible spectrum, or that need extending from the visible spectrum to NIR bands or SWIR, would benefit from using multispectral imaging.
The first multispectral systems created were either used for scientific imaging from space or for analyzing and digitizing paintings and objects of cultural heritage. The original LANDSAT 1 satellite launched in 1972 was equipped with a four-band multispectral imaging system including visible green and red channels plus two NIR wavebands.
By the time LANDSAT 7 was launched in 1999, the system had expanded to eight multispectral wavebands spanning from visible blue to thermal infrared. These and subsequent multispectral satellites are mostly used for agricultural and environmental analyses ranging from coastal and ocean current observations to vegetation analysis, drought stress, burnt/fire-affected areas, and even cloud cover patterns. These are extremely sophisticated – and expensive – systems, from the optics to the sensors used.
Similarly, advanced multispectral still cameras have been used for many years for applications in art and archaeology. These cameras use up to 18 multispectral wavebands to map and tentatively identify pigments and retouchings on works of art. The images are also used to digitize and/or visually enhance old and faded documents and relics. Conservators can also use multispectral imaging to distinguish original sections from in-paints and to select the proper conservation procedures.
There have been different types of multispectral systems developed over time, based on Fourier transform spectroscopy, liquid crystal tunable filters, wide and narrow band filters, etc. As the various approaches have been refined, they have migrated from ultra high-end satellite and art conservation systems into machine vision cameras offering combinations of resolution, frame rates, and prices that enable them to be used for a wide range of multispectral applications. In this technology guide, we will focus on these camera-based multispectral imaging techniques which are becoming increasingly popular in machine vision applications.
Two (or more) independent cameras (area or line scan)
The original method for adding more spectral range to a machine vision setup was to point multiple cameras at the target. For example, if a fruit producer wanted to inspect color and look for bruising, they might add an NIR camera to their setup in addition to a color camera. But combining the spectral data from both images into a single inspection step was extremely challenging and error prone. Even if the two cameras were placed right next to each other, there was still enough optical parallax to make it nearly impossible to align the pixels from the two images. As a result, any attempt at “fusing” the two images was typically unsuccessful. Instead, most customers treated additional spectral imaging as a completely separate inspection step, with separate camera, lighting, lensing, and mounting, (and expense), and with no way to leverage the image data from any other camera used in the overall process.
Filter-wheel camera (area scan)
The filter wheel camera, also known as a multiple narrow-band filter-based imager, captures multi-channel spectral images by rotating the filters in a filter wheel mounted in front of the sensor or lens. Such filter wheels typically can support up to 12 wavebands. The per-pixel spectral reflectances are then estimated from the multispectral images. The advantage of a filter wheel-based camera is the full spatial resolution per wavelength band. The filters can be customized based on the application requirements and the filter wheel can be modified. The drawbacks of this system include slow and time-consuming imaging, complex image registration, complex geometrical distortion and the high cost of customized filters. There is also the issue of adding a mechanical element (the motorized wheel) to the system which may require periodic maintenance or replacement.
Pixelated multispectral filter array (area scan)
Single-sensor imaging using the Bayer color filter array (CFA) and demosaicing is well established for current compact and low-cost color digital cameras. By extending the concept of the CFA to a multispectral filter array (MSFA) one can acquire a multispectral, and in some cases even a hyperspectral, image in one shot without increased size or cost. This method of capturing is also known as snapshot mosaic imaging. Snapshot mosaic sensors can support anything between 4 and 40 channels in VIS (visible), VIS-NIR and NIR-SWIR wavelengths. Achieving very high pixel-based consistency in batch-to-batch manufacturing has been challenging. The real-world wavebands can have a relatively high crosstalk which can affect the overall spectral sensitivity, pixel dependent noise parameters, and accuracy of spectral reconstruction. The algorithmic correction of these filters is quite complex. More importantly, multispectral demosaicing for the multispectral filter arrays has been a challenging problem because of very sparse sampling of each spectral band in the filter array. The more bands, the lower the spatial accuracy of each band becomes.
Two cameras with a beam splitter (area scan)
One way to work around the issues associated with the multiple independent camera method is to introduce a beam splitter element that can simultaneously capture images on multiple cameras from a common set of optics. With two Bayer pattern cameras for example, two 3-band images can be captured and reconstructed as a 6-channel (2 times RGB) spectral image. Or a Bayer camera can be combined with, say, an NIR camera to produce 4-channel RGB+NIR output. Additional splitters and cameras can be added to capture additional wavebands. This approach alleviates the image capture and image registration issues associated with the basic multiple camera approach. Spectral information can be correlated and combined between multiple captured images. The big drawback is that with multiple cameras in the system, the system becomes physically quite large and expensive. In addition, there is loss of light intensity by using beam splitters. High power illumination is often required for this approach and hence there is a trade-off between high speed and the system´s light sensitivity.
Multi-sensor dichroic prism camera (area or line scan)
While at first glance, this appears to be very similar to the beam splitter approach, there are two very significant differences. First, it is only the sensors, not complete cameras, that are mounted and aligned to the prism faces. This results in a significant reduction in size over the multi-camera beam splitter imaging systems described previously. Second, the prism blocks use hard dichroic coatings which act as interference filters to direct the appropriate spectral ranges from the incoming light to each of the sensors. So instead of splitting the same light to multiple channels and thereby reducing its intensity, each channel receives the full amount of light for the range it needs to capture, whether that is a wide or narrow band in the visible or non-visible regions of the spectrum. Unlike the mosaic approach, full spatial resolution per waveband can be achieved. In area scan scenarios, resolutions up to 3.2 MP at more than 100 fps per waveband are possible today whereas in line scan, cameras with 8192 pixels per waveband at 35 kHz are possible. The major limitation of this approach is the size of the prism – and thus the camera – needed to support multiple large sensors. This can limit the maximum resolution and/or pixel sizes of the sensors that can be utilized.
Multi-line camera (trilinear with filters, quad, TDI-style line scan)
Line scan cameras with multi-line sensors can also be used for multispectral applications. A line scan camera with a trilinear RGB sensor is popular for color imaging applications. A quad-line sensor camera can consist of R-G-B-NIR or R-G-B-monochrome. This is one of the ways to achieve multispectral imaging. The number of lines on a multi-line sensor can vary from 3 to several tens. The most popular cameras today have 8 to 16 lines where each line of pixels has a unique spectral band-pass filter and therefore provides multispectral image capture of up to 16 bands. The same technology can be extended to a TDI style sensor which consists of almost 200 lines divided into 3 or 4 spectral domains. Multi-line cameras can also be fitted with additional optical filters on top of the existing RGB sensors. This approach divides the horizontal line resolution in up to 4 sections, depending on the number of optical filters. A maximum of 15 spectral wavebands is possible by combining 5 optical filters with an RGB sensor. The drawback of this approach is that the higher the number of spectral channels, the lower the horizontal resolution of the system becomes.
Pushbroom cameras for multispectral imaging (line scan)
The pushbroom method, which is traditionally used in hyperspectral cameras, can also be applied to multispectral imaging, adding great flexibility to the number of spectral wavebands that can be captured. The x-λ scanning, i.e., across both the horizontal resolution and the multiple wavelength bands, takes place simultaneously whereas the scanning along the transport direction (the y-axis) is sequential. Capture of full spatial and spectral information line by line is possible with this technology. A pushbroom camera consists of three main components, a lens, imaging spectrograph and a silicon-based image sensor (in the case of VIS-NIR) or an InGaAs sensor (in the case of NIR-SWIR). The imaging spectrograph, which consists of a light dispersing unit and focusing optics, forms the key component of a pushbroom camera. Within the imaging spectrograph, the light passes through the input slit, a collimator, to the dispersing unit and then is focused onto the image sensor providing the x-λ coordinates of a single line. Today, line resolutions of up to 1024 pixels are possible with a free wavelength selection between 5 to 224 bands. The spectral range depends on the type of sensors used but VIS-NIR are popular. Although this technology offers good flexibility, the drawback is speed as the number of channels increase. In full range (224 bands), which is a hyperspectral approach, a frame rate of only 500 Hz. is possible. This is too slow for many industrial applications.
Area scan vs. line scan for multispectral imaging
Among the methods for multispectral imaging explained, there are only a very few that can be used for high speed, industrial applications. In area scan, the multi-sensor prism-based approach is well suited for inspection of mass-produced goods manufactured at high speeds. Other methods for area scan such as pixelated multispectral pixel array (snapshot mosaic) and the filter wheel-based approach are too slow for industrial imaging. Apart from this, the spatial resolution and reconstruction of pixel information using snapshot mosaic cameras is also very challenging.
Filter wheel-based cameras are bulky and consist of several moving parts which reduces the robustness of this approach. Having said that, snapshot mosaic and filter-wheel approaches provide higher numbers of spectral wavebands compared to the multi-sensor prism-based approach. Snapshot mosaic is suitable for agriculture, intelligent farming, and medical imaging applications which do not need good spatial accuracy. Filter wheel-based cameras are particularly well suited for digital archiving of old paintings and classical art. Multi-sensor prism-based cameras work well for precision agriculture, intelligent farming, inline inspection of goods such as fruits, vegetables, meat, sea food and industrial goods, such as food and pharmaceutical packaging, electronics and printed circuit boards.
For multispectral imaging with line scan cameras, there are two main approaches which have good potential. One is using the Pushbroom hyperspectral sensors which allow scaling down from a hyperspectral approach (225 spectral bands) to a multispectral approach (5 spectral bands at a 6.5kHz line rate), which makes the approach usable for industrial medium-speed applications such as inspection of food items, recycling and packed goods.
The multi-sensor prism-based line sensor approach allows very high speeds (up to 77 kHz at 4K pixels) and simultaneous imaging of visible and NIR bands for a combination of up to four spectral bands. The speed makes it possible to use this approach for all high-speed applications based on belt, lane or free-fall sorting.
The third approach – using standard trilinear line sensors with optical filters, reducing the horizontal line resolutions, and achieving 6 to 12 channels – has been trying to make inroads into printing, food, ceramic, and textile inspection for many years but has not been successful due to complicated calibration procedures, low accuracy and difficult to use APIs.
Ease of setup (system integration) - Using multispectral imaging is much more complex than using standard machine vision cameras. In order to setup and integrate different components of a multispectral imaging system, it is important to have good know-how, not only about the camera but also the calibration procedures involving the light source, nature of object to be inspected, and the bottlenecks arising from data processing and correction of image data. The total system integration may not be as complicated as a hyperspectral system, but this really depends on what the user wants to achieve with the multispectral imaging system.
Speed & Resolution: Industrial inspection procedures require high throughput. The readout architectures and structure of many multispectral systems is limited on speed. The speed depends on number of wavelength channels, the type of multispectral technology used, and the interface. The higher the number of spectral bands, the more difficult it is to capture the amount of light required in high speed applications. Spatial resolution can also be a challenge for multispectral imaging, especially when inspecting small objects. Cameras based on snapshot mosaic sensors use interpolation to estimate the missing spatial information from individual pixel values, but this is not very accurate when it comes to inspecting smaller defect sizes. Each application may require a different trade-off between the possible number of multispectral channels and the achievable speed and resolution.
Number of spectral wavebands: The number of spectral wavebands required for an application really depends on the nature of the object to be inspected, the required inspection accuracy, and what accuracy can be achieved on the image processing side by using additional spectral estimation techniques. In some applications such as red edge detection or NDVI analysis, it is clearly known which bands in the red and NIR regions are required to capture the desired data from the plants. This is also true when it comes to plastics and organic materials where the spectral data is well known. Another example is fluorescence endoscopy where the ICG absorption and fluorescence remission band is known. In such cases, it may not be required to go beyond a limited number of bands. However, there are also applications that involve a mix of different materials to be inspected, or multiple spectral bands are required to accurately identify a specific wavelength band, or spectral color measurement applications based on multispectral imaging. Such applications would need a relatively higher number of spectral wavebands.
Flexibility: Flexible or scalable multispectral systems are mainly preferred for applications where different types of materials are inspected on the same machine. Flexibility allows the user to adjust the multispectral imaging system according to the needs of an application. This flexibility is mainly on the number of spectral wavebands required which does increase or decrease the speed of the imaging system. Flexibility on some systems also means lower robustness as there may be changing or moving parts that need to be replaced (e.g. on a filter-wheel approach, the filter wheel can be easily replaced but it adds a moving component in the system which has an influence on the system robustness). On the other hand, there are cameras that have flexibility while manufacturing them but no flexibility once the product is completed. Multi-sensor prism-based cameras have flexibility during the manufacturing process where the desirable spectral response of the camera can be selected based on hard dichroic coatings and base prism parameters. However, once the prism-sensor assembly is made, it cannot be changed. Snapshot mosaic sensor-based cameras have the same logic. Once the multispectral filter array is fixed on the sensor, it cannot be replaced or modified during an inspection task.
Handling of the multispectral data cube and data-streaming: One of the challenges in multispectral imaging is handling the multispectral data cube. This is far less complex than a hyperspectral data cube which can be several hundreds of spectra per pixel, but it is more complex than handling a traditional RGB camera system. The system architecture must have the capability to handle, filter and interpret the multispectral data correctly. The lesser the number of spectral channels, the less complicated this is. The second challenge can arise from the methods used for data streaming from the camera to the processing station. In the case of multi-streaming, which has the advantage that individual data streams can be controlled independently, the challenge is to manage this on the application software. Dealing with multi-streams requires software architectures that can handle two or more streams at the same time. Software designed for only single streams expects a device to either send single frames or multipart payloads which are all available at the same time. Hence for both single frames and multipart payloads, the user can call in a single function and get the images from one stream. However, there are a few platforms, such as JAI’s eBUS Player, which can open the camera device a second or third time in a read-only mode and work with multiple data streams.
System costs – One driver of decision making is always the cost. Compact, user friendly, mass-produced cameras cost less than highly specialized and bulky systems. The cost is also driven by the inspection task that needs to be carried out. Applications which are end-consumer or close to end-consumer-driven, such as inspection of food items and agriculture, are more price sensitive as compared to applications in research, hi-tech, or scientific imaging. Today high-end hyperspectral imaging systems can start with prices around EUR 20,000 per camera system. Multispectral cameras in mass production should be well below the EUR 10,000 mark in order to be commercially attractive. Multispectral cameras based on multiple cameras are more expensive than other approaches such as multi-sensor prism-based cameras or cameras based on multispectral arrays. It is also important to note that the cost discussion must be weighted and driven in relation to the value that multispectral imaging can offer to solve or simplify an existing imaging problem.
Traditional spectrometers used to measure the spectral footprint of objects have been tried and tested for many decades. These are extremely accurate but do not serve a larger field of view and point measurement techniques have not gained much momentum in recent years.
Camera-based imaging techniques are the future of spectral imaging. Still, it is a well-known fact that hyperspectral imaging offers tremendous potential in all areas of imaging and image processing. This is because spectral imaging or spectral analysis is the ultimate physical footprint of any material. In the quest to identify this spectral footprint, hyperspectral imaging as a technology will continue to grow. It has, in fact, come a long way from space and satellite imaging to now be available for many machine vision applications. It is making great inroads in laboratory and research applications. But despite the potential of this technology, it has not been able to achieve many breakthroughs in industrial applications. We do see some areas of industrial imaging where hyperspectral imaging is used and we do hear a lot of hype surrounding the possibilities with this technology, however as explained in Chapter 4, it is still very complex, expensive and technically not up to the mark for most of the industrial inspection applications.
From a business perspective, companies want to implement niche technologies but also want to be affordable to the market at the same time in order to gain a large market share. This is not possible today with hyperspectral imaging. Given the limitations of hyperspectral imaging today, multispectral imaging serves as a bridge technology to industrialized mass production inspection of goods in high throughput, high resolution, high robustness, competitive pricing, easy to use and integrate based on a limited number of spectral wavebands.
Hyperspectral imaging remains very relevant and important for the success of multispectral imaging because it can be used as a core technology to identify the relevant wavelength bands required for a targeted application. A large number of industrial, agricultural and medical applications do not need a vast number of spectral wavebands which are available using the extremely powerful but expensive hyperspectral imaging technology. What is then needed is a multispectral approach that supports a high degree of customizability, where the information extracted from a hyperspectral analysis can be used to create a highly targeted multispectral camera system. While filter wheels offer one means to provide such customizability, a modular, prism-based approach may provide the ultimate solution for multispectral imaging (both area scan and line scan) in in the near future.
Learn more about JAI’s multispectral cameras: