Introduction
New developments in sensor technology and computer processing are rapidly enhancing consumer digital camera performance, leading to a wider adoption of imaging in everyday use. Additionally, image processing techniques based on artificial intelligence and machine learning are increasing the capabilities of cameras and smart devices for tasks such as object detection based on colour and two-dimensional geometric data. However, these conventional RGB-type cameras only make use of a limited range of wavelengths in the visible part of the spectrum, so are missing out on a lot of the available spectral information.
Spectral imaging combines the power of digital imaging and spectroscopy capturing the same geometric image but in multiple narrow spectral bands that can cover the broader visible, near infrared and shortwave infrared spectral ranges. This allows for identification of optical features of objects that are invisible to conventional cameras or the human eye. Spectral features can be directly attributable to the chemical properties of an object. Therefore, spectral imaging can enable tasks such as object detection, identification and classification, object segmentation and improved colour characterisation.
Imaging spectroscopy is often divided into two categories: multispectral and hyperspectral. Multispectral cameras measure light in a small number (typically 3 to 15) of spectral bands whereas hyperspectral cameras collect a much larger number (up to several hundred) of distinct, yet contiguous bands across a wide spectral range.
Traditionally, hyperspectral imaging has found applications in remote sensing and agriculture using scanning or push-broom cameras installed on satellites, aircraft, or UAVs. In recent times, developments in spectral imaging technology have been evolving at a pace. Advances in high resolution sensors, electronics and optics are providing enhanced push-broom technologies as well as enabling numerous other forms of spectral imaging. Alternatives to push-broom systems include methods using tuneable spectral filters as well as designs based on Fourier Transform spectroscopy. There are also snapshot spectral imaging systems employing mosaic arrays of filters and light field camera designs offering video-rate imaging spectroscopy. These technologies offer specific strengths that can bring some advantages compared to the traditional push-broom cameras and have the potential to facilitate many new and exciting applications.
The following paper provides a glimpse of some new developments within hyperspectral imaging technology. It is not intended to be a comprehensive review but highlights what is a rapidly changing imaging landscape.
Simply the Best: Traditional Push-Broom Cameras for the Highest Quality Data
High quality scientific hyperspectral imaging data is required when looking for subtle differences in spectra. Cameras based on push-broom architectures typically provide the best quality data but even within this category there are varying levels of performance. HySpex cameras, manufactured Norsk Elektro Optikk, (NEO) are a good example of high-quality devices that provide optimal spectral fidelity and sharp imaging optics. The care and attention taken to minimise optical distortions such as spatial and spectral misregistration and stray light are fundamental to final image quality and the ability to identify spectral features.
Internal Push-Broom Systems: from the Laboratory to the Field
Typically, for laboratory or field measurements, push-broom imaging applications require the sample to move within the view of the camera, either on a conveyor belt or by using an external translation stage or rotating tripod head. However, cameras have been developed with the scanning mechanism built into the camera body (the mechanical system moves the spectrometer entrance slit internally). With these cameras there is no need for an external translation stage or moving conveyor belt, instead the camera remains static. This can be useful when space is limited, and allows the camera to be easily attached to a microscope for hyperspectral microscopy applications. Examples of cameras using this approach include the VNIR and SWIR SnapScan cameras from Imec (Figure 2) and the VNIR and SWIR SOC-710 cameras from Surface Optics Corporation.
Tuneable Filter Cameras for High Spatial Resolution and Microscopy Applications
Hyperspectral cameras using liquid crystal or acousto-optical tuneable filters (AOTFs) have been available for many years. In this approach the spectral bands are transitioned sequentially to create a resulting spectral hypercube. More recently, tuneable filter cameras based on piezo-actuated and MEMS Fabry-Perot interferometry (FPI) have been developed by companies including Hinalea Imaging and Unispectral Ltd.
A Fabry-Perot interferometer consists of two parallel reflective mirror surfaces with a gap between the mirrors. The FPI acts as a tuneable bandpass filter with the transmission band determined by adjusting the size of the air gap. A spectral image is created by placing the filter in front of an imaging sensor and sequentially scanning through a range of spectral bands. The acquisition time depends on the number of spectral bands acquired and the time required to collect light at each band. These imagers are especially attractive for applications requiring high spatial resolution as they utilise all available pixels on the camera sensor.
The FPI technology can be easily integrated into small camera designs which has previously been difficult to achieve with some of the older tuneable filter technologies. MEMS-based FPI sensors allow further miniaturisation and integration onto mobile phone camera sensors. The Monarch sensor developed by Unispectral integrates optics, image sensor and controllers on a 60 x 40 x 14.5mm printed circuit board. It provides multispectral images in the 680nm to 940nm NIR wavelength range.
Fourier Transform Hyperspectral Imaging: Great for Low Light Conditions
Fourier Transform (FT) hyperspectral imagers combine a monochrome imaging camera with a Fourier Transform spectrometer. The FT approach results in high optical throughput, due to the absence of slits and gratings. Signal to noise is good as all the wavelengths are measured simultaneously, maximising the number of photons reaching the sensor.
The Hera camera from NIREOS uses a common-path birefringent interferometer design where the light is split into two co-linear delayed replicas that share a common optical path. The resulting interference pattern is measured by a detector as a function of their relative delay and the Fourier Transform of the interferogram yields the intensity spectrum of the light (Figure 5).
This camera works in a staring mode without the need for a translation stage or scanning. It has potential advantages for hyperspectral imaging in low level light conditions so is well suited to fluorescence imaging although it has also been shown to produce high quality spectral data for remote sensing and in art and cultural heritage applications.
Snapshot Hyperspectral: Hyperspectral Imaging at Video Rates
There are exciting new developments in the field of video rate spectroscopy with advances in the implementation of “light field” technology. The Ultris X20 camera from Cubert GmbH (Figure 6) incorporates a continuously variable bandpass filter, microlens array and an Ultra-HD CMOS sensor with 20 Megapixel resolution. In the light field design, both the intensity and direction of incident light rays are used to produce spectral images. The result is an image of 400 × 400 pixels, each with 130 spectral bands. That is an impressive 160,000 pixels, each with 130 spectral bands covering 350-1000nm. The entire spectral dataset is obtained during a single detector integration period, so truly snapshot. This spectral data can be captured in real-time at multiple frames per second thus providing a unique video hyperspectral imaging system.
The smaller Ultris 5 camera from Cubert is just 30x30x50mm and weighs 120g. This provides 50 spectral bands and 250 x 250 pixel resolution. The Ultris 5 camera produces snapshot hypercubes at 15Hz and is suitable for multiple applications including industrial, food quality, medical, remote sensing and machine vision.
A similar light field camera for multi-spectral imaging has been developed by Surface Optics Corp. Their LightShift camera also uses a microlens array and high-resolution imaging sensor but incorporates an array of 16 (4×4) bandpass filters at the entrance aperture. The filter array can contain polarizing as well as bandpass filters and is interchangeable so the camera can be re-tasked for different applications.
The volume of data generated by these light field snapshot systems is a key challenge, but they can run at video rates with real-time classification. This real-time imaging combined with video spectroscopy offers a real game changer for situations where there are dynamic processes and movement within a scene. There is also great potential to drive down costs in the future which should lead to more widespread use. Both Surface Optics and Cubert have recently developed these light field cameras for the visible, near infrared and SWIR spectral ranges.
Snapshot Mosaic Cameras for Targeted Applications: Easier Data Processing and Cost-Effective in High Volume
Mosaic filter-based cameras provide another approach to achieving snapshot spectral imaging although being of multi-spectral rather than hyperspectral type.
The team at nanotechnology institute Imec continue to expand their wafer-level CMOS processes to integrate thin-film spectral filters directly onto image sensors. Cameras operating in the visible and near infrared wavelength range have been available for a few years with mosaic patterns of 3X3, 4X4 or 5X5 filters. More recently imec has also launched SWIR mosaic cameras (Figure 7). These designs also enable video-rate spectral imaging with real-time classification. The spatial resolution of the multispectral image will be less than that of the native image sensor (divided by the number of spectral bands) but some spatial resolution can be recovered using super-resolution image processing techniques. The manufacturing process used by Imec is easily scalable so there is great promise for this mosaic filter approach to lead to high-volume, low-cost sensors.
What Comes Next?
Extending into the SWIR
Well established manufacturers of push-broom hyperspectral cameras usually have a portfolio of systems covering the visible, near-infrared and shortwave infrared ranges. For the newer spectral imaging technologies most of the first embodiments are initially realised in the visible and near infrared spectral ranges using silicon sensors. Real incentives exist to expand the imaging wavelength range into the short-wave infrared, but InGaAs sensors have historically been expensive with fewer and larger pixels than the equivalent silicon sensors. Recent innovations in SWIR sensor technology will potentially initiate some changes. Manufacturers such as Sony, Emberion and Imec, have developed a new generation of SWIR sensors.
Sony’s SenSWIR technology, comprises of photodiodes formed on an indium gallium arsenide semiconductor layer connected via Cu-Cu connections with a silicon layer which forms the readout circuit. This yields a SWIR image sensor that is sensitive over a broad range of wavelengths covering the visible to SWIR range (from 400nm to 1.7μm). Emberion has also developed new sensors sensitive over a similar VIS to SWIR wavelength range. Their design combines nanocrystalline optical absorbers with graphene-based transistors fabricated directly on standard CMOS wafers. Similarly, Imec recently presented a new image sensor that uses a thin film of PbS quantum dots to capture light in the NIR and SWIR. These novel sensors have the potential to be disruptive in terms of cost of SWIR spectral cameras.
Providing Actionable Data
For the past decade hyperspectral imaging has been an area of extensive research and development. Application of hyperspectral imaging has involved high costs and complex, time-consuming processing to produce meaningful data. This has somewhat limited its use to R&D environments in academic research institutes. Bridging the gap from being an interesting research tool to providing actionable data for routine analysis requires implementation of image processing algorithms that can automate tasks such as detection, classification, and separation of materials. Fortunately, great strides have been made in this area with the development of machine learning and AI-powered hyperspectral imaging analytics. Sophisticated analytical software packages from the likes of perClass BV and Prediktera AS (Figure 8) can now be used by a range of imaging cameras to implement tasks such as object recognition and classification.
The combination of new camera hardware and smart image processing is significantly increasing ease of use and general accessibility to a wider community. Some of the new technical approaches also have the potential to reduce the camera size, form factor and cost which should enable this once-complex technology to be further adopted into everyday life. So, it seems inevitable that spectral imaging will become more a part of mainstream imaging and will play an increasing role in important topical issues of the day including recycling plastics, precision farming, food analysis and environmental monitoring.
Further Reading
An in-depth review of the different technologies used in multi- and hyperspectral imaging systems is provided in this Pro-Lite Technical Note
The physics of “light field” hyperspectral imaging are explained in this Pro-Lite Technical Note
The Pro-Lite range of multispectral and hyperspectral imagers is presented here.
