Keywords: imaging spectroscopy, multispectral, hyperspectral, applications, sensors, algorithms, medical imaging, remote sensing, crop monitoring, machine vision, infrared, spectral unmixing, classification, phenomenology.
An abbreviated version of this paper was published in Ref. [1].
| 1 ........... | THE TECHNOLOGY |
| 2 ........... | APPLICATIONS |
| 2.1 ...... | THERMAL INFRARED APPLICATIONS (TIRIS) |
| 2.2 ...... | THE INTELLIGENT MISSILE SEEKER (IMS) |
| 2.3 ...... | OTHER APPLICATIONS |
| 2.3.1 .. | Crop Health Monitoring |
| 2.3.2 .. | Advanced Manufacturing Technology |
| 2.3.3 .. | Medical Applications |
| 2.3.4 .. | Small Target Detection & Search and Rescue Operations |
| 3 ........... | CLASSIFICATION TECHNIQUES |
| 3.1 ...... | CLASS SEPARABILITY ANALYSIS |
| 3.1.1 ... | Hyperspectral Data Compression |
| 3.1.2 ... | Hyperspectral Image Filtering |
| 3.2 ...... | NEURAL NETWORK BASED SUPERVISED CLASSIFICATION |
| 3.3 ...... | ALGORITHM BENCHMARKING |
| 4 ........... | SUMMARY |
| 4.1 ..... | WWW Resources |
| 5 ........... | REFERENCES |
To illustrate the strength of this technique, the combined spectral / spatial analysis allows the detection of optically unresolved objects (subpixel-size objects) in an image. Application of this technique range from to Earth remote sensing to early cancer detection. It is obvious why the combination of imaging and spectroscopy is very attractive. In general, one can improve image understanding by combining the best of two worlds: spatial and spectral analyses. The combined analysis may be considered as data fusion. Conventional color imagery touches upon the idea, except that color imagery is based on very broad bands and the colors are often achieved at the expense of spatial resolution (i.e., by use of RGB color striped CCDs).
Imaging spectrometers typically use a 2-D matrix array (e.g., a CCD), and produced a 3-D data cube (2 spatial dimensions and a third spectral axis). These data cubes are built in a progressive manner either by (i) sequentially recording one full spatial image after another, each at a different wavelength, or (ii) by sequentially recording one narrow image (1 pixel wide, multiple pixels long) swath after another with the corresponding spectral signature for each pixel in the swath. Some common techniques used in airborne or spaceborne applications are depicted in Fig. 3.
Figure 3. Some Remote Sensing Implementations of Multi and Hyper-spectral Imagery [Multi-spectral systems a (using a few single detectors/filter combinations) and b (using a few linear array / filter combinations) are based on whishbroom and pushbroom scanning techniques, respectively; hyper-spectral systems c (using a few linear arrays in a dispersive system) and d (using an area array in a dispersive system) are based on similar scanning techniques].
Before discussing specific examples it is worth noting that multispectral techniques utilizing a small number (less than 10) of spectral bands have been available since the deployment of LandSat in the 1960s. A question often asked is whether hyperspectral systems that utilize several tens or hundreds of bands are indeed better than multi-spectral systems. The answer is that often they are not. If so, why all the interest in hyperspectral techniques? The reason is that multispectral instruments and the bands they use are often tailored to a specific application. These bands may be therefore less than optimal or even completely unsuitable for other applications. On the other hand, hyperspectral systems have the advantage of providing data at high spectral resolution over a large number of bands, and may hence be used for a variety of applications. However, once the optimal bands have been determined for a specific application, a multi-spectral system can be deployed as it provides an overall better solution. Multispectral systems are less expensive, produce smaller datasets, and have a greater signal to noise ratio (S/N). The examples that follow should be considered in the light of this idea.
Figure 4. Assignment of Molecular Spectra in the MWIR/LWIR.
The sensor under development is the Thermal InfraRed Imaging Spectrometer (TIRIS), a 7.5 to 14.0 µm imaging spectrometer designed to demonstrate operations using uncooled optics for Dual Use applications (including military target detection and civilian applications). The environmental application is particularly attractive since with an imaging spectrometer, a toxic gas plume that is monitored can be simultaneously characterized spatially and spectrally, and the measurements can be registered to a map location via geographical information systems (GIS).
TIRIS-I is based on an Aerojet PATHS 64x20 Si:As focal plane array (FPA), indium-bump multiplexed to a readout IC. The required FPA operating temperature of 10K is achieved by using a closed cycle helium cryo-cooler, selected for logistical reasons since TIRIS will be flown from various locations where liquid helium is not readily available. A prototype TIRIS-I sensor is shown in Fig. 5.
The TIRIS's field of view is divided into 20 spatial pixels while spectral signature data are collected in 64 bands at about 0.1 µm resolution. When airborne, full spatial/spectral image data cubes are generated in the pushbroom mode (Fig. 3d). Spectral dispersion is obtained by a combination of two means. First, a custom linearly variable filter (LVF) was designed and manufactured to match the FPA dimensions and pixel pitch. The LVF is placed about 0.2 mm from the FPA and it is cooled to the FPA temperature. A custom diffraction grating is also installed in the optical train with dispersion matched to the LVF spectral spread. The combination of grating/ LVF produces a high optical efficiency. The grating, as well as all other fore-optics, function at room temperature. The nearfield thermal emission from these components is suppressed by the LVF.
Custom electronics were built to operate the FPA and interface it to an image acquisition system. All the digital logic required to generate the necessary clocks and biases are loaded from a PROM into an FPGA upon system power up. Data are read in 20 parallel channels and every three channels are multiplexed to a single A/D converter. Seven ADCs are used to digitize the data at 12-bits. Each data channel is then sequentially given access to a bus that is connected to a digital frame grabber installed in a desktop PC. A gray code is used to sequentially increment the rows from 1 to 64. Each row is read twice, once after incrementing the gray code, and a second time after a reset signal is applied to the row. The scheme allows using software implemented correlated double sampling or subtractive double sampling (CDS/SDS) to reduce the noise generated during the FPA readout.
The TIRIS prototype is currently undergoing spectral and radiometric characterization. Based on lessons learned, a second generation airborne TIRIS-II is being developed using a Hughes HYWAYS 64x20 Si:As FPA, Fig. 6. Among the key features of TIRIS-II, in addition to a rugged construction, is a 10K filter wheel with 8 positions containing a set of filters (narrow bandpass, ND) and other devices (blanks, pinholes) that aid in the spectro-radiometric calibration. Another feature is inclusion of two micrometer feedthrough which facilitate fine alignment of the LVF over the FPA. This fine alignment is needed for the proper positioning after the thermal contraction of various components during cool down to 10K. Finally, the dewar includes a 10K and 80K radiation shields. Electrivcally, all leads that carry signals to the FPA use micro-coaxial cables with the sheath grounded. This feature significantly reduces noise and cross talk between digital clocks and analog signals.
Two models are used in the algorithmic approach to TIRIS data analysis. The first is a phenomenology model (forward solution) that simulates and predicts the sensor response for a better understanding of its operational parameters and limitations. It is a radiative transfer model that accounts for the terrain infrared characteristics, sky radiation, and the toxic plume parameters, as depicted in Fig. 7. A plume may be observed in absorption or emission depending on the temperature contrast with the background. Two spectral libraries, one of organic compounds [3] and one of background materials [4], are used to model and predict the sensor response under a prescribed scenario. Other libraries can be incorporated into the code. The model, developed in IDL (and can run on all platforms that support IDL), has a convenient GUI allowing the user to experiment with various parameters, Fig. 8.
Figure 7. Radiative Exchange Model for TIRIS Use in Detection of Gas Plumes; 
, 
, 
are the spectral absorptivity, emissivity, and reflectivity, and T is temperature.
With this model, the user can predict the sensor output under various operational conditions such as a specific mixture of plume component (or end-members that are selected from a library shown on the top-left of Fig. 8), environmental parameters such as ambient temperature, relative humidity, plume temperature, etc., as depicted on the right side of the figure. The top right corner of the screen allows selecting a specific sensor model that includes the spectroradiometric calibration of the sensor. Finally, corrections for atmospheric transmission are calculated externally using the MODTRAN [5] code. Once all such parameters have been specified, the model calculates and plots the sensor anticipated output in various user-selected formats on the bottom half of the screen.
Figure 8. User Interface, and Phenomenology Model for the Forward Solution The model can be used as a sensor design tool to establish performance requirements. (Sponsored by USAF Armstrong Labs).
The second model (inverse solution) uses the sensor output to determine the composition of toxic gas plume that creates the observed signature. The inverse solution has two steps. First, the organic compounds library is scanned and reduced to a set of M potential end-members that might contribute to the signature, based on detection probability and false alarm rate considerations. In the second step, an MxM matrix is constructed in which the M compounds selected in the first step are incorporated into M linear equations, in M spectral bands (M is typically smaller than the number of spectral data bands). The system provides an exact solution to the problem rather than the traditional least squares approximation. The key to this approach is the proper selection of the M band.
This process essentially constitutes a linear subpixel unmixing model and has applications to a wide variety of situations including (some are discussed in the sequel) medical, crop health monitoring, mineral prospecting, search and rescue operations, etc.
Since targets and decoys can be hot objects, the IMS imaging spectrometer concept evaluation sensor was built to collect radiation in the VNIR and MWIR. A schematic of the optical layout of the sensor is shown in Fig. 9. The sensor has a common 6" Cassegrain telescope with a dichroic beam splitter. The VNIR signals are reflected into a 256x256 CCD (Manufactured by Dalsa) based imaging spectrometer operating from 500 to 1,000 nm, while the IR radiation is passed into a 160x120 InSb array (Cincinnati Electronics) operating between 2.5 to 5.0 µm. Both spectrometers have an adjustable width entrance slit, that determines the spectral resolution of the sensor (which is not necessarily determined by the number of pixels in the FPA along the dispersion direction).
Figure 9: Optical Layout of VNIR and MWIR Imaging Spectrometers of the IMS Sensor, and a Photograph of the Assembly (Sponsored by USAF Wright Labs).
To demonstrate the format of data recording, we show in Fig. 10 "frame" of data captured during calibration process. The data represent spectral and spatial information along the x and y image coordinates, respectively (the black region in the figure). The field of view is a narrow swath that includes a point source Hg(Xe) lamp. The y-position determines the lamp elevation above or below the horizontal plane, while the x-position indicates the spectral signature of that source. To create a complete 3-D data cube, the field of view has to be scanned. The surface plot in Fig. 10 represents the relative intensity of the source at each wavelength and the spatial extent of the source.
Figure 10. Spatial / Spectral Calibration of IMS VNIR Sensor Using Hg(Xe) Source.
A MWIR signature of a passenger jet upon takeoff, looking directly into the back of the plane from about 1 Km is shown in Fig. 11. The signature closely matches the theoretical spectral distribution of a 1,100K source. The absorption of radiation in 2.7 and 4.3 µm, due to H2O and CO2 in the intervening atmosphere can also be seen. Because of the presence of these two gases in the engine plume, where the temperature is much higher than that of the surrounding atmosphere, the so called "red and blue spikes" (due to Doppler line broadening) are also seen in the graph on the right.
Figure 11. Signatures of an Airliner's Exhaust Plume Obtained via the MWIR Part of IMS.
The greatest challenge for the IMS application is the signal processing [8]. Since the missile guidance and control system must be updated at about 50 Hz, the entire process of data acquisition and analysis must be repeatedly performed in less than 2 msec. One of the techniques developed for this purpose was termed "pseudo-signature." Basically, instead of analyzing large data sets corresponding to the entire measured signature, only a subset or a pseudo-spectrum is analyzed. This pseudo spectrum is a signature in a small number of bands (e.g., between 4 to 16 -- depending on the complexity of the situation) that are created (i) by removing bands in which atmospheric transmission is poor (attempts to correct the data in those bands are impractical), and then (ii) by binning bands into a wide group (to enhance signal to noise ratio). The latter step does not require the use of the same number of binned bands per group, nor do the binned bands have to be contiguous. Such a reduced size dataset can now be analyzed in real-time.
The pseudo signatures are transformed into a domain in which maximum separation exists between the signatures of target and decoy. The linear transformation is based upon the simultaneous diagonalization of the covariance matrices of the signatures corresponding to the targets and decoys. Target/decoy discrimination is accomplished in the transformed domain using for instance matched filters (see section 3).
The IMS project demonstrated the use of a multispectral seeker. It has shown that with hyperspectral system, the bands can be dynamically configured for specific situation for optimized performance (for instance, day /night bands selection may be different for instance when the sun glare off the background or the target are not present).
An added side benefit of a multi- or hyper-spectral seeker is for target ranging. The spectral signature in certain bands is strongly modified by the presence of CO2 in the atmosphere (Carbon monoxide is selected because the amount of CO2 in the atmosphere is pretty much a constant). The observed signature depends on the range (L) (attenuation caused by the path-integrated amount of intervening CO2 ), the wavelength (
), the extincion coefficient (
), and the target temperature (T). The unknowns in Eq. (1) are the target range, temperature, and factor that is a product of the emissivity, the target area and a view factor (assuming that the emissivity is not a strong function of wavelength, this parameter can be also considered constant). In this equation, BB is a Plank blackbody function. Using the measured values 
, in k bands, a constraint least squares solution can be obtained to the target range L.
 | (1) |
In the present system, four cameras acquire an image and the data are simultaneously transmitted to a computer via a frame grabber. Using the PCI bus (32 bits wide bus running at 33 MHz; with specifications also allowing for 64 bit wide bus running at 66 MHz) and a DMA bus mastering mode, it is possible now to transmit data to the host memory at rates as high as 132 MBytes/sec in a burst mode. For four 12-bits cameras, 512x512 pixels each, running at 10 MHz, the required transfer bandwidth is about 60 MBytes/sec, well within the PCI range. PCI frame grabbers that perform this function are less expensive than their predecessors that required large amount of data storage on board, and provided built-in expensive DSP functions. OKSI in conjunction with DIPIX (a frame grabber manufacturer, Ontario, Canada) and Dalsa (CCD camera manufacturer, Ontario, Canada) have tested a system that allows connecting two digital cameras to a single frame grabber. The two cameras are then pixel synchronized, not only in the read out speed but also in pixel registration on each camera.
An effective early detection technique requires in this case accurate subpixel unmixing analysis. A linear unmixing method was described earlier in connection with plumes containing gas mixtures. Such a linear technique is the first step in any analysis. Because of the specific conditions for radiative exchange, it is believed that non-linear effects may play a role in the measured signatures. Non-linear effects are caused by multiple scattering of photons off objects that have different spectral characteristics, as depicted in Fig. 12. Nonlinear analysis techniques are under development.
There are many manufacturing processes that can take advantage of multi- or hyper-spectral data. These include processes such as inspection of color and paint quality, detection of rust, or the inspection of defects in thin film coatings. Another potential application is the on-line, real-time inspection of weld quality. In this case the time evolution of spectroradiometric data can be translated to temperature maps and heat transfer processes in the welded parts. Heat flow maps can be correlated with good or cold welds. Real time detection of such defects can be used to correct the process before too many defective parts leave the production line.
Another interesting study of utilizing hyperspectral techniques conducted by UCLA/JPL is related to functional mapping of the brain. The idea being to identify areas of the brain that perform specific functions without doing histopathaology on every slice (Fig. 14).
[Major contributions to this work were made by Dr. Jacob Barhen of ORNL]
Supervised classification of multi- and hyper-spectral imagery is a robust technique that can be applied in almost all the fields that were discussed above. Customarily objects' signatures are presented in the feature-space which in our case is an N-dimensional space where N is the number of spectral bands, Fig. 15. Most classification algorithms operate in the feature space and are primarily based on statistical techniques including clustering algorithms. In contrast, neural network (NN) based classification may have the advantages that (i) they are distribution-free (i.e., do not make any assumption regarding the statistical distribution function of the data), (ii) they are non-linear, and (iii) they do not require a phenomenology based model to describe the data distribution, but can learn by example (as such they can easily identify classes with disjoint distributions). In spite of these potential advantages, NN are not frequently used in hyperspectral image classification. The reasons are (a) the inordinately long training time associated with backpropagation type algorithms training when using large data sets, and (b) the dependence of the final results on the starting parameters assigned to the NN weights. The latter problem is related to the fact that the NN training algorithms often get trapped in local minima as opposed to establishing global minimization of the error function. OKSI has addressed some of this issues with excellent results.
In general PCA does not guarantee to produce optimal separability among all classes in the image. The largest eigenvector in PCA produces and axis along which the entire image has maximum variance, but there is no guarantee that the variance is maximized along that eigen vector for all classes. Hence, instead of performing PCA on an entire image, doing so with training data of specific classes may produce better results.
Figure 16. Signatures in the Spectral (Physical) Domain (A); in the Feature Domain (B); Transformed for Maximum Separability (C). (Work Sponsored by U.S. DOC/NIST and by NASA GSFC)
One technique to quantify the class separability is based on the Spectral Angle Mapper (SAM) that measures the "angle" between the two vectors that point to the centers of the two clusters (Fig. 17). Another, is based on the Fisher Linear Discriminant, which is a measure of the "distance between classes normalized by the spread within the classes (also illustrated in Fig. 17).
A novel world-class NN training paradigm based on alternating direction singular value decomposition (AD-SVD) [15] was developed and demonstrated, reducing training time for large data sets and large number of input nodes, to a fraction of a second. The network topology is illustrated in Fig. 18. The AD-SVD allow determining the synaptic weights of the NN typically in a single pass. A similar backpropagation network required 10 to 15 minutes training. An example of the application of the NN classification to a hyperspectral image of a fresco is shown in Fig. 19.
Figure 18. Topology of the AD-SVD Neural Network.
Figure 19. Neural Networked Classified Hyperspectral Image of A Fresco (33-bands) (Data, Courtesy Dr. G. Bearman, JPL).
A recurrent NN with enhanced learning algorithm has been adapted based on prior work at JPL, to address hyperspectral analysis. This fully connected network, configured as a system of subnetworks was shown to perform very well on the benchmarking datasets, opening the way to use hyperspectral data in dynamical, time-dependent applications such as monitoring seasonal change, deforestation, soil erosion, etc.
Figure 20. Synthetic Data Cube for use in Algorithms Benchmarking.
Figure 21. Mean Signatures for 16 Classes in Spectral (left) and in Transform Domain (right).
Performance benchmarking was established for statistical classification methods including: maximum likelihood, minimum distance, parallelepiped, and spectral angle mapper, as well as common feed-forward error backpropagation architecture, recurrent networks (with enhanced learning techniques), and the new AD-SVD network.
Work was conducted addressing subpixel spectral unmixing, in which the dominant component in an unknown mixture is identified based on classification algorithms.
| [1] | Gat, N., Subramanian, S., Barhen, J. and Toomarian, N. "Spectral Imaging Applications: Remote Sensing, Environmental Monitoring, Medicine, Military operations, factory Automation and Manufacturing." Presented at 25th AIPR Workshop on Emerging Applications of Computer Vision, Oct. 16-18, 1996, SPIE Vol. 2962. Return to Text |
| [2] | Elachi, C. Introduction to the Physics and Techniques of Remote Sensing, Wiley Interscience, 1987. Return to Text |
| [3] | EPA's most Hazardous Air Pollutants (HAPs) spectral library. Return to Text |
| [4] | NPIC spectral library of background materials. Other libraries can be incorporated to the model, and the model can be adapted to other sensors. Return to Text |
| [5] | Bedrk, A., Bernstein, L.S. and Robertson, D.C. MODTRAN: A Moderate Resolution Model for LOWTRAN 7. AFGL Report GL-TR-89-0122, 1989. Return to Text |
| [6] | Gat, N., Barhen, J., Gulati, S., and Steiner, T.D. "The Intelligent Missile Seeker (IMS): Spectral, Spatial, and Temporal Domain-Based Target Identification and Discrimination: Part 1 - Sensor and Discrimination Algorithms," and "Part 2 High Performance Intelligent Control System," Proc. 3rd Automatic Target Recognizor Systems & Technology Conf. GACIAC PR-93-01, Vol. 1, pp. 117-128, 129-139, 1993. Return to Text |
| [7] | Gat, N., Barhen, J., Gulati, S., and Steiner, T.D. "Hyperspectral Imaging for Target/Decoy Discrimination: Sensor and Algorithms," Proc. Mtg. IRIS Specialty Group on Passive Sensors, IRIA Publication, 1994. Return to Text |
| [8] | Gat, N., Barhen, J., Gulati, S., and Steiner, T.D. "Hyperspectral Air-to-Air Seeker," SPIE Vol. 2231, Algorithms for Multispectral and Hyperspectral Imagery, Pp. 127-135, 1994. Return to Text |
| [9] | Alfano, A.A., et-al "Optical Spectroscopic Diagnosis of Cancer and Normal Breast Tissues," J. Opt. Soc. Am. B, Vol. 6, No. 5, Pp. 1015-1023, May 1989. Return to Text |
| [10] | Frisoli, J.K. et-al "Medical Applications of Laser Induced Fluorescence: Pharmacokinetics of Photosensitisers," SPIE 1882, Optical Methods for Tumor Treatment and Detection: Mechanisms and Techniques in Photodynamic Therapy, II, 1993. Return to Text |
| [11] | Bigio, I.J., et-al, "Optical Diagnostics Based on Elastic Scattering: An Update of Clinical Demonstration with the Optical Biopsy System," SPIE 2324, Optical Biopsy and Fluorescence Spectroscopy and Imaging, Pp. 46-54, 1994. Return to Text |
| [12] | Cube Constructed from Images Courtesy of Dr. Greg Bearman, JPL, and Dr. Art Toga, UCLA. Return to Text |
| [13] | Therrien, C. Discrete Random Signal and Statistical Signal Processing, Prentice Hall, New Jersey, 1992. Return to Text |
| [14] | Richards, J. A. Remote Sensing and Digital Image Analysis, Springer Verlag, 1995. Return to Text |
| [15] | A Milestone Paper under preparation (contact the authors for details) Return to Text |
Return to the Hyperspectrum Newsletter Index.
Return to Opto-Knowledge Homepage.
Browse more TechExpo stuff.
Legaleese & Copyright © Statement: 1997: You MAY copy this newsletter to your computer or print it for your own entertainment or amusement, or even for business and technical purposes (as long as the contents are preserved and the proper reference is included). You also MAY e-mail copies to your colleagues, family friends, and family, but you may NOT mirror or post the newsletter on another WWW site. You can, however, include a link to the newsletter on this site in your own documents.