IEEE SENSORS JOURNAL, VOL. 1, NO.

1, JUNE 2001

31

Target Acquisition Performance Modeling of Infrared Imaging Systems: Past, Present, and Future
James A. Ratches, Richard H. Vollmerhausen, and Ronald G. Driggers
AbstractThis paper provides a 40-year review of the infrared imaging system modeling activities at the U.S. Army Night Vision and Electronic Sensors Directorate (NVESD). The result of these modeling activities is a system model that describes the target acquisition performance of a human observer and an infrared imager. The model has been adopted by the military infrared imaging community as an assessment of how well an ensemble of observers perform the tasks of target detection, recognition, and identification. The model is used in infrared imager design and assessment, where military users understand how the metrics predicted by the model relates to system performance on the battlefield. This review begins with early work in the late 1950s and proceeds to present day modeling successes. Finally, the infrared imaging system modeling activities for the future are discussed. Index TermsInfrared imaging, modeling.

I. INTRODUCTION HIS paper is a review of a scientific effort that has been ongoing for more than 40 years at a U.S. Army Research & Development laboratory. The effort has resulted in a set of empirical analytic models that predict the target acquisition performance of a system composed of a human observer and an electro-optical imaging sensor. The emphasis of this paper is on the subset of electro-optics known as thermal imaging. The resultant models have been adopted by the military infrared imaging community as quantitative assessments of how well an ensemble of observers, using infrared imagers, will perform the tasks of detecting, recognizing and identifying tactical targets on a battlefield. Although this development has been extensively documented in the classified and defense unique literature, the complete technical story does not appear in the open literature. The scientific investigation and engineering processes have produced models which are routinely used by Department of Defense (DoD) agencies and their contractors to design and optimize military thermal imagers, frequently referred to as forward looking infrared (FLIRs). Most importantly, the military users of these models understand how the metrics predicted by the models relate to how the systems will perform on a battlefield. The effort was started approximately 40 years ago at the then U.S. Army Night Vision Laboratory (NVL). Today NVL is known as the U.S. Army Communications & Electronics Command (CECOM) Night Vision & Electronic Sensors
Manuscript received September 8, 2000; revised February 20, 2001. The associate editor coordinating the review of this paper and approving it for publication was Prof. Jan Soderkvist. The authors are with the U.S. Army Communications-Electronics Command, Research, Development and Engineering Center, Night Vision and Electronic Sensors Directorate, Ft. Belvoir, VA 22060-5806 USA (e-mail: [email protected]). Publisher Item Identifier S 1530-437X(01)04344-5.

Directorate (NVESD) but it is still referred to as NVL in the community and that acronym will be used in this paper. The models were a response to the Armys need to develop and field a whole family of night vision systems in order to fight under nighttime conditions. The modeling concepts that were investigated were driven by the outstanding FLIR system considerations at the various times during this 40-year span. Also, the thermal imaging sensor models are really only one model that has undergone a series of modifications or improvements that address more sophisticated aspects of thermal imaging systems. The initial model addressed simple detector arrays which scanned across the scene, but grew in complexity as detector arrays became focal plane arrays, resolution became nearly isotropic, system level noise grew more complex, sample data effects became more important, observer perception characteristics grew in importance, and other complexities that will be described in the paper. A similar story can be told for models of other military electro-optical systems, such as image intensifiers, albeit with their unique technical issues, but that is left for another paper. II. JOHNSON MODEL FUNDAMENTALS The seminal hypothesis for the NVL performance models was proposed by Johnson of NVL in 1958 [1]. Johnsons original concept was based upon work originally done by Schade [2], [29][31] with television (the first man made electro-optical system). Johnson proposed that the ability of an observer to acquire military targets in scenes (detect, determine orientation, recognize and identify) when viewing through an electro-optical device is dependent on how well he can resolve bar patterns of varying frequencies through the device at the same contrast as the scene target-to-background contrast. Since thermal imagers did not exist when Johnson conducted his initial experiments, contrast was defined as target brightness ( ) minus background brightness ( ) divided by background brightness ( ) for image intensifiers and television, CONTRAST (1)

When FLIRs became available, the concept was extended and contrast was replaced with target to background temperature dif) which represented the signal input to thermal imference ( agers SIGNAL (2)

and are the temperatures of the local (to the target) where background and target, respectively.

U.S. Government work not protected by U.S. copyright

32

IEEE SENSORS JOURNAL, VOL. 1, NO. 1, JUNE 2001

A series of experiments were conducted with an ensemble of observers viewing through image intensifiers to determine how well they could resolve bar patterns and also perform the discrimination tasks of detection, target orientation, recognition and identification of military targets. Detection is defined as discriminating the presence of an object of potential military interest from the background, orientation is the determination of the target aspect, recognition is determining the class of the target, e.g., truck, personnel carrier, tank, etc., and identification is the determination of the member of the class, e.g., M60, M48, Stalin tanks. Johnson hypothesized that the ability of the observer-sensor ensemble to discriminate the targets was a function on how well a critical dimension on the target could be resolved. The critical dimension for the various targets used was defined based on intuition but was usually chosen to be the minimum dimension. For combat vehicles this became the height of the vehicle. For a human target it was the width of the man. The discrimination level (e.g., detection, orientation, recognition, or identification) was then related to how many line pairs could be resolved across the defined critical dimension on the target for the limiting resolution measured with the device-observer combination. The number of line pairs resolvable ( ) across a target critical dimension was calculated by multiplying the highest bar pattern frequency ( ) that could be resolved at that contrast or by the observer-device ensemble times the target angular dimension ( ). The angle in milliradians was obtained by dividing the target critical dimension in meters by the target range in kilometers. The calculation of resolvable cycles is shown in (3). Table I shows the results of Johnsons experiments:

TABLE I JOHNSONS DATA RELATING RESOLUTION IN LINES RESOLVED ACROSS THE TARGET CRITICAL DIMENSION TO DISCRIMINATING TARGETS TO VARIOUS LEVELS

III. MRT MODEL The classical MRT model was first published by Ratches [4][6]. A recent historical perspective on the early model development appears in reference [7]. MRT is defined as the minimum temperature difference above 300K required by an observer viewing through the device to resolve a vertical four bar pattern of 7 : 1 aspect ratio. The MRT is a monotonically in cycles increasing function of spatial bar target frequency per milliradian. The expression for MRT is derived from the threshold signal-to-noise ratio required by an observer to resolve the pattern and is given by MRT NET

(4) cycles cycles/mrad mrad (3) is the threshold signal to noise ratio required to recwhere is also known as and ognize one bar [unitless]. found to be equal to 2.25 based on empirical results. NET is the noise equivalent temperature difference [Kelvin] and is calculated NET (5)

The original Johnson experiments were run with relatively high contrast targets and bar patterns. The performance measured corresponded to resolution-limited performance. The extrapolation to noise limited conditions or performance under any contrast or temperature difference came with the definition minimum resolvable temperature (MRT) difference. In 1969, Lloyd and Sendall [3] defined limiting resolution of bar patterns as the MRT and the measurement was standardized at NVL. MRT is the measurement of an observers threshold in bar temperature difference above ambient for recognizing a four bar pattern as a function of the bar frequency. That is, at a given bar frequency determined by the bar spacing (equal to 1/[2 bar width]), the temperature difference of the bars above ambient was reduced until the observers could no longer distinguish four bars; the temperature difference was then increased until the bars are just visible. The methodology for getting from MRT to target acquisition performance is the subject of Section IV. Much of the following modeling development has been published in the classified literature for infrared, the infrared information symposium (IRIS). The original articles were largely classified due to the validation data sets which indicated performance of new developmental infrared imagers. Most of the referenced articles are today declassified.

where -number of the optics [unitless]; noise bandwidth [Hz]; detector area [cm ]; average optical transmission [unitless]; number of detectors scanned and summed in series; specific detectivity [cm- Hz/Watt or Jones]; partial of radiance (Plancks equation) with respect to temperature [Watts/cm -sr-um-K]. The noise bandwidth is defined as (6) where

RATCHES et al.: TARGET ACQUISITION PERFORMANCE MODELING OF INFRARED IMAGING SYSTEMS

33

normalized noise power spectrum [unitless]; electronic modulation transfer function (MTF) [unitless]; measurement device MTF [unitless]. At the time of the Ratches model, first generation FLIR (serial scanned) was the primary sensor for modeling and testing, so the measurement device was well-defined with a particular MTF ; (7) and was set to that was defined as and was the dwell time of the detector

so that

MRT

(13) Note that the performance is driven by the eye integration time. A low frame rate is accompanied by a small noise bandwidth, a high dwell time, and the eye integrates only a small number of frames. A high frame rate is accompanied by a large noise bandwidth, a small dwell time, and the eye integrates a large number of frames. These conditions are equivalent in the human detection of bar targets. IV. FIRST FLIRs AND MODELS The MRT related system design parameters, such as , optics diameter, detector size, etc., to subjective observer recognition of four bar patterns. The Johnson model could then be applied to MRT for a given target signature, atmospheric condition and discrimination level to predict field performance. This became critically important during the late 1960s and 1970s when the U.S. Army was ready to design, engineer, produce, and field a whole family of thermal night sights across all weapons systems. A performance model was needed that could relate FLIR design parameters to field performance by the soldier using the device. In order to optimize the design and choose the best contractor candidate, a method was needed to relate, quantitatively, system parameters such as detector sensitivity, detector MTF, optical F/#, optical MTF, electronics MTF, display MTF, etc. to how well a soldier could acquire targets. At this time, a team of modelers at NVL led by Lawson was attempting to build on the work of Johnson to come up with a performance model for FLIR systems. The objective was to generate an engineering model that predicted probability of an ensemble of observers to detect, classify (tracked versus wheeled vehicle), recognize and identify tactical targets as a function of range, environment and system parameters. With the derivation of the MRT expression [equations (4) and (13)], it was straightforward to apply the Johnson concept and relate system parameters to performance in the field. The MRT provided the connection between Johnsons concept of resolvable bars across the target critical dimension and a system level measurement and it was a measurement that could be routinely carried out in the laboratory. The MRT measurement also included the subjective performance of the observer. (A minimum detectable temperature (MDT) difference was also defined which would be the connection between signal-to-noise ratio of a blob and the ability to detect hot spots without any higher level discrimination. A measurement was performed in which the threshold of an observer was measured for detecting the hot target as a function of target size [8].) The target acquisition performance model [9] that used the Johnson concept and MRT is shown in Fig. 1. An inherent target

(8) where horizontal field of view (FOV) [milliradians]; vertical FOV [milliradians]; overscan ratio (ratio of the vertical instantaneous field of view, IFOV, to the vertical sample spacing) [unitless]; frame rate [seconds ]; horizontal instantaneous field-of-view (IFOV) [milliradians] found by dividing the detector horizontal ; dimension in by the focal length in vertical instantaneous IFOV [milliradians]; number of detectors in parallel [unitless]; scan efficiency [unitless]. Returning to (4) scan velocity of the scan mirror [mrad/s]; integration time of the eye [approximately 0.2 s]; spatial integration of the eye over a bar; (9) The MTF terms in the eye integral are MTF of all the system components after the detector including the display; eye MTF; Fourier transform of the bar width. Since the bar is a rect function in space, then (10) is the bar width and is given by . where The MRT equation can be simplified with substitution of the NET, (5), into (4)

MRT

(11) A final reduction can occur since the scan velocity is the horizontal FOV times the frame rate [mrad/s] (12)

34

IEEE SENSORS JOURNAL, VOL. 1, NO. 1, JUNE 2001

signature in terms of a temperature difference ( ) of the target above the local background is attenuated by the atmospheric ) propagation to give an apparent temperature difference ( corresponds to a bar pattern frequency at the sensor. This which the sensor-observer system can resolve through the MRT curve. The number of resolvable bar pattern cycles from (3) which can be resolved across the target critical dimension (meters) at range (kilometers) is then given by (14) across the target critical This number of resolvable cycles dimension could then be related to probability of any level of discrimination through a set of empirically generated curves. These target transfer probability functions (TTPFs) were generated from an extensive set of field exercises given in reference [7]. The TTPFs represent the per cent of the ensemble of observers who could correctly perform the discrimination task, e.g., detection, classification, recognition, and identification.1 The number was a function of target signature ( and ), atmospheric propagation, and system parameters through the MRT. was calculated from a measured or predicted The target for the signature by calculating the absolute area weighted s and areas of each comwhole target from the absolute ponent of a target signature when broken up into sub-areas of constant temperature. The critical dimension of the target was usually the height, however, not always as is the case for a man target where the width is used. The choice of critical target dimension for new targets is usually left to a panel of experts with a great deal of experience in the performance of FLIRs. Tables of critical dimensions have been published in the literature on the NVL model. Atmospheric propagation is routinely calculated using some standard model, such as LOWTRAN [10] (or the newer version MODTRAN). The TTPF curves were generated from field performance data and one standard free-hand fit was used to match all the levels of discrimination. The one curve was translated horizontally over resolvable cycles and the posifor 50% probability. tion of the curve was specified by the then specified the entire curve. For example, the The for detection, aiming (a missile gunner could put a cross-hairs on the target with sufficient accuracy to fire the missile), recognition and identification were determined to be 1.0, 2.5, 4.0, and 8.0, respectively, during this time frame. Later as more field data was acquired, the values evolved. The entirely new model that had to be developed in order to implement this modeling approach described in Fig. 1 which related MRT to field performance was called the ACQUIRE (reference [9]) model. The ACQUIRE model related FLIR system design parameters through the MRT to observer-sensor performance in the field. These included detector noise, optical transfer functions, detector transfer functions, electronics

Fig. 1. MRT.

Model to predict performance based upon Johnson and using system

parameters, display characteristics and observer eye integration time.2 ACQUIRE could be used as a sensor design and contractor sensor selection tool. It was accepted by the community because it was validated with real field data, was well understood and was promulgated throughout the national and international infrared community. Most importantly, as the data from laboratory measured MRT and field measurement validation grew, the Army decision makers could rely on the fact that a given thermal imager set of design parameters resulted in a required level of target acquisition performance on the battlefield. Costly field test verification of military required performance was, thus, avoided and replaced with a relatively simple test that could be done on the production line. to a specific situation The selection of the appropriate was not always straight forward. Often times an expert was for a particular task and the target critrequired to select the ical dimension. In addition, all imagers at this time were linearly scanned in the horizontal direction and a vertical MRT was not defined due to the sampling effects. Hence, the model ignored resolution in the vertical direction. Notwithstanding, the MRT model was used as the basis for field performance predictions with useful fidelity. The MRT model was shown to give predictions that were, generally, representative of what was measured in the laboratory for linearly scanned thermal imagers. Typically, the predicted MRT curve crossed the measured system curve at some intermediate bar frequency and was optimistic at low frequency and pessimistic at high frequency. Field performance was shown to be within 20% in range for a given probability. In order to extend the MRT model to staring systems, an arbitrary cutoff of the MRT at one half the theoretical limiting frequency (one over the detector instantaneous field of view) was imposed. This was done in order to account for the fact that a staring imager could not resolve four bars beyond that frequency, although there was modulation in that region of frequencies. This shortcoming of the MRT model will be addressed in Sections VIVIII of this paper.
2There are other observer factors that influence target acquisition performance, such as training, motivation, reward, etc. These factors have never been incorporated into the model.

1The question of false alarm rate is frequently brought up with respect to this performance. However, these experiments were designed such that the false alarm rate was extremely low. The observers were instructed to respond only when they were very confident of their response.

RATCHES et al.: TARGET ACQUISITION PERFORMANCE MODELING OF INFRARED IMAGING SYSTEMS

35

V. MODEL IMPROVEMENTS FOR RESOLUTION AND NOISE

TABLE II TEMPORAL AND SPATIAL NOISE IN SECOND GENERATION THERMAL SYSTEMS

By the mid 1980s, a new generation of thermal imaging was being developed. Linearly scanned two-dimensional (2-D) arrays of detectors and staring arrays were becoming available. Material and growth improvements coupled with the increases in sensitivity to be realized from time-delay-and-integrate (TDI) and staring provided the opportunity for a quantum leap forward in performance for thermal imaging. With the advent of second generation scanning systems and staring sensors, shortcomings in the FLIR performance models became critical. System level noise became as important as the detector noise. Noise introduced by detector-to-detector nonuniformity, the scanning/framing processing, multiplexing, fixed pattern, and electronic processing had to be considered in performance models for second generation if the performance was to be accurately assessed. The 2-D scanned and staring arrays were sampled in two directions with electronic multiplexing and vertical resolution approached horizontal. The other major improvement in second generation systems over first generation was in digitization. The detector array became a focal plane array of IR sensitive detectors with read out circuits bonded to the detector array and which multiplexed the signal out of the dewar. This signal could now be digitized with A/D conversion and processed using state-of-the-art digital processing. The major military interest in digital processing was to implement aided/automatic target recognition (ATR) in the sensor package. Second generation scanned focal plane arrays that are going into the Armys Horizontal Technology Integration FLIR B-Kit were designed to facilitate ATR on weapons platforms. The B-Kit was a set of infrared system components that was intended to replace first generation infrared systems in a vehicle-independent manner. FLIR B-Kits have nearly isotropic resolution in both dimensions, no interlace, improved signal-tonoise through TDI, and can be sampled at greater than once per detector dwell time in order to provide processors the most computer friendly image possible in order to perform automated functions. The NVL modeling group was led at this time first by L. Obert and then by DAgostino and addressed the noise and increased resolution issues directly [11], [12] and implemented improvements in MRT modeling with the model FLIR90 and subsequently FLIR923[13]. DAgostino hypothesized that the total system noise could be reduced to eight components depending on whether the displayed noise had temporal variation or spatial horizontal or vertical variation in the plane of the display. The standard deviation of each noise component represented a real displayed and measurable noise. Table II shows the eight components and their description and potential source for that noise component. The development of digital processing permitted the measurement of each of these components in a system in a laboratory. The new noise concept was implemented mathematically in the FLIR92 MRT equation. The validity of the 3-D noise model for temporally coherent, spatially random noise was recently demonstrated [14].

Assuming statistical independence of the noise components, the total system noise in this three-dimensional (3-D) formulation for noise as a function of frequency is the square root of

(15) , and represent the eye/brain temporal and where , spatial integration associated with each noise component and the subscript indicates vertical or horizontal orientation of the four bars. The temporal and spatial integrators are approximately given by (16a) (16b) (16c) and are horizontal and vertical sampling rates (samand and are spatial integration ples/milliradian), limits (milliradian ), which are approximately the horizontal and vertical dimensions of the bar target, and , , and are the sample correlation factors. , , and are equal to 1 for staring systems and may be greater than 1 for scanning systems. is the frame rate and is the eye integration time. is the basic detector noise normally characterThe term is related to the actual system bandwidth ized by the NET. and not the artificial standard bandwidth used to measure NET. It becomes the NET when multiplied by the ratio of the equivalent noise bandwidth divided by the actual noise bandwidth. A second innovation that DAgostino introduced into the NVESD FLIR and target acquisition models was the use of resolution in both horizontal and vertical directions. Background experiments were performed at NVESD using simulated imagery which showed that more accurate performance predictions were made when resolution in both image directions was included. In order to preserve the existing well understood Johnson concept for imaging and to have as little as possible impact on the well established approach to predicting field performance from MRT, a 2-D MRT was defined which

36

IEEE SENSORS JOURNAL, VOL. 1, NO. 1, JUNE 2001

does not correspond to a physical measurement that could be performed on a FLIR. A fictitious MRT function was defined whose temperature difference value was defined as that value at a frequency equal to the square root of the product of the value horizontal and vertical MRT frequencies for the measured on the horizontal and vertical MRTs. The prediction of field performance was then identical as that shown in Fig. 1, however the critical dimension of the target now became the square root of the target area in order to have a consistent 2-D approach.3 Fig. 2 shows this 2-D model diagrammatically. Although, conceptually, one might envision this approach as using resolvable pixels on the target, the fundamental metrics are still linear one-dimensional (1-D) frequencies (square root of horizontal and vertical frequencies) and length (square root of area). It is important to note that at this time when 2-D resolution was being introduced into the model, more system component transfer options were introduced into the model also. Transfer functions for digital filters, CCD charge transfer, display sample and hold, and other electronic filters were added to the potential possible characteristics of model descriptions for candidate thermal imagers. The FLIR MRT model with 3-D noise and 2-D MRT has been released to the community under the name of FLIR92. The calculation of target signature, atmospheric propagation and field performance has been released under the name ACQUIRE [9]. FLIR92 also includes the calculation of MDT and ACQUIRE uses the MDT to compute hot spot detection ranges. ACQUIRE contains tables for the area, critical dimension for tactical targets and an analytic curve fit to the TTPFs. The TTPF values had to be changed somewhat in order to validate the model for second generation systems and to re-validate the 2-D model to the old database of performance with first generation are shown FLIRs. The original NVESD model values for in Table III compared to the new values used in ACQUIRE. Also, the aim discrimination level is dropped and a classificabrought closer tion level is introduced. The new values for agreement with the values used in image intensifier modeling. This was aesthetically pleasing since it brought the modeling of different EO technologies into closer agreement. Image intensifiers have isotropic resolution in all directions and the recogniequal to 3 for many years, tion criterion for them has been and now FLIR models had the same criterion as their resolution became isotropic in two dimensions. It is important to note that FLIR92 does not account for sample data effects any differently than the original model. An arbitrary asymptote is imposed on a staring system MRT at the Nyquist frequency. This is important in the future modeling activity to be discussed in Section VII of this paper. Two complementary developments to the modeling helped enable the significant improvements in the capability of the FLIR 92 and ACQUIRE models. These were the development of an Advanced Sensor Evaluation Facility [15] and the expan3The target area is the projected area of the target on the display. This means that the area can be different in different spectral regions due to the fact that different components of a target show up differently in the various spectral regions. The canvas of a 2.5 ton truck may be at the ambient temperature and have no temperature difference in the infrared. It would show up in the visible region.

Fig. 2. Two-dimensional approach to performance prediction.

TABLE III CHANGES IN JOHNSON CRITERIA FROM ORIGINAL NVESD MODEL ACQUIRE [18]

TO

sion of the applicability and use of perception testing [16], [17]. Laboratory testing of thermal imaging systems was upgraded in order to make use of digital processing. Signal trains in an imaging system could by digitized and processed in order to help characterize the system. Use of such equipment as frame grabbers enabled the isolation of the various noise components which permitted the validation of the noise modeling concepts. In addition, the ability to generate large amounts of simulated targets which could be used as input stimuli to automated perceptual testing enabled the generation of large amounts of target acquisition data. Many independent and dependent variables to observer-in-the-loop performance under controlled conditions could be studied that provided new in-depth understanding and new concepts for advanced models. The results from the perception testing had high statistical significance due to the large number of replications that could be performed under the controlled environments. VI. INCORPORATING EYE CONTRAST LIMITATIONS The FLIR92 model provides accurate performance predictions for first and second generation thermal imagers. However, this past decade has seen significant advances in the development of sensitive staring detector arrays, and these arrays are being incorporated into a wide variety of sensor systems. Staring arrays have characteristics which can lead to errors in

RATCHES et al.: TARGET ACQUISITION PERFORMANCE MODELING OF INFRARED IMAGING SYSTEMS

37

FLIR92 performance predictions. Due to the sensitivity of the new staring arrays, the contrast limitations of the eye can be important in establishing performance limitations, and these are not modeled in FLIR92. The addition of eye contrast limitations to the model is described below. Also, the performance loss due to sampling artifacts was typically not as severe as modeled by FLIR92. The change in the sampling theory is described in Section VII. A recent upgrade to the thermal model, called NVTherm, incorporates two changes in the eye model. The theory behind these changes is described in [19] and [20], and the two changes are briefly described here. The first change involves replacing the fixed signal to noise threshold at which a bar-pattern is assumed to be detected with a variable threshold which varies with display luminance and the spatial frequency presented to the eye. In the theory described by (13), the signal to noise ratio threshold (SNRT 2.25) at which a bar pattern is detected is assumed to be constant regardless of display luminance or spatial frequency. The experiments of Rosell and Wilson in the early 1970s, and experience with image intensifier sensors at NVL during the same period, demonstrated that SNRT is not fixed [21]. NVTherm incorporates a variable SNRT which is based on the measured contrast threshold function (CTF) of the eye and visual system. The CTF is one of the most common ways of characterizing human vision. The following procedure is used to measure CTF. The observer views a bar chart or sine-wave pattern. While holding average luminance to the eye constant, the contrast of the bar pattern is lowered until no longer visible to the observer. That is, the dark bars are lightened and the light bars darkened, holding the average constant, until the bar-space-bar pattern disappears. The contrast is then increased until the bars are once again visible. The average contrast between where the bars disappear and reappear is defined as threshold contrast for that bar pattern at that adapting luminance. The procedure is repeated for various bar spacingsthat is, for various spatial frequencies. The function of threshold contrast versus spatial frequency at each light level is called the CTF at that light level. When modeling a system, the display luminance and display size are specified. For the specified display luminance, the CTF measured for that luminance is an indicator of the eyes ability to integrate signal and discriminate noise at each spatial frequency. The fixed SNRT in (13) has been replaced with a variable SNRT shown in (17) SNRT CTF (17)

such that performance is limited by the eye as sensor noise becomes small. With both of these changes incorporated, (13) for sensor system MRT becomes MRT CTF where CTF (18)

is the MRT or . CTF

given in (13) without the 2.25 (SNRT)

is a constant. Eye MTF and variations in eye inwhere tegration time due to light level variations are included in the CTF, and these factors no longer explicitly appear in the modified MRT formula shown in (18). Eye contrast limitations affect the MRT equation in a second way. FLIR92 and previous models predict MRT based on the sensor noise and modulation transfer function (MTF) characteristics. Kornfeld and Lawson recognized that, in the limit as sensor noise decreases, the eyes noise or contrast threshold limitations would be the actual limit on performance [22]. In NVTherm, a second term has been added to the MRT equation

the CTF of the eye based on measured data; eye threshold calibration constant which replaces SNRT; 1050, which seems large, but the CTF of the eye is about 0.002 at peak sensitivity; scene thermal contrast which results in the is average display luminance. Typically, equal to half of the sensor dynamic range. The MRT equation now consists of two terms. The first term represents the contribution of sensor noise; the second term represents the contribution of eye noise or contrast threshold limitations. The relationship between these two terms depends on the amount of noise generated by the infrared detectors and on the setting of the sensor gain control. As the average scene temperature heats up, the sensor gain is reduced and the eye contrast term begins to dominate. As the scene cools off and gain is increased, or for intrinsically noisy detector arrays, the sensor noise dominates. In the limit of low detector noise or high scene thermal contrast, the MRT is dominated by the sensor optical blur and eye contrast limitations. In the limit of high sensor noise or very low thermal contrast, the sensor noise dominates MRT. The addition of eye contrast limitations to the MRT equation results in more realistic performance predictions from Acquire. For example, in order to maintain dark adaptation, display luminance is normally kept low during night operations. The new model correctly predicts performance under these realistic conditions. Further, the performance of new, sensitive staring arrays is now modeled correctly, because the contrast limitations become more important as the detector noise decreases. VII. MODEL IMPROVEMENT TO ADD SAMPLING The mathematical characterization of image artifacts that result from sampling, and the target acquisition performance loss that results from those artifacts, is described in several recently published books and papers [23][28]. This most recent and significant change to the thermal model is summarized here.

38

IEEE SENSORS JOURNAL, VOL. 1, NO. 1, JUNE 2001

Mathematically, the sensor and display are characterized by a sampled imager response function. The response function for a sampled imager is derived by examining the image formed on the display by a point source of light in the scene. The response function provides a quantitative way to characterize both the quality of the sampled imagers transfer response and its tendency to generate sampling artifacts. The sampled imager response function depends on the sensor pre-sample MTF, the sample spacing, and the post-sample and display MTF. These sensor characteristics are known to the design engineer or systems analyst. The sampled imager response function does not depend on the image samples, but rather on the process by which the samples are taken and displayed. Since the sampling artifacts produced by an imager depend on the scene being imaged, one might question a mathematical process which quantifies sampling artifacts without including an explicit description of the scene. In that regard, we rely on assumptions identical to those used for nonsampled imagers. MTF is widely used to characterize a nonsampled imager. MTF is the Fourier transform of the displayed point spread function. It describes the blur produced in the image by a point in the scene. In actual usage, the importance of a good MTF response at high frequency cannot be established until the high frequency content of the scene is established. The real impact or importance of the sensor blur is not known until the scene content and task are known. Nonetheless, MTF has proven to be a good indicator of the overall utility of the sensor. High frequency response in a nonsampled imager is a prized characteristic because of the possibilities it provides, not because good MTF is always important in accomplishing every task when looking at any scene. Experience has shown that MTF is a good way to characterize the quality of an imaging system. An image cannot be defined until the scene is described, but the characterization of the imagers response to a point source provides a good indication of the quality of images which can be expected under a variety of environments. A similar logic applies to sampled imagers. We cannot know how each detail in the scene will be corrupted by the sampling process until the exact scene and its angular relationship to the sensor are specified. However, the tendency of the imager to produce visible display raster or corrupt scene details can be characterized. For most practical, sampled imagers, the response function can be approximated by [23], [24]

(19) where pre-sample MTF; that is, the product of optics MTF, detector MTF, and any MTF losses due to the atmosphere or line-of-sight jitter; post-sample MTF; that is, the product of the electronics MTF, display MTF, and eye MTF; spatial frequency in cycles per milliradian; sample frequency in cycles per milliradian; sample phase.

Note that in (13) is the product of . The response function has two parts, the transfer term and the spurious response terms. The first term in (19) is the transfer response of the imager. This transfer response does not depend on sample spacing, and it is the only term that remains for small sample spacing. A sampled imager has the same transfer function as a nonsampled (that is, a very well-sampled) imager. However, a sampled imager always has the additional response terms, which we refer to as spurious response. These spurious response terms in (19) are filtered by the electronics, display, and eye MTFs in the same way that the transfer response is filtered. However, the position of the spurious response terms on the frequency axis depends on the sample spacing. If the sample spacing is large (the sample frequency is small), then the spurious response terms lie close to the baseband in frequency space. In this case, the spurious response is difficult to filter out and might even overlap the baseband. If the sample spacing is small (the sample frequency is high), then the spurious response terms lie far from the baseband in frequency space, and the spurious response is filtered out by the display and eyeball MTF. The transfer and spurious response functions of a sampled imager can be calculated using (19). These response functions provide the Fourier transform of the baseband, desirable, spatial image information and the Fourier transform of the sampling artifacts, respectively. The sampled imager response function mathematically describes the imaging behavior of the system. However, in predicting the effect of sampling on task performance, the response function must somehow be condensed into a sampling-goodness metric for the sensor. Some generalizations must be made and a goodness factor or factors calculated. Two aggregate quantities are defined which have proven useful in predicting how the spurious response of a sampled imaging system affects task performance. The utility of these quantities was discovered during experiments looking at the effect of sampling on target acquisition performance. The experiments are described in [25][27]. The two quantities are: total integrated spurious response ratio, SR, as defined by (20), shown at the bottom of the next page, and out-of-band spurious as defined by (21) and (22), as response ratio, SR - shown at the bottom of the next page. The results of perception experiments conducted by NVL show that in-band aliasing (aliasing which occurs at frequencies less than half the sample rate) did not degrade target identification performance, but out-of-band aliasing (such as visible display raster) degraded identification performance significantly. Aliasing had less impact on the recognition task than the identification task, but both in-band and out-of-band aliasing degraded recognition performance to some extent. Based on these experiments and other results reported in the literature, it appears that in-band aliasing has a strong affect on low-level discrimination tasks like hot-spot detection; out-of-band aliasing has only a minor impact on these tasks. For high-level discrimination tasks like target identification, however, out-of-band aliasing has a significant impact on performance, whereas in-band aliasing has a very minor affect. For intermediate-level discrimination tasks like target or character recognition, both in-band and out-of-band aliasing have a moderate impact on performance.

RATCHES et al.: TARGET ACQUISITION PERFORMANCE MODELING OF INFRARED IMAGING SYSTEMS

39

The performance loss associated with sampling is modeled as an increased blur on the imagery. The blur increase is characterized as a function of total spurious response for the recognition task and as a function of out-of-band spurious response for the identification task. Using the Fourier similarity theorem, an increase in blur is equivalent to a contraction of the modulation transfer function, thus MTF Squeeze. The squeeze for the recognition task is SR (23)

where SR is defined by (20). The squeeze factor for the identification task is SR - (24)

is defined by (21) and (22). A squeeze for where SR - the detection task would be speculation because experiments are not complete; a detection squeeze is not currently in the model. The spurious response is calculated independently in the horizontal and vertical directions, and the squeeze factor calculated. is calculated A new for recognition or for ID (26) (25)

The MTF Squeeze approach is an empirically-derived method for imposing a penalty for under-sampling. The penalty is not as severe as the half-sample rate limit imposed by FLIR92, but it gives performance below that of a well-sampled imager with identical pre-sample and post-sample transfer functions. VIII. FUTURE MODEL IMPROVEMENTS Johnsons target discrimination criteria have been widely used for 25 years for two reasons: the resulting model is simple, and the model predictions are reasonable. However, this elegantly simple model does not quantitatively predict all aspects of target acquisition performance. Johnsons original experiments were performed using high contrast targets. Rosell performed experiments using noisy targets; he found that the Johnson/NVL model is progressively more optimistic as noise increases [21]. If Rosells data are correct, then the NVL model predicts optimistically for poor

weather, battlefield obscurants, noisy detectors, and other cases where noise or contrast limits performance. Rosell, Biberman, and others have proposed alternatives to the NVL model to correct this behavior [25], [26]. Field experience has shown that these alternative models do not predict range performance as well as the NVL model. Nonetheless, it is still widely believed that the NVL model loses accuracy under high noise or low contrast conditions. For example, the current Army force-onforce war game requires (modulation) contrast to be more than 0.02 for target acquisition to occur. Without this limitation, the NVL model predicts that targets can be detected, recognized, or identified with a contrast as low as 0.002. Recent experiments at NVL show that the Johnson criteria is a decent predictor of the influence of noise and blur on target identification, but that other image metrics predict ID performance more accurately, especially when the images are noisy [27]. Another limitation of the current model, discussed more fully in [26] and [28], is that the model is a robust predictor of sensor system range performance only for generic detection, recognition, and identification tasks. Targets are represented only by an average area and average target to background contrast. The model provides an average or statistical prediction for a group of observers attempting the task many times with a variety of target types represented (that is, a variety of target types being detected, recognized, or identified). The model is useful in comparing sensor systems, because it predicts target acquisition performance for an ensemble of target types and aspects. However, the model does not predict the probability of discriminating a specific vehicle. As an example, the various generations of Russian tanks (T72, T62, T55) have the same general appearance, but look distinctly different from U.S. tanks (M1, M60). Discriminating an Iraqi-owned T72 from an Egyptian-owned T62 is much more difficult than discriminating the T72 from an Egyptian-owned M60. The target acquisition task always involves a comparison (target versus background for detection, target versus target for identification), and the difficulty of making these specific comparisons cannot be quantified by the current model. The current model improvement thrust at NVL is to improve the accuracy of the model when predicting the relative performance of diverse sensor technologies with different blur, noise, contrast, and sampling characteristics. This improvement will also insure that performance in poor weather or battlefield obscurants is also modeled correctly. Based on recent experiments,

SR

(20)

SR SR - SR SR -

(21) (22)

40

IEEE SENSORS JOURNAL, VOL. 1, NO. 1, JUNE 2001

this model improvement is both feasible and near term. The next step will be to address specific target-to-target and target-tobackground discriminations; that next step is expected to be a much more difficult, longer term task. REFERENCES
[1] J. Johnson, Analysis of image forming systems, in Proc. Image Intensifier Symp., 1958, pp. 249273. [2] O. H. Schade, Electro-optical characteristics of television systems, RCA Rev., vol. 9, no. 1, Mar. 1948. [3] R. Sendall and J. M. Lloyd, Improved specifications for infrared imaging systems, Proc. IRIS, vol. 14, no. 2, pp. 109129, 1970. [4] J. A. Ratches, W. R. Lawson, L. P. Ober, R. J. Bergemann, T. W. Cassidy, and J. M. Swenson, NVL static performance model for thermal viewing systems, U.S. Electronics Command Rep. ECOM 7043, AD-A011 212, Apr. 1973. [5] J. A. Ratches, Static performance model for thermal imaging systems, Opt. Eng., vol. 15, no. 6, Dec. 1976. , Comparison of NVL model and four contractor models for min[6] imum resolvable temperature (MRT), U.S. Electronics Command Rep. ECOM 7050, AD-A024 467, Jan. 1976. , NVL modeling; historical perspective, Proc. SPIE, vol. 3701, [7] pp. 112, 1999. [8] W. R. Lawson and J. A. Ratches, Modeling detection or the detection game, in Proc. IRIS Specialty Group on Targets, Backgrounds and Discrimination. Arlington, VA: Inst. Defense Anal., Aug. 1980. [9] J. A. DAgostino et al., ACQUIRE range performance model for target acquisition systems version 1 users guide, U.S. CECOM Night Vision & Electron. Sensors Directorate Rep., Fort Belvoir, VA, May 1995. [10] F. X. Kneizys, E. P. Shettle, W. O. Gallery, J. H. Chetwynd, Jr., L. W. Abreu, J. E. A. Selby, S. A. Clough, and R. W. Fenn, Atmospheric transmittance/radiance: Computer code LOWTRAN 6, AFGL-TR-830187, Air Force Geophys. Lab., Hanscom AFB, MA, Aug. 1983. [11] J. A. DAgostino et al., FLIR92 thermal imaging systems performance model analysts reference guide, Doc. UG5 008 993 USA CECOM Night Vision & Electron. Sensors Directorate, Fort Belvoir, VA, Jan. 1993. [12] L. B. Scott and L. Condiff, C2NVEO advanced FLIR systems performance model, Proc. SPIE, Apr. 1620, 1990. [13] J. A. DAgostino et al., FLIR92 thermal imaging systems performance model users guide, Doc. UG5 008 993 USA CECOM Night Vision & Electron. Sensors Directorate, Fort Belvoir, VA, Jan. 1993. [14] R. G. Driggers, R. Vollmerhausen, and K. Krapels, Target identification performance as a function of temporal and fixed pattern noise, Proc. SPIE, Apr. 2000. [15] J. C. Brown, C. Webb, P. Bell, R. Washington, and R. Riordan, AUTOSPEC image evaluation laboratory, Proc. SPIE, vol. 1488, 1991. [16] L. P. Obert, J. A. DAgostino, B. OKane, and C. Nguyen, An experimental study of the effect of vertical resolution on FLIR performance, in Proc. IRIS Specialty Group Passive Sensors, Mar. 1990. [17] J. D. Howe, L. B. Scott, S. P. Pletz, J. D. Horger, and J. S. Mark, Thermal model improvement through perception testing, in Proc. IRIS Specialty Group Passive Sensors, 1989. [18] B. OKane, M. Crenshaw, J. A. DAgostino, and D. Tomkinson, Human target detection using thermal devices, in Proc. IRIS Specialty Group Passive Sensors, vol. I, 1992, pp. 205218. [19] R. Vollmerhausen, Incorporating display limitations into night vision performance models, IRIS Passive Sensors, vol. 2, pp. 1131, 1995. , Modeling the performance of imaging sensors, in EO Imaging: [20] Systems and Modeling, L. Biberman, Ed. Andover, MA: ONTAR Corp., 2000, ch. 12. [21] F. A. Rosell and R. H. Wilson, Recent psychophysical experiments and the display signal-to-noise ratio concept, in Perception of Displayed Information, L. M. Biberman, Ed. New York: Plenum, 1973, p. 167. [22] G. H. Kornfeld and W. R. Lawson, Visual perception models, J. Opt. Soc. Amer., vol. 61, pp. 811820, June 1971. [23] R. Vollmerhausen and R. Driggers, Analysis of Sampled Imaging Systems, Tutorial Texts in Optical Engineering. Bellingham, WA: SPIE, 2000, vol. TT39. [24] R. Vollmerhausen, Display of sampled imagery, in EO Imaging: Systems and Modeling, L. Biberman, Ed. Andover, MA: ONTAR Corp., 2000, ch. 25.

[25] F. A. Rosell, Synthesis and analysis of imaging sensors, in EO Imaging: System Performance and Modeling, L. Biberman, Ed. N. Andover, MA: ONTAR Corp., Mar. 2000, ch. 14. [26] L. Biberman, Alternate modeling concepts, in EO Imaging: System Performance and Modeling, L. Biberman, Ed. N. Andover, MA: ONTAR Corp., Mar. 2000, ch. 11. [27] R. Vollmerhausen, R. Driggers, and M. Tomkinson, Improved image quality metric for predicting tactical vehicle identification, Proc. SPIE, Apr. 2000. [28] B. OKane, Modeling parameters for target identification: A Critical features analysis, in EO Imaging: System Performance and Modeling, L. Biberman, Ed. N. Andover, MA: ONTAR Corp., Mar. 2000, ch. 15. [29] O. H. Schade, Electro-optical characteristics of television systems, RCA Rev., vol. 9, no. 2, Mar. 1948. , Electro-optical characteristics of television systems, RCA Rev., [30] vol. 9, no. 3, Sept. 1948. , Electro-optical characteristics of television systems, RCA Rev., [31] vol. 9, no. 4, Dec. 1948.

James A. Ratches received the B.S. degree from Trinity College, Hartford, CT, in 1964 and the M.S. and Ph.D. degrees in physics from Worcester Polytechnic Institute, Worcester, MA, in 1966 and 1969, respectively. He is the Chief Scientist at the U.S. Armys Communications-Electronics Command Night Vision and Electro-Optics Directorate (NVESD). In his 28 years at NVESD, he has been a major contributor to the areas of modeling, simulation, characterization, and evaluation of electro-optical systems. His performance model for thermal imaging systems became the standard for the entire infrared community. As Director of the Visionics Division of NVESD, he reoriented the modeling, simulation, and evaluation activities from emphasis on human to machine performance. As Chief Scientist, he has become a spokesman for the military technical ATR community.

Richard H. Vollmerhausen currently heads the Model Development Branch at the Armys Night Vision Lab. The branch is updating the Armys target acquisition models to include the effect of sampling on performance and to make other model enhancements in order to predict the performance of advanced technology sensors. During his tenure at NVL, he was a Project Engineer and EO systems analyst for numerous Army weapon systems. His previous work included designing air-to-air missile seekers for the Navy and working as an Instrumentation Engineer for Douglas Aircraft on the Saturn/Apollo Program. He is the author of two books on electro-optical systems analysis and has published numerous journal and symposium papers.

Ronald G. Driggers has 12 years of electro-optics experience and has worked for or consulted to Lockheed Martin, SAIC, EOIR Measurements, Amtec Corporation, Joint Precision Strike Demonstration Project Office, and Redstone Technical Test Center. He is currently working for the U.S. Armys Night Vision and Electronic Sensors Directorate and is the U.S. representative to the NATO panel on advanced thermal imager characterization. He is the author of two books on Infrared and Electro-Optics Systems and has published over 30 refereed journal papers. He is Co-Editor of Marcel Dekkers Encyclopedia of Optical Engineering and is an Associate Editor of Optical Engineering.