Face Recognition Vendor Test

Ongoing

Still Face and Iris 1:1 Verification

Application Programming Interface (API)
VERSION 6.0

Patrick Grother
Mei Ngan
Kayee Hanaoka
Information Access Division
Information Technology Laboratory

Contact via [email protected]

April 6, 2023
 FRVT Ongoing 1:1

1 Revision History
2
3
Date Version Description
April 1, 2019 4.0 Initial document
June 24, 2020 4.0.1 Update feature extraction times in Table 1.3 from 1000ms to 1500ms
September 9, 2020 4.0.2 Update link to General Evaluation Specifications document
Adjust the legal similarity score range
March 22, 2021 4.0.3 Update 1:1 matching time limit in Table 1.3 from 5 milliseconds to 0.1
milliseconds (or 100 microseconds)
January 7, 2022 5.0 Add second version of createTemplate() function in Section 4.4.4 that
supports the existence of multiple people in an image
February 2, 2022 5.0.1 Add Figure 2 and Table 3 to illustrate the second version of
createTemplate() function from Section 4.4.4
March 24, 2022 5.0.2 Add verbiage to be more explicit about algorithmic behavior when the
software fails to find a face in an image in Sections 4.4.3 and 4.4.4
April 6, 2023 6.0 Add support for iris recognition
Remove references to deprecated Multiface data structure
4

NIST API Page 1 of 9

 FRVT Ongoing 1:1

5 Table of Contents
6 1. FRVT 1:1 ............................................................................................................................................................................. 3
7 1.1. Scope ........................................................................................................................................................................ 3
8 1.2. General FRVT Evaluation Specifications ................................................................................................................... 3
9 1.3. Time limits ................................................................................................................................................................ 3
10 2. Data structures supporting the API ................................................................................................................................... 3
11 3. Implementation Library Filename ..................................................................................................................................... 3
12 4. API Specification................................................................................................................................................................. 3
13 4.1. Header File ............................................................................................................................................................... 4
14 4.2. Namespace ............................................................................................................................................................... 4
15 4.3. Overview................................................................................................................................................................... 4
16 4.4. API............................................................................................................................................................................. 5
17
18 List of Tables
19 Table 1 – Processing time limits in milliseconds, per 640 x 480 image ..................................................................................... 3
20 Table 2 – Functional summary of the 1:1 application of Figure 1 ............................................................................................. 4
21 Table 3 – Functional summary of the 1:1 application of Figure 2 ............................................................................................. 5
22 Table 4 – Initialization ................................................................................................................................................................ 6
23 Table 5 – Template generation from one or more images of exactly one person .................................................................... 7
24 Table 6 – Template generation of one more people detected from a single image ................................................................. 8
25 Table 7 – Template matching .................................................................................................................................................... 9
26
27 List of Figures
28 Figure 1 – Schematic of 1:1 verification (template generation of one or more images of exactly one person) ...................... 4
29 Figure 2 – Schematic of 1:1 verification (template generation of one or more people detected in an image) ....................... 5
30

31

NIST API Page 2 of 9

 FRVT Ongoing 1:1

32 1. FRVT 1:1
33 1.1. Scope
34 This document establishes a concept of operations and an application programming interface (API) for evaluation of face
35 recognition (FR) or iris recognition implementations submitted to NIST's ongoing Face Recognition Vendor Test. This API
36 is for the 1:1 identity verification track. Separate API documents will be published for future additional tracks to FRVT.
37

38 1.2. General FRVT Evaluation Specifications

39 General and common information shared between all Ongoing FRVT tracks are documented in the FRVT General
40 Evaluation Specifications document - https://pages.nist.gov/frvt/api/FRVT_common.pdf. This includes rules for
41 participation, hardware and operating system environment, software requirements, reporting, and common data
42 structures that support the APIs.

43 1.3. Time limits

44 The elemental functions of the implementations shall execute under the time constraints of Table 1. These time limits
45 apply to the function call invocations defined in section 3. Assuming the times are random variables, NIST cannot regulate
46 the maximum value, so the time limits are median values. This means that the median of all operations should take less
47 than the identified duration.
48 The time limits apply per image. When K images of a person are present, the time limits shall be increased by a factor K.
49 NOTE: For developers that cannot meet the required time limit for matching two templates, please contact [email protected].
50 Table 1 – Processing time limits in milliseconds, per 640 x 480 image
Function 1:1 verification
Feature extraction enrollment 1500 (1 core)
640x480 pixels
Feature extraction for verification 1500 (1 core)
640x480 pixels
Matching 0.1 (1 core)

51 2. Data structures supporting the API

52 The data structures supporting this API are documented in the FRVT - General Evaluation Specifications document
53 available at https://pages.nist.gov/frvt/api/FRVT_common.pdf with corresponding header file named frvt_structs.h
54 published at https://github.com/usnistgov/frvt.

55 3. Implementation Library Filename

56 The core library shall be named as libfrvt_11_<provider>_<sequence>.so, with
57 • provider: single word, non-infringing name of the main provider. Example: acme
58 • sequence: a three digit decimal identifier to start at 000 and incremented by 1 every time a library is sent to
59 NIST. Example: 007
60
61 Example core library names: libfrvt_11_acme_000.so, libfrvt_11_mycompany_006.so.
62 Important: Public results will be attributed with the provider name and the 3-digit sequence number in the submitted
63 library name.

64 4. API Specification
65 FRVT 1:1 participants shall implement the relevant C++ prototyped interfaces in Section 4.4. C++ was chosen in order to
66 make use of some object-oriented features.

NIST API Page 3 of 9

 FRVT Ongoing 1:1

67 4.1. Header File

68 The prototypes from this document will be written to a file named frvt11.h and will be available to implementers at
69 https://github.com/usnistgov/frvt.

70 4.2. Namespace
71 All supporting data structures will be declared in the FRVT namespace. All API interfaces/function calls for this track will
72 be declared in the FRVT_11 namespace.

73 4.3. Overview
74 The 1:1 testing will proceed in the following phases: optional offline training; preparation of enrollment templates;
75 preparation of verification templates; and matching. NIST requires that these operations may be executed in a loop in a
76 single process invocation, or as a sequence of independent process invocations, or a mixture of both.
77
78
1:1 VERIFICATION Native or
updated
configuration
data
Enrollment initialization Verification initialization
Match initialization
Key
SDK SDK
Image(s) Comparison Image(s)
Enrollment Verification Algorithm
template engine template component
SDK behind FRVT API

Data passed by
NIST to
Score algorithm
79
80 Figure 1 – Schematic of 1:1 verification (template generation of one or more images of exactly one person)
81
82 Table 2 – Functional summary of the 1:1 application of Figure 1
Phase Description Performance Metrics to be reported by NIST
Initialization Function to read configuration data, if any. None
Enrollment Given K  1 input images of an individual, the implementation Statistics of the time needed to produce a template.
will create a proprietary enrollment template. That is, Statistics of template size. Rate of failure to produce a
createTemplate(role=FRVT::TemplateRole::Enrollment_11) template.
will be called. NIST will manage storage of these templates.
Verification Given K  1 input images of an individual, the implementation Statistics of the time needed to produce a template.
will create a proprietary verification template. That is, Statistics of template size. Rate of failure to produce a
createTemplate(role=FRVT::TemplateRole::Verification_11) template.
will be called. NIST will manage storage of these templates.
Matching (i.e. Given a proprietary enrollment and a proprietary verification Statistics of the time taken to compare two templates.
comparison) template, compare them to produce a similarity score. Accuracy measures, primarily reported as DETs,
including for partitions of the input datasets.

NIST API Page 4 of 9

 FRVT Ongoing 1:1

1:1 VERIFICATION Native or

updated
configuration
data
Enrollment initialization Verification initialization
Match initialization
Key
SDK SDK
Enrollment Comparison Verification Algorithm
template #1 engine template #1
Image Image component
SDK behind FRVT API
Enrollment Verification
template #2 template #2
Data passed by
Enrollment NIST to
Verification
MxN Similarity algorithm
template M template N
scores

83
84 Figure 2 – Schematic of 1:1 verification (template generation of one or more people detected in an image)
85
86 Table 3 – Functional summary of the 1:1 application of Figure 2
Phase Description Performance Metrics to be reported by NIST
Initialization Function to read configuration data, if any. None
Enrollment Given K = 1 input image, the implementation will create M Statistics of the time needed to produce M templates.
proprietary enrollment templates based on the number of Statistics of template size. Rate of failure to produce a
people detected in the image. That is, template.
createTemplate(role=FRVT::TemplateRole::Enrollment_11)
will be called. NIST will manage storage of these templates.
Verification Given K = 1 input image, the implementation will create N Statistics of the time needed to produce N templates.
proprietary verification templates based on the number of Statistics of template size. Rate of failure to produce a
people detected in the image. That is, template.
createTemplate(role=FRVT::TemplateRole::Verification_11)
will be called. NIST will manage storage of these templates.
Matching (i.e. Given a M proprietary enrollment templates and N Statistics of the time taken to compare two templates.
comparison) proprietary verification templates, cross compare them to Accuracy measures, primarily reported as DETs,
produce MxN similarity scores. including for partitions of the input datasets.
87

88 4.4. API

89 4.4.1. Interface
90 The software under test must implement the interface Interface by subclassing this class and implementing each
91 method specified therein.
C++ code fragment Remarks
1. class Interface
2. {
public:
3. virtual ReturnStatus initialize( Supports algorithm initialization
const std::string &configDir ) = 0;
4. virtual ReturnStatus createFaceTemplate( Supports template generation from one or more
const std::vector<Image> &faces, face images of exactly one person
TemplateRole role,
std::vector<uint8_t> &templ,
std::vector<EyePair> &eyeCoordinates) = 0;

NIST API Page 5 of 9

 FRVT Ongoing 1:1

5. virtual ReturnStatus createIrisTemplate( Supports template generation from one or more iris
const std::vector<Image> &irises, images of exactly one person
TemplateRole role,
std::vector<uint8_t> &templ,
std::vector<IrisAnnulus> &irisLocations) = 0;
6. virtual ReturnStatus createFaceTemplate( Supports template generation of one or more
const Image &image, people detected from a single image
TemplateRole role,
std::vector<std::vector<uint8_t>> &templs,
std::vector<EyePair> &eyeCoordinates) = 0;
7. virtual ReturnStatus matchTemplates( Supports comparison between two templates
const std::vector<uint8_t> &verifTemplate,
const std::vector<uint8_t> &enrollTemplate,
double &similarity) = 0;
8. static std::shared_ptr<Interface> getImplementation(); Factory method to return a managed pointer to the
Interface object. This function is implemented by
the submitted library and must return a managed
pointer to the Interface object.
9. };
92
93 There is one class (static) method declared in Interface. getImplementation() which must also be implemented
94 by the implementation. This method returns a shared pointer to the object of the interface type, an instantiation of the
95 implementation class. A typical implementation of this method is also shown below as an example.
96
C++ code fragment Remarks
#include "frvt11.h"

using namespace FRVT_11;

NullImpl:: NullImpl () { }

NullImpl::~ NullImpl () { }

std::shared_ptr<Interface>
Interface::getImplementation()
{
return std::make_shared<NullImpl>();
}

// Other implemented functions

97 4.4.2. Initialization
98 The NIST test harness will call the initialization function in Table 4 before calling template generation or matching. This
99 function will be called BEFORE any calls to fork()1 are made.
100 Table 4 – Initialization
Prototype ReturnStatus initialize(
const string &configDir); Input
Description This function initializes the implementation under test. It will be called by the NIST application before any call to
createTemplate() or matchTemplates(). The implementation under test should set all parameters.
This function will be called N=1 times by the NIST application, prior to parallelizing M >= 1 calls to
createTemplate() via fork().
Input Parameters configDir A read-only directory containing any developer-supplied configuration parameters or run-
time data files. The name of this directory is assigned by NIST, not hardwired by the
provider. The names of the files in this directory are hardwired in the implementation and
are unrestricted.
Output none
Parameters
Return Value See General Evaluation Specifications document for all valid return code values.

1 http://man7.org/linux/man-pages/man2/fork.2.html

NIST API Page 6 of 9

 FRVT Ongoing 1:1

101 4.4.3. Template generation from one or more images of exactly one person
102 The functions of Table 5 supports role-specific generation of template data from one or more images of exactly one
103 person. Template format is entirely proprietary. Some of the proposed datasets include K > 2 image per person for some
104 persons. This affords the possibility to model a recognition scenario in which a new image of a person is compared
105 against all prior images. Use of multiple images per person has been shown to elevate accuracy over a single image.
106 NOTE: For any given submission, developers may only implement ONE of the functions in Table 5. That is, a single
107 submission may only support face recognition or iris recognition. For the functions that are not implemented, the
108 function shall return ReturnCode::NotImplemented.
109 Using this function, NIST will enroll K >= 1 images under each identity. The method by which the face or iris recognition
110 implementation exploits multiple images is not regulated. The test seeks to evaluate developer provided technology for
111 multi-presentation fusion.
112 This document defines a template to be a developer defined data-structure, capable of holding zero or one embeddings.
113 A developer may include embeddings, other information derived from the image, a header; the contents is developer
114 defined. An algorithm might internally fuse K feature sets into a single model or maintain them separately. In any case,
115 the resulting proprietary template is contained in a contiguous block of data. A template may have length zero bytes. In
116 all cases, the matchTemplates() function must accept two templates. The matchTemplates() function will be called
117 even if the developer’s implementation of a template is of zero bytes.
118
119 Table 5 – Template generation from one or more images of exactly one person
Prototype for ReturnStatus createFaceTemplate(
face const std::vector<Image> &faces, Input
recognition TemplateRole role, Input
std::vector<uint8_t> &templ, Output
std::vector<EyePair> &eyeCoordinates); Output
Prototype for ReturnStatus createIrisTemplate(
iris const std::vector<Image> &irises, Input
recognition TemplateRole role, Input
std::vector<uint8_t> &templ, Output
std::vector<IrisAnnulus> &irisLocations); Output

Description Takes a vector of image(s) and outputs a proprietary template and associated eye coordinates or irisLocations. The
vectors to store the template and eye coordinates/iris locations will be initially empty, and it is up to the
implementation to populate them with the appropriate data. In all cases, even when unable to extract features, the
output shall be a template that may be passed to the matchTemplates() function without error. That is, this routine
must internally encode "template creation failed", and the matcher must transparently handle this. The table below
specifies algorithmic behavior based on whether a face or iris was detected/features were extracted from the input
image(s).

ReturnCode Output Template

Zero faces/irises detected A non-successful return code One template (could be zero bytes)
K = 1 faces/irises detected Success One template
Input faces or irises Implementations must alter their behavior according to the number of images contained in
Parameters the structure and the TemplateRole type.
role Label describing the type/role of the template to be generated. Valid values are
FRVT::TemplateRole::Enrollment_11 or FRVT::TemplateRole::Verification_11.
Output templ The output template. The format is entirely unregulated. This will be an empty vector when
Parameters passed into the function, and the implementation can resize and populate it with the
appropriate data.
eyeCoordinates or The function shall return
irisLocations − For face images, eye coordinates – the estimated eye centers for left and right eyes
− For iris images, iris locations - estimates of the limbus center and pupil and limbus radii

NIST API Page 7 of 9

 FRVT Ongoing 1:1

Return Value See General Evaluation Specifications document for all valid return code values.

120 4.4.4. Template generation of one or more people detected from an image
121 This function supports role-specific generation of one or more templates that correspond to one or more people detected
122 in an image. Some of the proposed test images include K > 1 persons for some images and situations where the subject of
123 interest may or may not be the foreground face (largest face in the image). This function allows the implementation to
124 return a template for each person detected in the image. For testing, NIST will
125 1. Generate one or more enrollment templates from a single call to this function or the function of Table 5
126 2. Generate one or more verification templates from a single call to this function or the function of Table 5
127 3. Match all enrollment templates from 1) with all verification templates from 2)
128 4. Use the maximum similarity score across all template comparisons from 3) in our calculation of FMR and FNMR
129 (this applies to both genuine and imposter comparisons)
130 NOTE: The implementation must be able to match any combination of enrollment and verification templates generated
131 from this function and the function of Table 5. In other words, the output template format should be consistent between
132 this function and the function of Table 5. A template may have length zero bytes. In all cases, the matchTemplates()
133 function must accept two templates. The matchTemplates() function will be called even if the developer’s
134 implementation of a template is of zero bytes.
135
136 Table 6 – Template generation of one more people detected from a single image
Prototypes ReturnStatus createFaceTemplate(
const Image &image, Input
TemplateRole role, Input
std::vector<<std::vector<uint8_t>> &templs, Output
std::vector<EyePair> &eyeCoordinates); Output
Description This function supports template generation of one or more people detected from a single image. It takes a single
input image and outputs one or more proprietary templates and associated eye coordinates based on the number of
people detected. The vectors to store the template(s) and eye coordinates will be initially empty, and it is up to the
implementation to populate them with the appropriate data. If the implementation is unable to extract features, the
output shall still contain a single template that may be passed to the matchTemplates() function without error. That
is, this routine must internally encode "template creation failed", and the matcher must transparently handle this.
The table below specifies algorithmic behavior based on the number of faces detected from the input image.

ReturnCode Output Template

Zero faces detected A non-successful return code Vector of templates, length = 1
K >= 1 faces detected Success Vector of templates, length = K
Input image A single image that contains one or more people in the photo
Parameters role Label describing the type/role of the template to be generated. Valid values are
FRVT::TemplateRole::Enrollment_11 or FRVT::TemplateRole::Verification_11.
Output templs A vector of output template(s). The format of the template(s) is entirely unregulated. This
Parameters will be an empty vector when passed into the function, and the implementation can resize
and populate it with the appropriate data.
eyeCoordinates For each person detected in the image, the function shall return the estimated eye centers.
This will be an empty vector when passed into the function, and the implementation shall
populate it with the appropriate number of entries. Values in eyeCoordinates[i] shall
correspond to templs[i].
Return Value See General Evaluation Specifications document for all valid return code values.

NIST API Page 8 of 9

 FRVT Ongoing 1:1

137

138 4.4.5. Matching

139 Matching of one enrollment against one verification template shall be implemented by the function of Table 7.
140 Table 7 – Template matching
Prototype ReturnStatus matchTemplates(
const std::vector<uint8_t> &verifTemplate, Input
const std::vector<uint8_t> &enrollTemplate, Input
double &score); Output
Description Compare two proprietary templates and output a similarity score, which need not satisfy the metric properties.
When either or both of the input templates are the result of a failed template generation (see Table 5), the
similarity score shall be -1 and the function return value shall be VerifTemplateError.
Input Parameters verifTemplate A verification template from createTemplate(role=Verification_11). The
underlying data can be accessed via verifTemplate.data(). The size, in bytes, of
the template could be retrieved as verifTemplate.size().
enrollTemplate An enrollment template from createTemplate(role=Enrollment_11). The
underlying data can be accessed via enrollTemplate.data(). The size, in bytes, of
the template could be retrieved as enrollTemplate.size().
Output score A score resulting from comparison of the templates.
Parameters
Measure of similarity or dissimilarity between the enrollment template and
verification template.
− For face recognition, a similarity score - higher is more similar
− For iris recognition, a non-negative measure of dissimilarity (maybe a
distance) - lower is more similar
An algorithm is free to assign any value to a candidate. The distribution of values
will have an impact on the false-negative and false-positive rates.

The score values should be reported on the range that is used in the developer’s
software products. We require scores to be non-negative. Developers often use
[0,1], for example. Our test reports include various plots with threshold values
e.g. FMR(T), to allow end-users to set thresholds in operations. These plots may
become difficult to interpret if scores span many orders of magnitude.
Return Value See General Evaluation Specifications document for all valid return code values.

141

NIST API Page 9 of 9