Friend Link
Ministry of Commerce of the people's Republic of China
State Administration for market Regulation
Standardization Administration of China
National Center of Standards Evaluation,SAMR
China National Institute of Standardization
China zhijian Publishing House
TechnicalSupport
Performance testing and reporting are key to maintaining accurate and unbiased systems.
Biometric systems are used to automatically authenticate and identify someone. Most of us will interact with these systems when using face or fingerprint ID to unlock our mobile devices. Other applications include iris scanning, signature verification, and more. Essentially, these systems rely on specific data derived from biological and behavioural traits of an individual.
By 2028, the global biometric systems market is tipped to be valued at USD $85 billion1. It is a fast-growing industry with far-reaching applications. As with all accelerated technologies there are several risks, particularly in relation to privacy, data collection, and storage and system performance.
To respond to these risks, Standards Australia recently updated part one of its biometric performance testing and reporting standard (AS ISO/IEC 19795.1:2022, Information technology - Biometric performance testing and reporting, Part 1: Principles and framework). The revision provides guidance on achieving unbiased and accurate reporting, while introducing a level playing field for vendor testing and evaluation.
Below, we explore the emergence of biometric systems, the importance of testing and reporting, and the impact of this revision.
The emergence of biometric systems
According to Stacy Oerder – Chair of Standards Australia IT-032 committee, Biometrics and Identification – the growth in biometric systems is partially linked to the aftermath of the terrorist attack of September 11, 2001. The attack propelled the international community to push for interoperability, specifically for facial recognition in identity and travel documents.
Shortly after, it was recommended to install biometric screening systems at entry and exit points at the US border2.
In 2003, the Australian Government passed legislation allowing officials to obtain biometric information from non-Australian citizens. This legislation aimed to prevent identity fraud in the visa application process. In 2005, the Government implemented the ePassport, a micro chipped passport that people can use when traveling through SmartGates across airports. The SmartGate system uses the biometric information within the ePassport to automate customs and immigration checks3.
Since then, biometric authentication systems are commonly used across various industries, from banking, healthcare to consumer electronic devices and more.
Biometric system performance
The performance of biometric systems can be affected by various factors, including the composition of the training data, the matching algorithm and the interpretation of the matching errors. Simply put, data can be skewed towards certain characteristics if the associated biometric system is trained on a less diverse data set.
Facial recognition software has been criticised for the potential to be biased, and testing has found this to be the case4. To evaluate the performance of a biometric system the US National Institute of Standards and Technology (NIST) conducts large-scale Facial Recognition Vendor Tests (FRVT) on multiple algorithms under various test scenarios. The tests provide a benchmark to vendors and potential users of the performance of the algorithms.
A recent demographic effects test conducted by NIST confirmed that there are demographic differentials that affect performance and that developers should design algorithms to eliminate this bias. This test is only one example of how the performance of a biometric system can be affected. There are numerous factors that can cause errors in biometric systems, in both capturing and enrolling the biometric data. For example, errors can occur when the samples collected are either poor or damaged, which may be a result of the capture device or the environment.
The risk of these errors mean that the biometric system won’t work as intended for the user, or the user won’t be able to use the system at all.
How does AS ISO/IEC 19795.1:2022 help?
“The importance of this standard cannot be overstated,” says Ms Oerder.
“It provides a set of principles and guidelines on testing to introduce a level playing field for all vendor testing, which enables the comparison of products.
“Without standards and practices like this, potential customers or implementers are dependent on their suppliers providing unbiased reporting on performance and capabilities.”
AS ISO/ IEC 197951:2022 specifies requirements on test protocols with the intended goal to reduce bias due to inappropriate data collection or analytic procedures.
The standard creates the general principle for testing the performance of biometric systems with reference to the error rates and throughput rates with specified performance requirements.
It aims to identify performance metrics for biometrics systems, help achieve the best estimate of field performance for the expended effort and enhance understanding of the limits of applicability of the test results.
"The updated standard informs people of the performance algorithms for both verification and identification and promotes uniform testing of algorithms,” continued Ms Oerder.
"By using biometrics we had to know how algorithms match as well as how biometric systems performed. The complexity that we have seen over time is because algorithms are getting smarter and better.”