This paper describes objective technical results and analysis.
Any subjective views or opinions that might be expressed
in the paper do not necessarily represent the views of the U.S. Department of Energy or the United States Government.
HRLs EXPLAINED 1
SAND2021-4299J
Human Readiness Levels Explained
Judi E. See
Sandia National Laboratories
Author Note
Sandia National Laboratories is a multimission laboratory managed and operated by
National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of
Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security
Administration under contract DE-NA0003525.
Correspondence concerning this article should be addressed to Judi E. See, Sandia
National Laboratories, P.O. Box 5800 MS 0151, Albuquerque, NM 87185-0151. Email:
[email protected]
Feature at a Glance
The human readiness level scale complements and supplements the existing technology
readiness level scale to support comprehensive and systematic evaluation of human-system
aspects throughout a system’s lifecycle. The objective is to ensure humans can use a fielded
technology or system as intended to support mission operations safely and effectively. This paper
defines the nine human readiness levels in the scale, explains their meaning, and illustrates their
application using a helmet-mounted display example.
Keywords: technology readiness level, usability, human systems integration, human
systems performance
HRLs EXPLAINED 2
Human Readiness Levels Explained
The human readiness level (HRL) scale is a simple nine-level scale designed to
complement and supplement the technology readiness level (TRL) scale widely used throughout
U.S. government, industry, and academia (GAO, 2020; Mankins, 1995; NASA, 1991). Whereas
the TRL scale focuses on the technical maturity of a developing technology or system, the HRL
scale evaluates, tracks, and communicates whether that technology or system is ready for
humans to use. In the HRL scale, human readiness refers to the maturity of a technology with
respect to use by intended users in the intended operational environment. The term does not
signify the current state of physical or mental readiness of human operators who will use the
technology to accomplish their mission work. In other words, the HRL scale supports
evaluations of suitability and usability of a technology for human use, not an operator’s
transitory fitness for duty at a given moment in time.
Mirroring the TRL scale’s familiar structure, the HRL scale provides assurance to
decision makers that humans, both individuals and teams, can use the fielded technology or
system as intended to support mission operations safely and effectively. HRLs support detection
and mitigation of human systems issues early in design to enhance usability and minimize human
error in fielded systems. In effect, the HRL scale shifts attention from lagging indicators of
human readiness, such as human error in fielded systems, to leading indicators—namely, the
evidence-based measures of system usability readiness derived through application of the HRL
scale. The assigned HRL rating readily communicates the readiness of a technology or system
for human use to program and acquisition managers to support decisions regarding future
program direction and allocation of funding.
HRLs EXPLAINED 3
Evolution of the HRL Scale
The HRL concept was first proposed in 2010 (Acosta, 2010). Since that time, multiple
organizations and experts in the human systems integration (HSI) community have had
significant roles in research, maturation, evaluation, and peer review of the HRL scale (see Table
1).
Table 1
Evolution of the HRL Scale
Year HRL Activity
2010 Dr. Hector Acosta proposes HRL concept (Acosta, 2010)
Naval Postgraduate School (NPS) master’s thesis provides first instantiation of a nine-level
2010
HRL scale (Phillips, 2010)
NPS master’s thesis proposes a framework to standardize HSI throughout development
2014
(O’Neil, 2014)
Chief Scientist of the Air Force, Dr. Mica Endsley, advocates requirements to augment TRL
2015
scale with nine-level HRL scale during system development (Endsley, 2015)
Department of Defense HSI working group refines HRL scale with more detailed
2015
descriptions (Phillips, 2015)
Sandia National Laboratories, Old Dominion University, and NPS chair a second working
2019
group to mature the HRL scale and assess utility (Salazar, See, Handley, & Craft, 2021)
Efforts over the past 10 years have transformed the HRL scale into a comprehensive
nine-level scale comparable to the TRL scale. At each of the nine HRL levels, the current scale
provides names and descriptions, trigger questions identifying fundamental human systems
topics necessary to evaluate system usability readiness, additional guidance for consideration
during human systems evaluations, exit criteria, and supporting evidence. At present, a
committee is in the final stages of writing a formal technical standard for the HRL scale.
Explaining the HRL Scale
The inspiration for the current paper came from an article in the Air Force Magazine that
explains TRLs (Grudo, 2016). That article adopted a concise, straightforward approach to briefly
define the nine TRL levels, explain their meaning, and illustrate their application through
HRLs EXPLAINED 4
concrete real-world examples. The objective of the current effort is to apply a similar approach to
the HRL scale, using a helmet-mounted display (HMD) example. An HMD is a device used
primarily in aircraft to project text and icons onto transparent visors worn by aircrew members,
providing an overlay that augments views of the natural world (Bayer, Rash, & Brindle, 2009;
Dansereau, Colombi, Miller, & Robbins, 2015). HMDs enable support of tasks such as aircraft
control, targeting, and weapon cueing, with improved response time, workload, and situation
awareness, as compared to head-up displays.
This paper describes the HRL scale in the context of three primary lifecycle phases:
(1) basic research and development, (2) technology demonstration, and (3) production and
deployment. As with the TRL scale, application of the HRL scale is designed to reduce program
risk. Whereas the TRL scale decreases the risk of integrating an immature technology, the HRL
scale reduces the risk of fielding a technology that is not ready for people to use. Accordingly,
human performance risk starts at a very high level during basic research and development at
HRL 1 and decreases over time as progressively higher HRL levels are satisfactorily met. This
reduction in risk is possible because the HRL scale supports repeated demonstration and testing
to evaluate the efficacy of recommended strategies designed to prevent or mitigate identified
human systems issues.
Basic Research and Development Phase
During this phase in the lifecycle, scientific research, analysis, and preliminary
development occur on paper and in the laboratory. This phase culminates in a validated proof of
concept that addresses human needs, capabilities, limitations, and characteristics.
HRLs EXPLAINED 5
HRL 1 – Basic Human Research
Definition
Basic principles for human
characteristics, performance, and behavior observed and reported.
What That Means
HRL 1 represents a broad, high-level exploration of human ramifications within a developing
concept or proposed practical application.
Helmet-Mounted Display Example
Potential human-centered issues and risks such as poor fit, excessive weight, or visual fatigue
due to insufficient display resolution are flagged as focus areas for continued exploration.
HRL 2 – Human-Centered Design Guidelines
Definition
Human-centered concepts,
applications, and guidelines defined.
What That Means
Implications for human use are analyzed. Human-centered design guidelines inform human
use requirements and preliminary designs.
Helmet-Mounted Display Example
Anthropometric guidelines are applied to ensure accommodation of 5th to 95th percentiles of
human head dimensions.
HRL 3 – Human-Centered Requirements
Definition
Human-centered requirements to
support human performance and human-technology interactions
established.
What That Means
Human needs are mapped to expected operational and system demands to establish human-
centered requirements.
Helmet-Mounted Display Example
Image quality requirements are determined to support human operational and functional needs
during HMD interactions.
Technology Demonstration Phase
The technology is repeatedly demonstrated at increasing levels of fidelity, first in the
laboratory and later in relevant environments. This phase concludes with demonstration of the
HRLs EXPLAINED 6
representative system in a high-fidelity simulation or actual environment, with representative
users supporting human systems evaluations.
HRL 4 – Part-Task Testing
Definition
Modeling, part-task testing, and
trade studies of human systems design concepts and applications
completed.
What That Means
Human systems design concepts are analyzed via trade studies and evaluated in laboratory
environments to identify viable options.
Helmet-Mounted Display Example
Accuracy and timing data are collected for representative tasks to evaluate effectiveness of
current image quality as compared to legacy systems.
HRL 5 – Prototypes in Mission-Relevant Simulations
Definition
Human-centered evaluation of
prototypes in mission-relevant part-task simulations completed to inform
design.
What That Means
Fidelity of key elements has increased significantly, and users participating in testing are
independent from the design team.
Helmet-Mounted Display Example
Display lag is evaluated in a fixed-based laboratory flight simulator, using human performance
metrics such as mental workload and user satisfaction.
HRL 6 – Human Systems Design Fully Matured
Definition
Human systems design fully
matured and demonstrated in a relevant high-fidelity, simulated
environment or actual environment.
What That Means
Human systems design maturity is evaluated with a functional prototype across the full range
of usage scenarios and tasks.
Helmet-Mounted Display Example
Effects of vibration on objective metrics such as image quality and display jitter are
demonstrated in a motion flight simulator.
HRLs EXPLAINED 7
Production and Deployment Phase
Final testing, verification, validation, and qualification occur in this phase, with human
performance evaluations based on representative users. This phase concludes with operational
use of the system and continued systematic monitoring of human-system performance.
HRL 7 – Human Systems Design Fully Tested
Definition
Human systems design fully
tested in operational environment with system hardware and software and
representative users.
What That Means
Final development system is evaluated to determine if recommendations to support human use
have been satisfactorily incorporated and have resolved identified human performance issues.
Helmet-Mounted Display Example
Data are collected during a military exercise, with representative users and a range of HMD
scenarios and tasks in realistic operational environments.
HRL 8 – Human Systems Design Verification and Approval
Definition
Human systems design fully
tested, verified, and approved in mission operations, using completed
system hardware and software and representative users.
What That Means
Human performance is verified with production system in a representative environment before
full-rate production and final system fielding.
Helmet-Mounted Display Example
Final production helmet is evaluated to verify metrics for successful human performance have
been met, possibly during development test and evaluation (DT&E).
HRL 9 – Operational Use and Monitoring
Definition
System successfully used in
operations across the operational envelope with systematic monitoring of
human system performance.
What That Means
Qualified system is fielded in the operational environment and operated by intended users.
Helmet-Mounted Display Example
After fielding, human systems experts conduct additional human-centered evaluations to
resolve emerging issues with HMD night vision acuity.
HRLs EXPLAINED 8
Practical Applications
The HRL scale, like the TRL scale, has been intentionally designed to provide a common
language applicable across a diverse range of technologies, for use by organizations throughout
the Department of Defense, Department of Energy, other federal agencies, industry, and
academia. Successful application of the HRL scale requires the participation of one or more
human systems experts who have specific education, training, and experience in human factors
engineering, human systems integration, or a related field. Such subject matter experts apply
their human systems expertise to evaluate the topics that must be addressed at each HRL level,
estimate the HRL rating, and provide justifications for that rating. As with the TRL scale, if a
team of subject matter experts is involved in determining a rating with the HRL scale, any
discrepancies among the individual ratings are resolved to obtain a single HRL rating for the
technology at a given point in time. Resolution may occur by reviewing supporting evidence, the
activities that have been completed, and the nature of the observations and results to date. The
supporting evidence that human systems experts provide is used to determine whether the HRL
level has been satisfactorily addressed and the exit criteria have been met to advance to the next
level.
Perhaps the most critical practical application of the HRL scale is to support
communications. Human systems experts are responsible for communicating the HRL rating at
multiple program levels. The intent is to provide the HRL rating at the same time as the TRL
rating to support a more robust and comprehensive assessment of a technology’s maturity and
facilitate decision making for future program directions and resource allocation. For example, an
HRL rating that lags a TRL rating by two or more levels signals increased risk to the program if
additional funding and labor are not applied to align the human readiness of the developing
HRLs EXPLAINED 9
technology with its technical maturity. Ultimately, applying the HRL scale during design and
development facilitates proactive, comprehensive, and systematic evaluation of the human-
related aspects of a system to supplement TRL evaluations of technical maturity. It should be
noted that, while the TRL and HRL ratings are intended to be reported simultaneously at key
decision points in the program, the ratings are developed independently. That is, the TRL level
does not need to be known to estimate an HRL level.
Once completed, the HRL technical standard provides additional details to supplement
the high-level overview provided in this paper. The standard describes detailed guidance to
effectively and comprehensively fulfill each HRL level. Exit criteria to determine when it is
appropriate to advance to a higher HRL level are also covered along with the types of evidence
that should be submitted to support such exit decisions. The completed technical standard
provides a reference that can be used to support application of the HRL scale in formal programs
of record.
HRLs EXPLAINED 10
REFERENCES
Acosta, H. (2010, May 9-13). Human readiness levels: Implementing HSI – Connecting some
dots [Panel discussion]. 81st Annual Scientific Meeting of the Aerospace Medical
Association, Phoenix, Arizona.
Bayer, M. M., Rash, C. E., & Brindle, J. H. (2009). Introduction to helmet-mounted displays. In
C. E. Rash, M. B. Russo, T. R. Letowski, & E. T. Schmeisser (Eds.), Helmet-mounted
displays: Sensation, perception, and cognition issues (pp. 47-108). U.S. Army
Aeromedical Research Laboratory. http://citeseerx.ist.psu.edu/viewdoc/download?
doi=10.1.1.452.4762&rep=rep1&type=pdf
Dansereau, M. R., Colombi, J. M., Miller, M. E., & Robbins, M. J. (2015, May 30 - June 2). A
design evaluation framework for helmet mounted displays in fighter aircraft [Paper
presentation]. 2015 Industrial and Systems Engineering Research Conference, Nashville,
TN. https://www.researchgate.net/profile/Matthew_Robbins/publication/275657024_
A_Design_Evaluation_Framework_for_Helmet_Mounted_Displays_in_Fighter_Aircraft/
links/5543a1b00cf23ff716851fa5.pdf
Endsley, M. (2015, February 10-11). Human readiness levels: Linking S&T to acquisition
[Plenary address]. 2015 National Defense Industrial Association Human Systems
Conference, Alexandria, VA. https://ndiastorage.blob.core.usgovcloudapi.net/ndia/2015/
human/WedENDSLEY.pdf
GAO (2020). Technology readiness assessment guide: Best practices for evaluating the
readiness of a technology for use in acquisition programs and projects (Report No.
GAO-20-48G). U.S. Government Accountability Office. https://www.gao.gov/assets/
710/704769.pdf
HRLs EXPLAINED 11
Grudo, G. (2016, August). Technology readiness levels, explained. Air Force Magazine, 99(8),
22-23. https://www.airforcemag.com/PDF/MagazineArchive/Magazine%20Documents/
2016/August%202016/0816infographic.pdf
Mankins, J. C. (1995). Technology readiness levels: A white paper. NASA.
http://www.artemisinnovation.com/images/TRL_White_Paper_2004-Edited.pdf
NASA (1991). Integrated technology plan for the civil space program (Report No. NASA-TM-
107988). Office of Aeronautics and Space Technology. https://www.lpi.usra.edu/lunar/
strategies/NASALunarArchitecture/exp_tech_plan.pdf
O’Neil, M. P. (2014). Development of a human systems integration framework for Coast Guard
acquisition (Unpublished master’s thesis). Naval Postgraduate School.
https://apps.dtic.mil/dtic/tr/fulltext/u2/a608012.pdf
Phillips, E. L. (2010). The development and initial evaluation of the human readiness level
framework (Unpublished master’s thesis). Naval Postgraduate School.
https://apps.dtic.mil/sti/pdfs/ADA525365.pdf
Phillips, H. (2015, May 4-8). DOD HFE TAG: Human readiness level (HRL) working group.
[Paper presentation]. DOD HFE TAG 69th Meeting, Orlando, FL.
Salazar, G., See, J. E., Handley, H. A. H., & Craft, R. (2021). Understanding human readiness
levels. Proceedings of the 2020 Human Factors and Ergonomics Society 64th Annual
Meeting, 64(1), 1765-1769. https://doi.org/10.1177/1071181320641427
Judi E. See is a systems analyst and human factors engineer at Sandia National
Laboratories in Albuquerque, New Mexico. She earned a PhD degree in
experimental psychology in 1994 and BCPE professional certification in human
factors in 2009. In August 2021, she completed a Master of Engineering degree
in systems engineering. Her research interests include vigilance, signal
detection theory, visual inspection, and human readiness levels. She can be contacted at
[email protected].
