Rand Rra1198-2

Research Report
C O R P O R AT I O N
DAVID SCHULKER, MATTHEW WALSH, NELSON LIM, AJAY K. KOCHHAR
How the U.S. Air Force

Can Incorporate New
Data Technologies into
Its Talent Management
System
Framework and Use Cases for Technology-Enabled
Talent Management
For more information on this publication, visit www.rand.org/t/RRA1198-2.
About RAND
The RAND Corporation is a research organization that develops solutions to public policy challenges to help make communities
throughout the world safer and more secure, healthier and more prosperous. RAND is nonprofit, nonpartisan, and committed to the
public interest. To learn more about RAND, visit www.rand.org.
Research Integrity
Our mission to help improve policy and decisionmaking through research and analysis is enabled through our core values of quality
and objectivity and our unwavering commitment to the highest level of integrity and ethical behavior. To help ensure our research
and analysis are rigorous, objective, and nonpartisan, we subject our research publications to a robust and exacting quality-assurance
process; avoid both the appearance and reality of financial and other conflicts of interest through staff training, project screening,
and a policy of mandatory disclosure; and pursue transparency in our research engagements through our commitment to the open
publication of our research findings and recommendations, disclosure of the source of funding of published research, and policies to
ensure intellectual independence. For more information, visit www.rand.org/about/research-integrity.
RAND’s publications do not necessarily reflect the opinions of its research clients and sponsors.
Published by the RAND Corporation, Santa Monica, Calif.
© 2022 RAND Corporation
is a registered trademark.
Library of Congress Cataloging-in-Publication Data is available for this publication.
ISBN: 978-1-9774-1043-6
Cover: Manager: mavoimages/stock.adobe.com; network: Belkin & Co/stock.adobe.com.
Cover design: Rick Penn-Kraus
Limited Print and Electronic Distribution Rights
This publication and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is
provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to its webpage
on rand.org is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research products for
commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.
About This Report
Technology-enabled practices have potential to greatly enhance how the U.S. Air Force (USAF) manages its
people. These technologies include data management, information technologies, and artificial intelligence
(AI). These technologies can enhance human resource management by making available new sources of data
and by using AI to extract new information and guide decisions using these data. By adopting technology-
enabled practices, the USAF can shift from being governed by broad, standardized policies informed by
periodic analyses to a system in which decisions are continually refined to meet goals using the best available
information. This report builds on prior research to present a framework to help policymakers understand
the distinctive elements of technology-enabled practices, along with descriptions of use cases in each area
of talent management. The report closes with a description of three broad implementation challenges and a
structured approach to move forward with adoption.
This report is one in a set aimed at helping the USAF understand the elements necessary for technology-
enabled talent management. Other reports in this set are
• Douglas Yeung, Elicia M. John, Jeannette Gaudry Haynie, James Ryseff, Bonnie L. Triezenberg, and
Nelson Lim, Implementing Technology-Enabled Human Resources Capabilities in the U.S. Air Force:
Insight from the Private Sector and Military Services, RR-A1198-1, 2022.
• Don Snyder, Funding Technology-Related Business Initiatives in the Department of the Air Force,
RR-A1198-3, 2022.
The research reported here was commissioned by the Director, Force Development, Deputy Chief of
Staff for Manpower, Personnel and Services, Headquarters U.S. Air Force, and conducted within the Work-
force, Development, and Health Program of RAND Project AIR FORCE as part of a fiscal year 2021 project,
“Enabling Future Technology-Enabled Human Resources Management for the United States Air Force.”
RAND Project AIR FORCE

RAND Project AIR FORCE (PAF), a division of the RAND Corporation, is the Department of the Air Force’s
(DAF’s) federally funded research and development center for studies and analyses, supporting both the
United States Air Force and the United States Space Force. PAF provides the DAF with independent analyses
of policy alternatives affecting the development, employment, combat readiness, and support of current and
future air, space, and cyber forces. Research is conducted in four programs: Strategy and Doctrine; Force
Modernization and Employment; Resource Management; and Workforce, Development, and Health. The
research reported here was prepared under contract FA7014-16-D-1000.
Additional information about PAF is available on our website:
www.rand.org/paf/
This report documents work originally shared with the DAF on April 29, 2021. The draft report, issued on
August 30, 2021, was reviewed by formal peer reviewers and DAF subject-matter experts.
Acknowledgments
We thank Lt Gen Brian Kelly, Deputy Chief of Staff for Manpower, Personnel and Services, for his guidance
and support of this work. We also thank Gregory D. Parsons, Director of Plans and Integration, and the staff
iii
How the U.S. Air Force Can Incorporate New Data Technologies into its Talent Management System
of AF/A1X for providing valuable information and feedback, and multiple Air Force interviewees for pro-
viding their insights. We also thank Darrell Jones and our colleagues—Benjamin Boudreaux, Laura Miller,
David Orletsky, Albert Robbert, and Sean Robson—for insightful comments and constructive feedback that
improved the quality of this report.
iv
Summary
The U.S. Air Force (USAF) and the other military services have a long history of innovation in human
resource management (HRM). The recent industry boom in data-related technologies has prompted USAF
leaders to sponsor research on how these technologies could further improve HRM decisions. This report
describes the common theme of this research portfolio, which is that adopting HRM practices that are
technology-enabled could lead to more-effective talent management. Of course, technologies exist on a
spectrum, and the USAF, like all other organizations, already rely on some technologies to perform HRM
functions. However, by pursuing the latest technological advances, the USAF can continue to improve the
efficiency and effectiveness of HRM processes.
To help policymakers understand the contrast between technology-enabled practices and practices already
in place that make use of rich data, we describe industry practices that fit under the umbrella of technology-
enabled talent management and present a framework highlighting their distinctive features. We focus prin-
cipally on the use of artificial intelligence (AI) and other analytic techniques to derive insight from data at
speed and scale. Then, we present use cases in which recent research has demonstrated technology-enabled
practices in the USAF context, discuss barriers to further implementation, and present an implementation
structure for moving toward greater adoption of these practices. In researching the path to a technology-
enabled talent management system, we found the following:
• Large industry firms, such as IBM, use technology-enabled techniques to improve employee experiences
by customizing talent management decisions at a large scale.
• All firms face challenges in applying technology-enabled techniques to talent management, but features
of USAF talent management processes and associated data place the organization in a good position
with regard to technical feasibility.
• Recent research has demonstrated the functionality of elements of technology-enabled business prac-
tices, and particularly of AI, in most areas of talent management.
• Legacy policy structures, existing culture, and limitations in the USAF data infrastructure stand out as
barriers to fully leveraging emerging technologies for HRM.
• A structured implementation approach to adoption of technology-enabled practices would address
(1) organizational and policy foundations; (2) the technological foundation; (3) data curation, data man-
agement, and data services; (4) analysis systems, methods, and services; and (5) enterprise integration
and deployment.
v
Contents
About This Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
Figures and Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
CHAPTER 1
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Technology-Enabled Talent Management Becoming a Commercial Reality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
CHAPTER 2
How the USAF Uses Data for Human Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Technology-Enabled Business Practices: The Future of USAF Talent Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
USAF Well Positioned to Join These Organizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
CHAPTER 3
Use Cases for Technology-Enabled Talent Management in USAF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Tactical Recruiting Challenge: Helping Recruiters Manage Their Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Strategic Recruiting Challenge: Providing Enough Resources for Recruiters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Specialty Classification Challenge: Matching Airmen to the Right Job . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Development Challenge: Labor-Intensive Board Selection Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Assignments Challenge: How to Best Use the New “Talent Marketplace” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Promotion Challenge: Unpredictable Effects of Policy Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Retention Challenge: Preventing Personnel Shortages or Surpluses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
The Path to a Technology-Enabled Capability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Implementation Challenges for a Technology-Enabled Talent Management System . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
CHAPTER 4
A Way Forward for Technology-Enabled Talent Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Organizational and Policy Foundations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Technological Foundation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Data Curation, Data Management, and Data Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Analysis Systems, Methods, and Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Enterprise Integration and Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
CHAPTER 5
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
vii
Figures and Tables
Figures
2.1. Data-Informed Business Practices of the USAF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2. Essential Elements of Technology-Enabled Business Practices of the USAF . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.1. Technology-Enabled Talent Management System Encompasses All Stages of HR Functions . . . . . . 11
3.2. Example of Technology-Enabled Recruiting Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.3. Results of Applying Prescriptive Assignment to Each Outcome Measure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.4. Technology-Enabled Assignment Marketplace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.5. Overview of U.S. Air Force Personnel Policy Simulation Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.6. Technology-Enabled Initial Skills Training Classification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Tables
1.1. IBM Implemented AI-Enabled Talent Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3.1. Technology-Enabled Talent Management Concepts and Research Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
4.1. Charting the Growth of Technology-Enabled Talent Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
ix
CHAPTER 1
Introduction
Like corporations across the globe, the U.S. Air Force (USAF) is exploring the benefits of moving away from
a traditional human resources (HR) system that emphasizes standardization and process efficiency toward
a more-responsive and more-personalized system empowered by data, information technologies (IT), and
artificial intelligence (AI) (Wright et al., 2020). Although the USAF has requirements, missions, and policy
structures that make it difficult to fully leverage emerging technologies for human resource management
(HRM), there are many important uses for new capabilities in talent management in the service.
This report—one in a set aimed at helping the USAF understand the elements necessary for success in
transforming IT and business systems for HRM1—concerns the elements necessary for success in pursu-
ing greater adoption of data technologies in talent management. Technologies exist on a spectrum, and the
USAF, like all other organizations, already relies on some technologies to perform HRM functions. However,
by pursuing the latest technological advances, the USAF can continue to improve the efficiency and effective-
ness of HRM processes.
After providing a framework for understanding the distinctive features of these new technologies (which
we term technology-enabled practices), we describe several applied use cases from recent research, followed
by a discussion of potential barriers to implementation. We focus principally on the use of AI, which depends
on data and supporting IT. The companion reports to this one examine best practices for implementation
and address strategies for securing sufficient funding and resources for digital transformation (Snyder, 2022;
Yeung et al., 2022).
In recent years, advances in data management, IT, and AI have transformed talent management and
other business practices (Cappelli, Tambe, and Yakubovich, 2018). For example, 50 percent of respondents
to the 2020 McKinsey Global Survey of businesses reported that their companies had adopted AI in at least
one business function (McKinsey Analytics, 2020, p. 2).2 About 10 percent adopted AI for “optimization of
talent management” (such as recruiting and retention), and about 7 percent for “performance management”
(McKinsey Analytics, 2020, p. 3). Many of the applications were apparently highly effective: Among the
companies that adopted AI for talent management, more than one-half reported that doing so had increased
revenue or decreased cost. In the USAF context, the effectiveness of such applications goes beyond efficiency
and includes whether the applications are aligned with organizational values and the rule of law. (For a more
detailed discussion of these issues, see Appendix C of National Security Commission on Artificial Intelli-
gence, 2021.)
The USAF and other military services have a long history of innovation in applications of (what is now
called) data science to talent management problems. For instance, in the 1960s and 1970s, U.S. Department of
Defense (DoD) research pioneered the Armed Services Vocational Aptitude Battery (ASVAB) to solve the dif-
ficult problem of how to predict recruit success in one of the hundreds of potential occupations in which they
1 Note that these closely related volumes share some material, such as descriptions of USAF priorities, study approach, and
private-sector and government-technology landscapes.
2 A sample of 2,395 organizations participated in an online survey in 2020 (McKinsey Analytics, 2020, p. 13).
1
How the U.S. Air Force Can Incorporate New Data Technologies into Its Talent Management System
might serve (Welsh, Kucinkas, and Curran, 1990). In the early 1990s, the USAF developed its own custom
test battery for pilot selection (Carretta, 2011).
How could the industry-hyped data technologies of recent years add value to a USAF system that already
has a set of customized quantitative methods for HRM and, thus, is highly attuned to the power of data?
Taking advantage of new technology-enabled techniques requires a shift in thinking about AI applications
to business practices. When discussing applications of AI to DoD missions, strategy documents emphasize
that AI will generate transformational capabilities that contribute to military advantage. On the business
side, however, the same strategy documents seem to emphasize that AI will enable incremental gains in the
efficiency of processes without fundamentally restructuring them. Consider this quote from the DoD AI
strategy (DoD, 2019, p. 6):
The ability of AI to reduce inefficiencies from manual, laborious, data-centric tasks will be harnessed
across the Department with the objective of simplifying workflows and improving the speed and accuracy
of repetitive tasks.
A technology-enabled talent management system is not primarily one in which business processes operate
more efficiently through automation; rather, it is a system that enables altogether new business processes.3
The functions of these new business processes may remain the same—recruiting, training, promotion, and
retention—but the way they are performed may be transformed. In this way, technology-enabled talent man-
agement fulfills the vision articulated by Brose (2020, p. 6):
The question is not how new technologies can improve the US military’s ability to do the same things it has
done for decades but rather how these technologies can enable us to do entirely different things—to build
new kinds of military forces and operate them in new ways.
A concerted approach to technology-enabled talent management coupled with an openness to think

differently about HRM could open opportunities for performance gains in many areas, such as those that
already use data extensively in decisionmaking.
Technology-Enabled Talent Management Becoming a Commercial

Reality
As companies apply AI to business functions for the first time, advances in technology and analytics are
transforming all phases of talent management. O’Shea and Puente (2017) reported that talent management
technologies are moving away from single solutions for isolated HR functions, and toward end-to-end sys-
tems that encompass all phases of talent management, from recruiting to separation.4 “These more com-
prehensive solutions help promote a unified, holistic view of talent management within organizations and
reduce the potential incompatibilities and redundancies inherent in the use of separate solutions for each
talent management function” (O’Shea and Puente, 2017, p. 551). The authors also point out that these systems
provide a “rich source of data [that] can be used to build statistical workforce-planning models and even
3 Improving and integrating Air Force personnel enterprise resource management systems may produce cost savings and
temporal efficiencies. However, this report focuses on the transformative potential of technology-enabled talent management
rather than the evolutionary gains that greater automation would allow.
4 The same is true of enterprisewide information systems for non-HR functions in industry.
2
Introduction
recast traditionally episodic activities, such as assessment validation, into dynamic processes that can be
evaluated over time” (O’Shea and Puente, 2017, p. 551; also see Oswald et al., 2020).
IBM is a case study in the application of technology-enabled business processes to all phases of talent
management (Table 1.1; also see Guenole and Feinzig, 2018). IBM’s HR function was among the first to
adopt AI technology, and its use cases now span myriad HR functions. For example, chatbots are used to
direct potential applicants to position openings, machine learning (ML) uses information gathered during
the application process to predict performance of applicants, and AI assistants are used to deliver personal-
ized training and career coaching. IBM reported that these applications netted not only positive HR benefits
but also corporate savings of $107 million in 2017 alone (Guenole and Feinzig, 2018).
TABLE 1.1
IBM Implemented AI-Enabled Talent Management
HR Function Use Case
Attract Chatbots that use natural language processing to answer job seekers’ frequently asked questions about the
company and to recommend relevant position openings.
Hire Algorithms that (1) determine the match between an applicant’s résumé and the job requirements and
(2) predict future performance using information collected about the applicant during the application process.
Engage Automated audits and alerts that nudge managers to act when appropriate. For example, a manager might be
alerted that an employee has acquired the skills and experience necessary to be promoted.
Retain Algorithms that use employee data and economic conditions to suggest competitive compensation packages.
Develop Algorithms that (1) tag and index content in large corporate learning management systems and (2) track
individual needs to personalize the training that is delivered.
Grow An AI assistant that interacts with employees to shape career trajectories. Career coaching has traditionally
been costly and time intensive, so it has historically been reserved for a limited number of people.
Serve Intelligent assistants that guide employees through benefit enrollment decisions or performance management
tools or that help employees navigate their organization by identifying the right point of contact for an inquiry.
SOURCE: Guenole and Feinzig, 2018.
3
CHAPTER 2
How the USAF Uses Data for Human Resources

The USAF has a long history of data-informed talent management. To meet human capital needs each year,
the USAF must recruit tens of thousands of individuals, train and equip them to perform hundreds of occu-
pationally specialized tasks, and retain them at the right levels to meet operational requirements while sat-
isfying cost constraints. It would be impossible for the USAF to execute these talent management functions
without making data-informed decisions.
Data-informed decisionmaking involves gathering and analyzing data on a specific case and using that
information to make decisions about that case. In contrast, technology-enabled business practices continu-
ally gather and fuse data from diverse sources and use the information to adjust ongoing HR processes.
Data sources are traditional ones, such as administrative databases (a staple of the USAF HR manage-
ment), and nontraditional sources, such as data that users generate while interacting with IT systems
(sometimes termed digital exhaust).
Figure 2.1 depicts how data-informed business processes support ongoing HR decisions in the USAF.
Certain policy questions are revisited at regular intervals. Data are gathered and analyzed, and the results
are used to inform recommendations and update policies. One example of a data-informed process is the
USAF’s approach to compensation, as illustrated in the selective retention bonus program. DoD authorizes
the services to determine bonus eligibility criteria with the goal of shaping retention toward their skill and
experience needs (DoD Instruction 1304.31). The USAF Directorate of Force Management Policy evaluates
each enlisted specialty at least once per year to determine the appropriate level of bonus eligibility, factoring
in data inputs on projected shortages; training costs; and research on the relationships among compensation,
demand for the military skill in the civilian labor market, and retention (Air Force Instruction 36-2606). The
data inputs and analysis inform recommendations on updated bonus offerings, and those new offerings are
published in the form of new policies in an annual announcement.1
FIGURE 2.1
Data-Informed Business Practices of the USAF
Policy questions Data Analysis Recommendations New policies
? √
1 Other policy questions regarding compensation are addressed as the need arises. These questions may require different
types of expertise, data sources, and analytic approaches. For example, the USAF has invested in research programs designed
to help manage the pilot force (Robbert et al., 2015; Mattock et al., 2016), bringing them to the point where they can advocate
appropriations that are based on cost-benefit analyses of different approaches to pilot production and compensation (Mat-
tock et al., 2019). The USAF relies on internal offices (such as the Studies, Analysis and Assessments Directorate [AF/A9]) and
external organizations (such as federally funded research and development centers, academia, and industry) to address these
types of questions in a data-informed way.
5
Technology-Enabled Business Practices: The Future of USAF Talent

Management
How would a data-enabled approach to compensation differ from existing USAF processes that draw heav-
ily on data and models? In a recent report on this topic (Guenole and Feinzig, 2018), researchers at IBM
described adoption of an AI decision-support system that provides managers with customized recommen-
dations that are based on the market demand for skills and the organizational costs of replacement. The
technology-enabled approach at IBM differs from the selective retention bonus process in that it uses AI
to generate recommendations that are customized to individuals while continually learning from manager
decisions to override those recommendations. The system also promotes transparency, and possibly reten-
tion, by providing the same information to employees so that they can see how their compensation relates
to their market value.
Although compensation in the USAF is not set by frontline managers, the contrast illustrates that transi-
tioning from data-informed decisionmaking to technology-enabled business practices allows for customiza-
tion at a large scale, with continual learning, to potentially replace HRM practices that have historically been
standardized. Figure 2.2 depicts a general mockup of how data-enabled business practices could support
ongoing HR decisions in a relatively short time. These business practices do not simply use “big data” or “AI.”
Rather, they depend on an integrated architecture that brings together data, machine intelligence, and
human intelligence to improve decisions and outcomes.
Leveraging New Types of Data to Enhance Decisions

Data, the essential fuel of technology-enabled business practices, encompass all potential sources of informa-
tion to improve decisionmaking. The data can take many forms, such as text in documents or emails, videos
FIGURE 2.2
Essential Elements of Technology-Enabled Business Practices of the USAF
Learn from decisions
Enhance decisions
Machine intelligence
Data Decisions Outcomes
0110
1011 ?
0010
Human intelligence
Inform decisions
6
or images posted to social media, audio recordings, readings from physiological sensors, and geolocation
data from phones and satellites. Vast and continual data streams produce an unprecedented volume of data
that can be used to inform HR processes.
If accurate predictions are the essential decision input, technology-enabled practices do not commit to a
theoretical model that links certain variables, measured using specific data sources, to HR outcomes (Putka,
Beatty, and Reeder, 2018). Rather, all available data are synthesized without specifying in advance how each
element relates to a theoretical model, and those data are used to predict key HR outcomes. The term big data
was coined to capture the volume, variety, and velocity of data accumulated and used in business practices
(Diebold, 2021). In other applications, as our use cases show, such unstructured predictions can be combined
with a structural representation of the HR mechanism or system to generate new insights.
Leveraging New Forms of Machine Intelligence to Enhance Decisions

If data are the fuel for technology-enabled practices, machine intelligence is the engine. Specifically, machine
intelligence describes the use of analytical algorithms to extract information from the data. Machine intel-
ligence consists of traditional methods (such as operations research models or econometric techniques); AI;
and ML, which is a subset of AI algorithms.2 Chui, Kamalnath, and McCarthy (2020) defines ML as
a collection of algorithms that detect patterns and learn how to make predictions and recommendations
by processing data and experiences, rather than by receiving explicit programming instruction. The algo-
rithms also adapt in response to new data and experiences to improve efficacy over time.
ML differs from traditional analytic approaches because it seeks to maximize the accuracy of predic-
tions without manually coding or structuring the necessary relationships between input variables and HR
outcomes. This allows the structure of ML models to grow increasingly complex if the complexity improves
performance (Mullainathan and Spiess, 2017, pp. 87–88). Critically, when the volume and complexity of
data exceed a human’s processing capacity, ML can discover structure in the data to help improve deci-
sions and outcomes.
Introducing Interactivity Among Machine Intelligence, Human Intelligence, Data,

and Decisions
If data are the fuel and machine intelligence is the engine of a technology-enabled business practice, human
intelligence is the driver. As stated earlier, the purpose of machine intelligence is to extract information
from data and to provide recommendations to decisionmakers. In complex real-world scenarios, the
human decisionmaker may have information about context, constraints, and objectives that the machine
does not, so the interaction between ML and the human is essential to ensure that decisions are robust.
There are three distinctive feedback loops between human and machine intelligence and other elements
of technology-enabled business practices (Figure 2.2). The first feedback loop is from human and machine
intelligence to data flowing into the system. Decisionmakers shape this flow via data collection techniques,
data capture and storage, and the engineering of data features that may inform decisionmaking. Machine
intelligence can also shape the data flow into the system. For example, adaptive sampling techniques, which
2 ML refers to the collection of learning models (Alpaydin, 2016), whereas AI refers to a broader concept “defined as the
ability of a machine to perform cognitive functions we associate with human minds, such as perceiving, reasoning, learning,
interacting with the environment, problem solving, and even exercising creativity” (Chui, Kamalnath, and McCarthy, 2020).
7
underlie computer adaptive testing and adaptive design optimization, can be used to determine what addi-
tional data to collect to maximally disambiguate a decision (Myung, Cavagnaro, and Pitt, 2013).
The second feedback loop is between human intelligence and machine intelligence. Humans and machines
interact in a variety of ways. On the most basic level, humans must select modeling techniques and identify
business objectives for the machine to optimize.3 In addition, humans may provide feedback on the quality of
the machine’s predictions or prescriptions so that the algorithms can learn from experience. If the outputs of
the machine intelligence are delivered in an explainable manner, they may give the human new insights into
the underlying decision process (Arrieta et al., 2020).
The interactivity between human and machine intelligence also addresses important concerns about bias
and errors (Osoba et al., 2019; Osoba and Welser, 2017; Tambe, Cappelli, and Yakubovich, 2019). Humans
and ML are both fallible, but they fail in different ways. In properly executed technology-enabled business
practices, human and machine intelligence check one another. Although recent research has emphasized
the concern over AI bias, algorithms can also be an effective countermeasure for human bias (Kahneman
et al., 2016), which means that there are risks associated both with inaction and adoption. For these reasons,
the National Security Commission on Artificial Intelligence recommended that the design of such systems
should provide an explicit analysis of outcomes that would violate American values and that design efforts
should consider explicitly incorporating value considerations into the objectives of the system (National
Security Commission on Artificial Intelligence, 2021).
The third and final feedback loop is from business outcomes arising from the system to improved human
and machine understanding of the decision space. In some sense, all research uses past data to explore les-
sons learned from decision outcomes. But not all policies are informed by research, and almost none are con-
tinually informed. The result is that policy practices become entrenched—they outlive the environment that
justified the initial implementation.
The technology-enabled framework addresses this problem by supporting continual calibration and feed-
back to refine ongoing talent management decisions. By continually tracing granular decisions and outcomes,
technology-enabled business practices allow decisionmakers to react more efficiently and effectively when cir-
cumstances change. Furthermore, technology-enabled business practices may incorporate constant experimen-
tation (or A/B testing) to seek out changes that will produce better outcomes (Kohavi and Longbotham, 2017;
Siroker and Koomen, 2013). Major firms, especially in the technology sector, routinely run A/B tests to discover
effective changes (Luca and Bazerman, 2020).4
Experimentation entails some risk and must be used carefully, given the consequential nature of HR out-
comes. Yet failing to adapt also entails risk. Interactivity between outcomes and human and machine intel-
ligence reduces this risk in technology-enabled business practices.
USAF Well Positioned to Join These Organizations

All firms face challenges incorporating new technologies into talent management processes. These chal-
lenges relate to the complexity and measurement difficulty inherent in HR outcomes (such as which attri-
butes define a good employee), limitations on the availability of data, the fairness and transparency in algo-
rithmic decision systems, and the reactions of affected employees (Tambe, Cappelli, and Yakubovich, 2019).
3 Most ML applications are examples of narrow AI, in that ML is applied to part of the task defined by a human.
4 IBM used this approach to determine whether an AI chatbot was more successful than the traditional process at convert-
ing interested people into applicants (Guenole and Feinzig, 2018). Google has also done extensive experimentation in the HR
domain (Tambe, Cappelli, and Yakubovich, 2019). Finally, economists and consultants have used experimentation to test
whether new management practices improve productivity (Bloom et al., 2013).
8
Although the USAF faces these same challenges, certain design elements of the historical personnel manage-
ment system put the USAF in a good position to move forward with technology-enabled talent management.
These elements fall into four areas:
• Standardization and stability: All USAF members fall under a standard management system, in which
each aspect of the system that contains relevant information for decisionmaking—such as the occupa-
tional classification scheme, organizational structure, job titles, career development patterns, personnel
skills inventories, and even compensation policies—is defined, documented, and relatively stable over
time. For example, officers are considered for promotion after serving for a predetermined number of
years at each rank, and the percentage selected for promotion is roughly constant from year to year.
This structure makes the outcomes of the system much more predictable and increases the prospects
for high-fidelity decision-support tools.
• Availability of career histories: The unique mission and demands of military service mean that the
USAF must grow and develop talent from within. The benefit of this constraint for talent management
is that the USAF has access to complete and ongoing career histories for all members. Demographic and
aptitude data are collected about individuals starting with their first meeting with a recruiter; medical
and performance data are collected as they advance through the training pipeline; career data are col-
lected during each assignment; and performance reports are submitted on a routine basis. Provided that
the USAF can capture and connect the information from all nodes of the HR system, this rich longitu-
dinal data holds promise for technology-enabled talent management.
• Measurable outcomes: It is not easy to measure what constitutes a good employee (Tambe, Cap-
pelli, and Yakubovich, 2019). A potential advantage for the USAF in developing its data-enabled
talent management system is that it has a standardized performance management system that uses a
common language with discernible performance signals (Schulker et al., 2020). These outcome mea-
sures create the potential for training models to optimize upstream HR processes (such as recruiting)
to drive better outcomes.
• Quantity of data: Often, the size of an organization can limit its ability to apply data-enabled practices,
because small firms may not perform enough HR actions to create the data needed to train ML models
(Tambe, Cappelli, and Yakubovich, 2019). The USAF consists of over 300,000 active-duty airmen.
Individual-level data are available about the current force and the historical force over several decades,
comprising billions of observations. Even in the case of relatively rare outcomes (such as conduct or
legal infractions) or relatively small personnel categories (such as female pilots at a certain rank), this
amounts to thousands of observations for training ML models.
To understand why these natural advantages position the USAF to use new technologies for talent man-
agement, consider the example of skill validation, which touches on multiple elements discussed earlier. All
firms need to validate the documented skills and expertise of external job applicants—whose performance
they have not yet observed—so that they can prioritize scarce HR resources for the most-promising appli-
cant. Thus, innovative efforts continue to produce new approaches to derive this information from social
networks (Yan et al., 2019). The USAF does not generally have this problem, however, because its organiza-
tional elements develop nearly all occupational training content so that the USAF can train members accord-
ing to its skill standards. Thus, most skills in the USAF HR system are prevalidated, and, in the long run, the
USAF will have access to every training event, academic transcript, job experience, and performance evalu-
ation each member has ever had. To fully capitalize on these advantages, the USAF must improve capabili-
ties for collecting, managing, and operationalizing data (National Academies of Sciences, Engineering, and
Medicine, 2020).
9
These advantages apply to the active component of the USAF and, to a lesser degree, to the reserve and
civilian components. For instance, all active-duty component developmental experiences are captured by
USAF personnel data systems, but part-time reservists and civilians have work experiences with other
firms that are not recorded. Civilian and reserve career paths, performance measurement practices, and
compensation policies also tend to be less standardized than those of the active-duty component. None-
theless, technology-enabled talent management practices are still applicable to the reserve and civilian
components of the USAF.
10
CHAPTER 3
Use Cases for Technology-Enabled

Talent Management in USAF
It is somewhat misleading to speak holistically of a technology-enabled talent management system because
the reality we are describing is an iterative series of incremental improvements. Bringing quality technology-
enabled decision-support to areas where none exist could be transformational, but improvements in areas
that are already data-informed and highly optimized could be more incremental. For example, one of our
use cases involves the officer assignment system. In the past, matching officers to new assignments has relied
on the judgment of officer assignment teams without formal data capture on features that relate to match
quality and subsequent business outcomes. This major HRM process is potentially ripe for a redesign in the
technology-enabled mold (and such a redesign is underway). By contrast, the enlisted assignment system is
already continually informed by data and algorithms. Improvements in those assignments might be possible
by enriching the data and envisioning new ways to learn from outcomes, but the potential performance gains
are less obvious. Therefore, embracing technology-enabled practices involves exploring use cases according
to perceived business value, testing for feasibility, and measuring the impact of new processes (Guenole and
Feinzig, 2018).
Much of this work has already begun. In the past several years, the USAF and other DoD services have
sponsored a body of research exploring technological applications for each HR function shown in Figure 3.1.
This effort established the technical feasibility of technology-enabled talent management concepts in most
areas of HR. Collectively, these applications show that it is possible to improve the allocation of monetary
and human capital resources, to increase the efficiency of HR practices, and to enhance the quality of HR
decisions and outcomes.
Table 3.1 summarizes the research at a high level. The columns of Table 3.1 show that prior research has
explored technology-enabled concepts in most HR functions. The rows of Table 3.1 show components of
technology-enabled talent management. Cells shaded in solid blue indicate that technology-enabled talent
FIGURE 3.1
Technology-Enabled Talent Management System Encompasses All Stages of HR Functions
Classification
Recruiting and training Assignment Development Promotion Retention
√
$
√ √
√
√ Allocate resources
√ Improve efficiency
√ Aid and evaluate decisions
11
TABLE 3.1
Technology-Enabled Talent Management Concepts and Research Results
Data-Enabled
Recruiting Recruiting Specialty
Talent Management
(tactical) (strategic) Classification Development Assignment Promotion Retention
Concepts
Use of diverse and
nontraditional data
sources
Use of
machine intelligence
Human and machine

interactivity with data
collection
Human and machine
interactivity for
decisionmaking
Human and machine
interactivity with
outcomes
Possible Demonstrated
management concepts have been demonstrated in DoD; cells shaded in striped blue indicate that RAND
researchers have examined the concepts and determined that they are potentially applicable. The key take-
away is that most concepts of technology-enabled talent management have been demonstrated in at least one
HR area within DoD, and there are potential applications of all concepts to each of the HR functions that we
have examined.
However, Table 3.1 also highlights that there is relatively little demonstration support for two of the
technology-enabled talent management concepts: human-machine interactivity with data collection and
human-machine interactivity with outcomes. These forms of interaction differentiate static analyses,
which resemble the historical data-informed paradigm, from technology-enabled business practices. If
the USAF lacks the ability to adjust the data flowing into the system, its performance will likely plateau.
Similarly, if the USAF cannot use outcomes to adjust ML models and decision processes, the performance
of the system will likely plateau.1 In short, there have been many successful demonstrations of technology-
enabled talent management in the USAF, but the USAF will be limited unless it can implement these
remaining concepts.
In the following sections, we provide vignettes explaining the talent management challenge for each area
of HR, and then describe the exploratory results developed by various divisions of the RAND Corporation.
1 A classic example of this problem occurs with applicant screening rules, such as minimum ASVAB scores. Cognitively
demanding specialties have high ASVAB requirements, reflecting the need for high aptitude to succeed in these fields. How-
ever, once policy sets a minimum score for screening, the subsequent relationship between test scores and success could be
weak, or even negative. It would be wrong to conclude that this finding means the test is no longer a good predictor of success;
rather, it occurs because the only opportunity to observe success after implementing the screening policy is among a select
group of high-aptitude recruits. Experimentation addresses this problem by occasionally varying the minimum score to pro-
duce information on its continued effectiveness as a screening mechanism.
12
Use Cases for Technology-Enabled Talent Management in USAF
Tactical Recruiting Challenge: Helping Recruiters Manage Their Time

The USAF recruiting enterprise must generate roughly 30,000 new recruits each year, and each recruit must
meet an array of qualifications, such as health and physical fitness standards and minimum aptitude scores.
The enterprise must accomplish this task while adapting to economic conditions and the seasonality that is
inherent in recruiting primarily high school graduates. Historically, the USAF has met its recruiting goals
with a footprint of roughly 1,200 full-time recruiters with budgets for enlistment bonuses, advertising, and
other activities.
The USAF tasks individual recruiters with monthly goals for contracts, and recruiters then interview, per-
suade, and shepherd interested applicants, or leads, through the enlistment process with the hope of meeting
monthly targets. Recruiters have expressed difficulty managing their workload (Knutson, 2019), potentially
placing some at risk for burnout. Given that every locale is unique and there are a variety of potential sources
for recruiting leads, it is unclear how recruiters should prioritize their limited time to focus on the most-
promising applicants.
RAND Research Shows How Technology-Enabled Recruiting Could Improve

Recruiter Efficiency
The Office of Accession Policy within the Office of the Undersecretary of Defense for Personnel and Readi-
ness asked researchers at RAND’s National Defense Research Institute to assemble an advisory group of
knowledgeable people from DoD recruiting organizations, marketing agencies, and the DoD Joint Advertis-
ing Market Research & Studies organization. The RAND team combined those perspectives with informa-
tion from the business and academic domains on the use of technology to support marketing and recruiting
(Lim, Orvis, and Hall, 2019).
In the conceptual technology-enabled recruiting process shown in Figure 3.2, technology enhances a
recruiter’s effectiveness by enriching the records of applicants with supplementary data gathered from such
brokers as Acxiom, Experian, Equifax, and TransUnion, then scoring those augmented records based on
FIGURE 3.2
Example of Technology-Enabled Recruiting Process
Recruiting as a leads funnel

• Thinking about recruiting as a marketer would
• Industry practices would take all available information into account to help recruiters prioritize the most-promising leads
Online job order

Salespersons
Candidate browsing Automate, learn, and predict over time and managers
Lead driver
Enrich Score Route Assign Execute Job offer
Email open
Service-level
Posting agreement reassign
Campaigns
Closed-loop marketing optimization
SOURCE: Lim, Orvis, and Hall, 2019.

NOTE: Although not shown in this figure, downstream outcomes, such as success in training or job performance, would inform the scoring of
recruiting leads.
13
applicant likelihood of accepting an offer or succeeding in the military. Using enriched data and scores, the
algorithm routes promising applicants to the recruiters who are most likely to persuade those individuals. All
these steps are upstream from recruiter actions. In this way, the system helps recruiters to spend their limited
time attempting to attract the best prospects.
Feedback loops in this technology-enabled process provide a layer of interactivity so that the automated
steps improve over time. Thus, a technology-enabled process has the potential to make individual recruiters
more productive, which could lead to resource savings if fewer recruiters are needed. A technology-enabled
process also has the potential to improve the quality of accessions (i.e., individuals entering military service)
if recruiters can focus on more-qualified prospects. Finally, a technology-enabled process would help the
USAF steer the lead-refinement process and allow recruiters to focus on other high-priority attributes that
are difficult to prospect for, such as geographic or ethnic diversity. The conceptual model shown in Figure 2.2
has not yet been implemented for Air Force recruiting, but it provides a characterization of how technology-
enabled talent management may be applied to this HR function.
Strategic Recruiting Challenge: Providing Enough Resources for

Recruiters
Because of the USAF’s size and its need for a continual stream of new recruits, planners also face strate-
gic decisions about how to posture the recruiting enterprise. Planners must seek Operation and Mainte-
nance funding to purchase advertising and support local events (such as airshows or sponsorships), Military
Personnel appropriations that fund enlistment bonuses, and even programming support for the number of
recruiter positions. The most credible case for these resource requests will always be one that is backed by
evidence, which can also be provided through technology-enabled methods.
RAND Built a Model That Optimizes Recruiting Resource Levels

Researchers in RAND’s Army Research Division developed a capability, termed the Recruiting Resource
Model (RRM), that allows planners to translate resource plans (such as numbers of recruiters or funding for
advertising) in each environment into expected contracts, accessions, and costs (Knapp et al., 2018). Plan-
ners can use the RRM to observe the expected outputs of different planning scenarios, or they can allow
the model’s optimization routine to adjust resource levels until it finds the most cost-effective bundle that
meets recruiting targets. A key finding from this work was that the Army could reach its high accession tar-
gets (typically double the level needed in the USAF) at a significantly lower cost by shifting resources from
bonuses to television advertising (Knapp et al., 2018).
Specialty Classification Challenge: Matching Airmen to the Right Job

Each year, thousands of people enlist in the USAF and are assigned to hundreds of occupational special-
ties, denoted by Air Force Specialty Codes (AFSCs). To assign individuals to AFSCs, the USAF engages in
activities beginning at recruiting and extending through basic military training. These activities and the
corresponding programs and policies all compose the enlisted classification system (Air Force Instruction
36-2101). The existing system is an example of a data-informed process that uses past research to establish
minimum standards for each AFSC to ensure an acceptable level of training attrition. Some qualified recruits
enter with a job guarantee in their contract; others can choose among a limited set of available specialties in
basic military training.
14
Although most airmen successfully complete initial skills training (IST), about 10 percent are elimi-
nated because of performance deficiencies and for other reasons. These individuals are either separated
from the USAF or reclassified into different AFSCs. Qualitative research findings suggest that airmen
sometimes receive AFSCs without necessarily knowing what the specialty entails or possessing attributes
required for success (Robson et al., 2022). These eliminations might have been preventable if the USAF
had a more accurate process for matching the airmen to specialties. Even for the 90 percent of airmen who
complete IST, the initial specialty they receive continues to affect subsequent job performance, first-term
completion, and reenlistment.
RAND Used Data Science Techniques to Match Airmen to Promising Specialties

Researchers at RAND Project AIR FORCE (PAF) explored the use of data science techniques to match airmen
to specialties to optimize four training and early career outcomes: (1) IST graduation, (2) promotion to E-5
in the first term, (3) first-term completion, and (4) reenlistment. The RAND team identified more than 50
predictors spanning seven categories that are available when individuals first enlist, and they explored differ-
ent ML techniques to achieve the best possible predictive performance for each metric (Robson et al., 2022).2
Once the RAND team fitted predictive models to the four training and early career outcomes, they used
the models to prescriptively assign individuals to specialties to maximize the four outcomes. The results
for each outcome are shown in Figure 3.3. Each point is an individual airman. The axis labeled “actual
FIGURE 3.3
Results of Applying Prescriptive Assignment to Each Outcome Measure
Optimal pipeline
Probability of completing IST Probability of completing term

96% 98% 67% 72%
Optimal pipeline
Probability of reenlistment Probability of promotion

47% 54% 31% 36%
Actual pipeline Actual pipeline
Increase Decrease Same
SOURCE: Robson et al., 2022.
2 Predictor categories were enlistment contract, demographics, cognitive aptitude, career field preference, education, and
physical and medical fitness. RAND PAF compared several ML algorithms. An approach called Bayesian Additive Regres-
sion Trees (BART) did best at predicting the four outcomes. BART learns sequences of yes-or-no decision rules, which form
decision trees that can be used to predict outcomes. Rather than using just one decision tree, however, BART combines the
predictions from an ensemble of trees. A Bayesian procedure is used to reduce the complexity of decision rules.
15
pipeline” shows the predicted probability of the outcome for each airman given the AFSC that they were
assigned. The axis labeled “optimal pipeline” shows the predicted probability of the outcome if all airmen
were prescriptively assigned. The analysis shows that prescriptive assignments could increase the prob-
abilities of positive training and career outcomes for the average airmen by 2 to 7 percentage points. These
gains are substantial when one considers the population size at stake. A “back-of-the-envelope” calculation
that applies the improvement of 5 percentage points in first-term completion to a new cohort of 25,000
trained recruits would suggest that prescriptive assignments could result in 1,250 fewer early separations.
Development Challenge: Labor-Intensive Board Selection Processes

Developing military officers to be ready to command at ever-higher levels is a 15- to 30-year process. The
monumental HR task that the development system must accomplish is to fill all critical positions with quali-
fied personnel while at the same time curating positions for individuals to meet developmental goals for the
future. To manage the complexity of this problem, the development and assignment systems view officers
through two lenses. The macro lens takes a very general view of the types of broadening experiences that
members should have by using rough categories (such as a unit-level assignment versus a staff assignment).
Assignment officers then look at each officer’s experiences and seek to balance their résumé with each new
assignment, if possible, depending on available opportunities. As officers become more senior and the num-
bers of officers become more manageable, development policy shifts toward viewing officers through a micro
lens; a board of experienced leaders reviews records by hand and determines whom to select for profes-
sional military education, command, and promotion to the next grade level. The micro lens draws on higher-
fidelity information on officer experiences and performance, but the processes are labor intensive and used
sparingly. Moreover, the micro lens still depends on subjective human judgments and uses only a subset of
information available about a given individual.
RAND Built an ML Model That Automatically Finds Officers with Good

Performance Indicators
Data science methods offer the opportunity to build a system that automates the micro lens and potentially
applies it to development areas where the volume of records is too large for manual review. Researchers at
RAND PAF examined the feasibility of such a system by extracting a large sample of officer records that had
previously been through the micro lens process by O-5 and O-6 promotion boards (Schulker et al., 2020). The
RAND team then used an ML model to learn which patterns of text in officers’ performance reports were
most closely associated with higher levels of development (as measured by the O-5 and O-6 promotion board
decisions).3 The results showed that the ML model could account for promotion outcomes based on the text-
based performance reports, and they revealed that the model had keyed in on words and phrases that any
experienced officer would recognize as important performance signals. The fact that the ML model parsed
the language of performance writing without being told which words and phrases to look for is promising
because the same techniques can adapt as the content and structure of evaluations evolve over time. Impor-
3 The application involves applying bag-of-words techniques to convert free-text data into a record of the words that make up
a performance report. RAND PAF compared several ML algorithms for predicting promotion board decisions based on words
contained in performance reports. An approach called a support vector machine (SVM) did best at predicting outcomes. SVM
finds a boundary that separates records with each of the two outcomes—promote versus non-promote—based on the words
contained in the corresponding performance reports.
16
tantly, in this and other applications, the ML model could provide a recommendation or input to a human
decisionmaker, so outcomes would not be fully determined by the model.
Assignments Challenge: How to Best Use the New “Talent Marketplace”

The officer assignment system is at the center of many other HR functions. The system matches officers to
each successive opportunity that will shape their careers and determine their competitiveness for senior lead-
ership while indirectly affecting retention behaviors (Keller et al., 2018). In recent years, the USAF has shifted
to an assignments approach, dubbed the “Talent Marketplace,” that mimics a job market. In the new system,
individuals can apply for positions and coordinate with job owners directly to inform them of special skills
and/or to obtain a realistic preview of the job environment. The Talent Marketplace concept is ideal when
jobs and locations must change on a regular schedule (as for assignments of O-5 and below), when a person’s
fit in a job (or person-job fit) is paramount to many human capital goals, and when the need to gather infor-
mation required to accurately discern person-job fit exceeds the capacity of assignment officers.
RAND Found New Opportunities for Information Exchange

Researchers at RAND PAF analyzed new opportunities for improved decisionmaking made possible through
the Talent Marketplace system.4 The reason that the marketplace concept could deliver a better match in
each assignment cycle is that it has the potential to generate more information about person-job fit than the
old system (while relying on decentralized execution to process the new information). The implementation
of the Talent Marketplace is still primitive in that it lacks necessary avenues for applicants and job owners
to exchange information (National Academies of Sciences, Engineering, and Medicine, 2020). But a fully
formed marketplace does provide a useful framework for the assignment arm of a technology-enabled talent
management system (Figure 3.4). In such a system, the following occurs:
• Officers contribute a more detailed résumé to highlight relevant attributes to job owners that might
increase their likelihood of getting a preferred position.
• The system captures those preferences and tracks job satisfaction and post-assignment outcomes (such
as performance in the position).
• Job owners, in turn, provide detailed information about the advertised positions and preferred qualifi-
cations while ranking applicants and evaluating subsequent performance on the job.
• Assignment teams continue to provide developmental constraints and prioritize operational needs.
In the longer term, interactions between human and machine intelligence could become embedded in the
system through the feedback loops. Historical satisfaction and performance data would inform recommen-
dation engines for assignments that future officers should consider, and job owner rankings would provide
feedback on what officers must improve to be competitive for their preferred positions. Mirror images of
these feedback loops could also become available to job owners. Historical performance data can feed rec-
ommendation engines to suggest the most-promising applicants, and feedback from applicants can help job
owners address negative characteristics of the work environment to better compete for talent. At the strategic
level, this same information can help assignment teams understand how to steer person-job matches toward
improved force management outcomes for the broader organization.
4 Unpublished RAND research by David Schulker and Matthew Walsh.
17
FIGURE 3.4
Technology-Enabled Assignment Marketplace
Job owner
• Detailed job description

• Unique requirements
• Ranked candidates
• On-the-job performance
• Recommended
applicants
• Applicant feedback
• Recommended on job attributes
assignments
Data-enabled
• Development
feedback from talent marketplace
job owner
Officer Assignment team

• Detailed résumé • Needs of Air Force
• Assignment preferences • Development policy
• Satisfaction goals
• Career outcomes
• Optimal policy to boost

performance or force
management outcomes
Promotion Challenge: Unpredictable Effects of Policy Changes

The officer promotion process is somewhat unique among talent management functions in terms of the
decision stakes. Whether and when officers reach the senior field grades determines not only which officers
will command combat forces at increasing levels of responsibility but also which officers will be competitive
for selection to the general officer ranks. Furthermore, nuanced structures governing how the USAF deter-
mines promotions, such as how the process addresses differences in officer functional backgrounds, make
the effects of different policy changes on the promotion system very difficult to predict.
RAND Developed a Tool to Predict the Inventory of Officers

The USAF is making changes to how it manages officer development and promotion. As part of these
changes, the USAF has divided the Line of the Air Force—a single developmental category (DevCat) that
previously accounted for over 80 percent of officers and 40 career fields—into six separate DevCats. The
purpose is to allow officers in different career fields to pursue tailored developmental pathways while
remaining competitive for promotion. The Deputy Chief of Staff for Manpower, Personnel, and Services
asked RAND PAF to develop an inventory projection capability that presented essential features of the
promotion system. The RAND team developed a Personnel Policy Simulation Tool (PPST) (Walsh et al.,
2021), outlined in Figure 3.5, that combines smaller simulation modules, each representing mechanisms
in the personnel system, to model the flow of individuals into the officer inventory (that is, up through
the ranks after being promoted) and out of the inventory (such as after separating). Some of the modules
re-create processes described in Air Force instructions on officer promotion; other modules incorporate
ML models trained to predict fine-grained outcomes, such as promotion board decisions and individual
separation decisions (Air Force Instruction 36-2501).
18
FIGURE 3.5
Overview of U.S. Air Force Personnel Policy Simulation Tool
User inputs
ML models Grade strength Projected inventory
Structural models
End strength Promotion
Accessions Separation
Projected inventory Aging
SOURCE: Adapted from Walsh et al., 2021.
The PPST allows planners to explore the policy outcomes of changes in promotion timing and officer
functional groupings through a representation of the developmental patterns in each specialty and ML-based
retention rates learned from granular data on historical promotion outcomes.5 PPST’s holistic representa-
tion of the planning problem yielded immediate value, as researchers discovered many unforeseen “self-
correcting” properties of the system. For instance, when researchers used PPST to analyze the effects of lower
retention, the system revealed that initial decreases in retention create more vacancies, which drive higher
promotion rates to maintain grade strength. It further helped the USAF anticipate potential diversity impacts
of policies that affect career fields differently, since demographics vary greatly across career fields.
Retention Challenge: Preventing Personnel Shortages or Surpluses

Managing retention is especially critical to USAF talent management because of the requirement to grow
talent from within and because of the time and resources needed to develop individuals for technical and
leadership positions. Losses can be extremely costly, especially when the per-person training cost for certain
skill sets can approach $11 million (Mattock et al., 2019). Maintaining adequate numbers of skilled person-
nel across the diverse spectrum of military occupations is a complex, long-term planning problem that spans
many organizations within the USAF. To manage personnel shortfalls or overages, HR planners must effec-
tively use a variety of retention tools, such as service commitments, contracts, and incentives.
RAND Created a Retention Early Warning System

Given the scope of this talent management challenge, USAF planners requested an early warning system
that would alert them to areas of acute concern. Following the analogy of an early warning radar, the
5 Once again, RAND PAF compared several ML algorithms for predicting annual separation decisions. An approach called
Extreme Gradient Boosting (XGB) performed best. XGB learns an ensemble of decision trees, which are sequences of yes-or-no
decision rules used to classify outcomes. Each new tree in the ensemble reduces the residual classification error that remains
after applying all the earlier trees.
19
retention early warning system (REWS) that RAND PAF developed provides low-resolution information
about targets at a great distance—in this case, the forecasted number of individuals in a personnel category
at a future date (Schulker et al., 2021). The forecasts are based on ML models trained from historical data
about officers, enlisted personnel, and annual retention behavior. The early warning system triggers other
systems to evaluate and select responses—in this case, policy options to shape retention in the affected per-
sonnel categories. When using REWS, planners in any office can (1) select the personnel dimensions, such
as career fields or education and experience levels, that are relevant to their decisions; (2) apply ML-based
forecasts to highlight areas of concern; and (3) explore policy options to mitigate the warnings. Compared
with the standard approach used by the Air Force, which accounts only for career field and service tenure,
the ML-based approach was more sensitive to variations in other personnel categories that are associated
with retention behavior.
The Path to a Technology-Enabled Capability

The main takeaway from these case studies is that technology-enabled talent management is applicable and
achievable across the complete USAF HR life cycle (Table 3.1). However, a gap remains between conducting a
proof-of-concept study and operationalizing the concept as an HRM decision-support system. For example,
Figure 3.6 shows how the principles demonstrated in the case study on specialty classification could be inte-
grated into a business intelligence tool for recruiting and initial classification. Data about new accessions
are passed to training pipeline managers and the ML models. The ML model may suggest that the manager
gather additional information about an accession to disambiguate the ideal classification by, for example,
giving a special skills test to an accession who may be suited for a cyber career field. The manager provides
constraints related to IST seat availability, accession constraints, and other factors not otherwise considered
by the ML model. Using information about the accession and the set of constraints, the ML model recom-
FIGURE 3.6
Technology-Enabled Initial Skills Training Classification
Learn from decisions
Enhance decisions
Machine intelligence
• Career
predictions
• Optimization
Data routine Decisions Outcomes
• IST graduation
• Aptitudes • Promotion
0110
1011 • Experiences ? • IST to E-5
0010
• Physical classification • First term
and/or completion
Human intelligence
medical • Reenlistment
• IST
constraints
• Individual
constraints
• “Intangibles”
Inform decisions
20
mends a list of occupations, which the manager and recruit select from. The individual enters IST and pro-
gresses through their first enlisted term. As this happens, outcomes related to IST completion and early
career outcomes are passed back to the ML model to enable continual learning. To begin to create real value
for the USAF, the specialty classification idea, along with the others described in the case studies, must be
built out into decision-support systems that cover all the bases in the technology-enabled framework from
Figure 3.2.
The case studies illustrate several common themes about the path to technology-enabled capabilities:
• Data about USAF goals, current and prospective personnel, and environmental conditions (such as the
number of individuals in a certain personnel category that the USAF must retain, the characteristics of
individuals in that category, and economic conditions) must be automatically provided to the system.
• Decisions may be informed by gathering additional data (such as a special skills test given to a new
accession or a survey of intentions given to individuals approaching retention decisions) to address
blind spots.
• The system interface must allow the HR decisionmaker to provide additional priorities, assumptions,
and constraints (such as the amount of human and monetary resources available to meet recruiting goals
or number of training seats and length of training pipelines in different occupations) to the machine.
• The interface must also allow the machine to return predictions and recommendations (such as the
recruiting outcomes expected given different resource allocations) to the decisionmaker.
• The system must track outcomes, such as feedback from the decisionmaker, to allow ML models (such
as changes in accession, promotion, and retention rates in different career fields and demographic cat-
egories over time) to be continually updated.
• Novel data sources require new methods to extract information. For example, natural language pro-
cessing can extract information from textual data, and this information can be used to develop a richer
profile of individuals for assignment purposes or to characterize an individual’s career development
based on performance reports.
Of the applications considered, four were demonstrated as partial end-to-end system prototypes (that
is, specialty classification, development, promotion, and retention). However, as Table 3.1 shows, none
incorporated all the distinctive features of technology-enabled talent management. Still, the implication
is that, given further development and validation, the Air Force could use these systems to great effect in
the near term. The remaining applications (that is, recruiting and assignment) have been demonstrated
in industry and could be used by the Air Force in the midterm. Notwithstanding this potential, several
implementation challenges must be overcome to remain at the leading edge of technology-enabled talent
management technologies.
Implementation Challenges for a Technology-Enabled Talent

Management System
Technology-enabled business practices can already enhance the quality of decisions by providing front-line
decisionmakers with operationally relevant information in real time, and the potential of these practices will
become even greater as technology improves and ML algorithms mature. Relatively few organizations have
been able to realize the full potential of technology-enabled business practices using modern data manage-
ment, IT systems, and AI. For instance, Ransbotham et al. (2017) found that 77 percent of executives reported
that their organizations had not adopted AI-enabled business practices, and just 5 percent reported that their
organizations had “extensively incorporated” them. Similarly, Bisson et al. (2018) found that executives who
21
responded to their survey reported that “their companies are struggling to capture real value” from invest-
ments in AI.
These findings invite an obvious question: If technology-enabled practices are so beneficial, why are
broad adoption and value-generation relatively uncommon? The answer is that there are significant orga-
nizational challenges that need to be overcome before such organizations as the USAF can capitalize on
potential benefits (Fountaine, McCarthy, and Saleh, 2019). We can categorize the challenges that the USAF
will face in implementation of new technologies for talent management into three areas: policies, practices,
and technology.
Policies Can Hinder Application of New Technologies to Talent Management

We discussed earlier how technology-enabled practices feature the continual gathering and fusing of
diverse data from a variety of sources through an interactive relationship between human-machine intel-
ligence and data collection. The results in Table 3.1 illustrate that research has not tested and demonstrated
the value of this element of a technology-enabled system. The reason for this gap is that policy structures
greatly limit the flexibility of USAF data collection. In the USAF HR domain, any data pertaining to
individuals fall under the policy frameworks of official record-keeping. These policies were not designed
for technology-enabled talent management, and the USAF lacks the authority to modify these policies
because they are rooted in statutes and federal regulations. For example, Lim, Orvis, and Hall (2019)
documented several challenges associated with the use of technology-enabled outreach and recruiting
techniques in DoD. The Privacy Act of 1974 (U.S. Code, Title 5, Section 552a[4]) regulates the collection,
maintenance, use, and dissemination of personally identifiable information that is maintained in DoD
systems of records. Under the Privacy Act, all DoD agencies, including the Department of the Air Force,
must publish a System of Records Notice in the Federal Register on the establishment and/or modification
of a system of records to indicate what data are being collected, why, and how those data are intended to be
used. In addition, the guidance on implementation of the DoD Privacy Program (Department of Defense
Instruction 5400.11) further restricts the use of personal identifiers to link DoD data to other sources and
iteratively explore the value of the new data for modeling. Thus, the idea of a technology-enabled system
that has an iterative and exploratory relationship with HR records is incompatible with the structure of the
policies governing the underlying data.6
Existing Organizational Practices and Culture Can Hinder Applications of New

Technologies to Talent Management
In addition to policies, existing organizational practices or other unique aspects of USAF culture can be a
barrier to successfully applying technology-enabled practices. For technology-enabled practices to succeed,
employees must understand the new practices and buy into the organizational goals. The technology-enabled
framework makes this reality clear: It cannot function without a high level of interactivity between willing
humans and other components of the system. Consequently, survey research has found that most companies
that succeed in adopting large-scale AI practices spend more on the organizational activities related to adop-
tion than they do on the analytics themselves (Fountaine, McCarthy, and Saleh, 2019).
Employees who do not buy into the utility of new technologies have a variety of means to undermine
its transition. They can practice what information warfare professionals call denial and deception—either
6 As we discussed earlier, the USAF is in a relatively good position for many HRM processes because it already has rich data
stores to draw on. The privacy program limitations primarily affect new data collection and linking of DoD data with non-
traditional data sources using personal identifiers.
22
hiding behavior from data capture or providing intentional noise into the system. One example of these chal-
lenges well known to those who study outreach and recruiting is that, although accurate information about
recruiting leads is foundational, incentivizing recruiters to record accurate information is a long-standing
challenge because recruiters alter data inputs according to how the data affect decisions (Lim, Orvis, and
Hall, 2019). In addition to individual employees, organizational functions (and their data) can be stovepiped,
lacking a strong incentive to pursue a cross-functional priority if doing so does not present clear benefits to
the functional agendas. In short, the USAF may be more likely to adopt technology-enabled practices if it
resourced the activities that technology adoption requires, such as additional staffing for new data entry and
data management requirements in support of the organization’s adoption goals.
Data Management Infrastructure Can Hinder Applications of New Technologies

to Talent Management More Than the Capabilities of Data-Driven Systems
In addition to policy and organizational limitations, existing data management infrastructure is likely to
present challenges for applying the latest technologies to talent management. Various HR functions in the
USAF own and manage functionally aligned data through separate transactions databases. For example,
the Air Force Recruiting Service owns and manages a database that first identifies individuals when they
sign a contract to serve in the USAF. Then, Air Education and Training Command is responsible for track-
ing any training that those individuals receive and how they perform. General HR records belong to several
databases maintained by the Air Force Personnel Center, an agency that is directly subordinate to the USAF
chief of HR. Academic transcripts reside in separate repositories maintained by commissioning sources (for
undergraduate education) or by the Air Force Institute of Technology (for advanced academic degrees). Even
within a USAF function, separate databases can be managed and maintained by industry partners whose
contracts do not necessarily contain requirements for integrating with the rest of the ecosystem. Simply
establishing a human capital history for an individual member would require an HR analyst to orchestrate
permission and transfer and reconcile data from all these disparate organizations.
Establishing the right infrastructure for technology-enabled practices presents two further hurdles for
the USAF. First, the USAF’s infrastructure is already complex because of the size of the HR enterprise and
the volume of business needs. In this situation, with a large amount of “technical debt,” it can be difficult to
identify incremental choices that can permit new approaches without reworking the entire system. A second
hurdle is justifying the investment of resources to enable cross-functional capabilities that yield indirect and
diffuse benefits, compared with a weapon system or military capability that directly benefits a functional
stakeholder. To secure scarce funding to develop infrastructure for technology-enabled talent management
practices, HR leadership must articulate the benefits of such technologies. DoD increasingly recognizes the
value of data strategy and software architecture (DoD, 2020). The remaining question is whether the USAF
will succeed in applying these principles to HRM.
23
CHAPTER 4
A Way Forward for Technology-Enabled

Talent Management
In the previous chapter, we discussed a variety of use cases for technology-enabled talent management in the
USAF along with certain barriers to implementation. In the next chapter, we close by charting a way forward
for digitally transforming USAF talent management. The DoD data strategy clearly articulates a vision for
being able to use data “at speed and scale for operational advantage and increased efficiency,” and it names
business analytics as one of its three focus areas (DoD, 2020). The DoD services, including the USAF, are in
the process of developing their implementation plans for this strategy, and we wrote this chapter to provide a
high-level overview of factors for policymakers to consider in that process.1
The USAF can structure its approach to address ongoing and emerging HR needs by disaggregating the
challenges into their component parts and taking an enterprisewide view to enable deployment of solutions
to those challenges. Both organizational and technological aspects of infrastructure need to evolve. Table 4.1
charts the stages of development broadly, highlighting focus areas where the USAF should emphasize devel-
opment efforts. The focus areas for each stage may serve as a checklist to evaluate proficiencies and outline
future steps to enable more-advanced IT capabilities. As the table shows, the USAF can take a structured view
of its approach to continuing to implement the latest technologies to HRM processes through five stages:
(1) organizational and policy foundations; (2) technological foundation; (3) data curation, data management,
and data services; (4) analysis systems, methods, and services; and (5) enterprise integration and deployment.
Organizational and Policy Foundations

The first stage emphasizes development of the organizational and policy foundation necessary to execute a
successful implementation, leveraging future warfighting concepts to align HRM with USAF operational
readiness and motivate HRM system acquisitions. Major USAF operational concepts, initiatives, and pro-
grams are combined with stakeholder inputs from across the organization to specify the technology-enabled
talent management strategy. Doing so informs which business processes will have the greatest benefit to
operational needs. There is an existing set of policies that precisely defines the organizational roles for indi-
viduals or groups, their responsibilities, and the authorities necessary to execute processes. Implementation
plans should adjust these policies and organizational structures or relationships to institutionalize the use of
emerging technologies to enhance data-driven decisions while improving the analytic capacity of the HRM
1 A significant challenge that we do not discuss in detail is evaluating the necessary changes in organizational structures and
the alignment of responsibilities and staff under the new paradigm. Some of the emphasis areas in Table 4.1 will fall to the
Department Chief Data Officer functional responsibility; others will necessarily be part of the HR function. Furthermore,
certain areas of HR, such as operational aircrew training and staffing, are already shared with other functional authorities.
This omission is not meant to suggest that these institutional concerns are not challenges; rather, they are difficult to address
before the new HRM capabilities have been identified, designed, and tested.
25
TABLE 4.1
Charting the Growth of Technology-Enabled Talent Management
Stage Scope Areas of Emphasis
1 Organizational • Solicit and document stakeholder inputs and needs (operational and command level) to
and policy determine what changes will have the greatest short- and long-term benefit.
foundations • Specify ties to major USAF and joint operating concepts, initiatives, and programs.
• Specify workforce roles and personnel considerations required for developing and sustaining
technology-enabled business practices.
• Develop a quantitatively assessed maturity model applicable to all sites that incorporates
operational stakeholder needs and strategies to reduce operational risks as maturity of the
implementation increases.a
• Define policies that coherently define USAF roles, responsibilities, and authorities for
multidisciplinary teams and individuals to implement technology-enabled business practices and
requirements for cross-role collaborations.
2 Technological • Determine the major areas of technological infrastructure that will need to be built or will require
foundation change, and the role and extent of automation to be provided.
• Determine the USAF’s ability to leverage existing USAF (such as The Air Force Research
Laboratory [AFRL]), DoD, and federal capabilities, resources, and standards for
technology-enabled business practices.
• Specify the software development and deployment practices that will be used to implement
information security (such as zero trust)b relevant to DoD systems, and define the risk
management practices (such as DoD Risk Management Framework) that are to be implemented
in the technological foundation.
• Specify the operational architecture and development pipeline for incorporating foundational
technological components and employing technology-enabled business practices.
3 Data • Specify standards and technical policies for secure data collection, information exchange and
curation, data traceability,c and information storage across sites (DoD enterprise and the USAF).
management, • Enumerate the workflows that comprise existing and future business practices.
and data • Develop a detailed strategy for migrating data repositories to new repositories.
services • Determine the specific performance, functional, and user experience requirements for
technologically provided data services.
• Develop the necessary tools for curation and technical management of HRM data assets.
4 Analysis • Specify USAF goals for leveraging advanced analytical methods, how they should function, and
systems, the requirements for employing new analysis tools.
methods, and • Develop comprehensive standards for analysis, such as metrics and supporting methodology
services measurement and analysis.
• Define quality-oriented requirements for using new analysis tools.
• Specify the architectural strategy for integrating new analytical processes into HRM systems and
for enterprisewide analytics.
• Define the USAF’s preferred approach to development and testing, such as the functional
requirements for development and testing environments.
5 Enterprise • Define the plan for user acceptance testing and operator training for all relevant workforce roles.
integration and • Define the short- and long-term technical methods for deploying new technology and services
deployment into operations and sustaining their use.
a A maturity model typically defines five levels of an organization’s achievement in developing enterprise capabilities, such as Initial, Repeatable,
Defined, Managed, Optimizing (Rosenstock, Johnston, and Anderson, 2000). The level of granularity in the model reflects the implementation
complexity of the capabilities being sought. The degree of organizational achievement can be systematically measured and supported by
quantitative assessments or gauged qualitatively.
b National Institute of Standards and Technology (NIST) Special Publication 800-207 (Rose et al., 2020) describes zero trust: “Zero trust refers to
an evolving set of security paradigms that narrows defenses from wide network perimeters to individual or small groups of resources.”
c In security, traceability generally refers to a systematic ability to reference all potential risks with security policies and control measures and to
the ability to track security incidents as they occur and their impacts. For a broader introduction, see, for example, NIST Special Publications
800-37 Revision 2 (National Institute of Standards and Technology Joint Task Force, 2018) about the Risk Management Framework and Special
Publication 800-160, Vol. 1 (Ross, McEvilley, and Oren, 2016), about Systems Security Engineering.
workforce through changes in hiring, training, and fostering a collaborative culture (National Academies of
Sciences, Engineering, and Medicine, 2020). Prior structures, in which managers cluster around relatively
narrow areas with very specific process knowledge, might need to become more flexible (Sanders and Wood,
2020). Policy changes to facilitate engagement with non-USAF stakeholders should also be addressed and
used when possible. Finally, an effective strategy for communicating with stakeholders about implementa-
26
A Way Forward for Technology-Enabled Talent Management
tion will need to be developed; so will an understanding of the USAF workforce required to execute and sus-
tain the goals. Preparing a robust, full-cycle communication strategy about the implementation to promote
adoption and provide the appropriate levels of transparency about the effort is also an important organiza-
tional consideration. All these tasks can be applied to develop a robust maturity model that is quantitatively
assessable across USAF components.
Technological Foundation
Modern, enterprisewide systems are typically built using foundational components that (1) promote flex-
ibility in achieving operational goals, (2) deliver a secure infrastructure for information processing, and
(3) enable automation in a way that is commensurate with business practices and analysis goals. It is impor-
tant to prioritize the definition of the information architecture over individual enabling technologies. In this
stage, the USAF has an opportunity to identify and leverage existing capabilities, resources, and standards
from across the USAF (such as AFRL, Platform One, and Cloud One), DoD, and federal sources (such as
NIST) to detail the operational architectural strategy and development pipeline for building technology-
enabled decision-support tools.
Data Curation, Data Management, and Data Services

Data analysis systems rely on comprehensive technical strategies for curating, managing, and providing data
to organizational stakeholders. This entails enumerating the anticipated workflows for HRM talent man-
agement, technical policies for securely transacting information that is to be exchanged, and developing the
tools necessary to work with, manage, and prepare data for analytical methods. The technological approach
to migrating data into new repositories is resolved in this stage along with functional and operational per-
formance requirements of the data-oriented systems themselves. Also important to resolve are requirements
for user experience and usability requirements, interoperability and reporting services, personnel-centric
analysis (such as recruiting career tracks, professional development, health services), and data services for
readiness-oriented assessments of the operational force.
Analysis Systems, Methods, and Services

Analysis systems are predicated on work developed in the preceding stages. That is, USAF efforts to pro-
vide an organizational, policy, and technological foundation—along with core operating capabilities for
data-related services—inform the analytical capabilities being sought by the enterprise for technology-
enabled business practices. In this stage, USAF efforts focus specifically on the topic of analysis. Given an
adequate set of data resources, what are the precise analytical goals, methods, and standards to be imple-
mented? Both the functionality of analysis systems and assurances about their outputs must be defined.
Functional requirements define the operational goals of a system. In addition, a detailed strategy for inte-
grating new analysis processes into HRM systems and the approach to their development and testing will
be required. Analysis systems may require special consideration in the case of ML, for which training data
sets may need to be robustly developed to achieve the expected performance benefit for enterprise and
HRM talent management goals.
27
Enterprise Integration and Deployment

The final broad category looks ahead to enterprise integration and deployment. The main courses of action
focus on the technical framework for deploying technology for technology-enabled practices in the short and
long term and prepare the USAF workforce to incorporate that framework into their operational activities
and sustain it over time.
28
CHAPTER 5
Conclusion
The USAF and other DoD services have established themselves over decades as leaders and innovators in
capitalizing on data to improve HRM decisionmaking. These efforts have placed the USAF in a position
where it is feasible to further adopt many HRM practices and technologies that are on the cutting edge. But
homing in on true areas of value for the USAF amid the hype associated with industry practices can be dif-
ficult. Some possible decision-support systems would produce only marginal improvements compared with
existing practices. Others may not apply to the military HRM context at all.
Our framework defining a technology-enabled talent management system provides some clarification on
what it would look like if the USAF HRM community were to move incrementally toward AI and ML adop-
tion. The next step involves moving away from periodic decision cycles informed by rerunning an analytic
script and toward continual, highly customized decisions enabled by interactive systems that perform better
and better over time through feedback from decisionmakers. The use cases from prior research also suggest
that it has been easiest in the short term to explore ways to use machine intelligence with existing data on
past outcomes. Reaping value from technology-enabled practices requires addressing barriers to dynamically
adapting data collection or experimentally implementing policies to generate feedback for rapid improve-
ments to the performance of HRM systems.
The overarching lesson from this research is not to allow past successes to lead to entrenched practices
that become barriers to further improvements. Our use cases show that the USAF can continue to build on
its legacy by integrating new data-centric technologies into talent management, but becoming more effec-
tive requires adjustments in many areas, such as technology, culture, and long-standing policy frameworks.
Companion reports in this series address key topics that will help the USAF grapple with these necessary
changes. One report covers effective practices that similar organizations in government and industry have
employed to successfully transform talent management systems and provides concrete and relevant examples
(Yeung et al., 2022). The other report specifically focuses on factors that will help technology-enabled HRM
initiatives succeed when competing for funding in the planning, programming, budgeting, and execution
process (Snyder, 2022).
29
Abbreviations
AFRL The Air Force Research Laboratory
AFSC Air Force Specialty Code
AI artificial intelligence
ASVAB Armed Services Vocational Aptitude Battery
DoD U.S. Department of Defense
HR human resources
HRM human resource management
IST initial skills training
IT information technologies
ML machine learning
NIST National Institute of Standards and Technology
PAF RAND Project AIR FORCE
PPST Personnel Policy Simulation Tool
REWS retention early warning system
RRM Recruiting Resource Model
USAF U.S. Air Force
31
References
Air Force Instruction 36-2101, Classifying Military Personnel (Officer and Enlisted), Department of the Air Force,
June 25, 2013, change 2, March 9, 2017.
Air Force Instruction 36-2501, Officer Promotions and Selective Continuation, Department of the Air Force,
May 4, 2020.
Air Force Instruction 36-2606, Reenlistment and Extension of Enlistment in the United States Air Force,
Department of the Air Force, September 20, 2019.
Alpaydin, Ethem, Machine Learning, Massachusetts Institute of Technology Press, 2016.
Arrieta, Alejandro Barredo, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto
Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al., “Explainable Artificial
Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges Toward Responsible AI,” Information
Fusion, Vol. 58, 2020, pp. 82–115.
Bisson, Peter, Bryce Hall, Brian McCarthy, and Khaled Rifai, Breaking Away: The Secrets to Scaling Analytics,
McKinsey Global Institute, 2018.
Bloom, Nicholas, Benn Eifert, Aparjit Mahajan, David McKenzie, and John Roberts, “Does Management Matter?
Evidence from India,” Quarterly Journal of Economics, Vol. 128, No. 1, 2013, pp. 1–51.
Brose, Christian, The Kill Chain: Defending America in the Future of High-Tech Warfare, Hachette Books, 2020.
Cappelli, Peter, Prasanna Tambe, and Valery Yakubovich, Artificial Intelligence in Human Resources
Management: Challenges and a Path Forward, Social Science Research Network, 2018.
Carretta, Thomas R., “Pilot Candidate Selection Method: Still an Effective Predictor of U.S. Air Force Pilot
Training Performance,” Aviation Psychology and Applied Human Factors, Vol. 1, No. 1, 2011, pp. 3–8.
Chui, Michael, Vishnu Kamalnath, and Brian McCarthy, “An Executive’s Guide to AI,” webpage, McKinsey
Global Institute, 2020. As of October 11, 2022:
https://www.mckinsey.com/capabilities/quantumblack/our-insights/an-executives-guide-to-ai
Department of Defense Instruction 1304.31, Enlisted Bonus Program, Office of the Under Secretary of Defense
for Personnel and Readiness, November 5, 2020.
Department of Defense Instruction 5400.11, DoD Privacy and Civil Liberties Programs, Office of the Chief
Management Officer of the Department of Defense, January 29, 2019; Change 1 Effective: December 8, 2020.
Diebold, Francis X., “What’s the Big Idea? ‘Big Data’ and Its Origins,” Significance, Vol. 18, No. 1, February 2021,
pp. 36–37.
DoD—See U.S. Department of Defense.
Fountaine, Tim, Brian McCarthy, and Tamim Saleh, “Building the AI-Powered Organization,” Harvard Business
Review, July–August 2019.
Guenole, Nigel, and Sheri Feinzig, The Business Case for AI in HR: With Insights and Tips on Getting Started,
IBM Corporation, 2018.
Kahneman, Daniel, Andrew M. Rosenfield, Linnea Gandhi, and Tom Blaser, “Noise: How to Overcome the
High, Hidden Cost of Inconsistent Decision Making,” Harvard Business Review, October 2016.
Keller, Kirsten M., Kimberly Curry Hall, Miriam Matthews, Leslie Adrienne Payne, Lisa Saum-Manning,
Douglas Yeung, David Schulker, Stefan Zavislan, and Nelson Lim, Addressing Barriers to Female Officer
Retention in the Air Force, RAND Corporation, RR-2073-AF, 2018. As of March 1, 2021:
https://www.rand.org/pubs/research_reports/RR2073.html
Knapp, David, Bruce R. Orvis, Christopher E. Maerzluft, and Tiffany Berglund, Resources Required to Meet the
U.S. Army’s Enlisted Recruiting Requirements Under Alternative Recruiting Goals, Conditions, and Eligibility
Policies, RAND Corporation, RR-2364-A, 2018. As of March 17, 2021:
33
Knutson, Felix, People First: Improving Equitability of Air Force Recruiting Operations, RAND Corporation,
RGSD-433, 2019. As of March 17, 2021:
https://www.rand.org/pubs/rgs_dissertations/RGSD433.html
Kohavi, Ron, and Roger Longbotham, “Online Controlled Experiments and A/B Testing,” in Claude Sammut
and Geoffrey I. Webb, eds., Encyclopedia of Machine Learning and Data Mining, Springer, 2017.
Lim, Nelson, Bruce R. Orvis, and Kimberly Curry Hall, Leveraging Big Data Analytics to Improve Military
Recruiting, RAND Corporation, RR-2621-OSD, 2019. As of July 23, 2020:
Luca, Michael, and Max H. Bazerman, “Want to Make Better Decisions? Start Experimenting,” MIT Sloan
Management Review, Vol. 61, No. 4, 2020.
Mattock, Michael G., Beth J. Asch, James Hosek, and Michael Boito, The Relative Cost-Effectiveness of Retaining
Versus Accessing Air Force Pilots, RAND Corporation, RR-2415-AF, 2019. As of October 11, 2022:
Mattock, Michael G., James Hosek, Beth J. Asch, and Rita T. Karam, Retaining U.S. Air Force Pilots When the
Civilian Demand for Pilots Is Growing, RAND Corporation, RR-1455-AF, 2016. As of October 11, 2022:
McKinsey Analytics, The State of AI in 2020, November 2020.
Mullainathan, Sendhil, and Jann Spiess, “Machine Learning: An Applied Econometric Approach,” Journal of
Economic Perspectives, Vol. 31, No. 2, 2017, pp. 87–106.
Myung, Jay I., Daniel R. Cavagnaro, and Mark A. Pitt, “A Tutorial on Adaptive Design Optimization,” Journal of
Mathematical Psychology, Vol. 57, Nos. 3–4, 2013, pp. 53–67.
National Academies of Sciences, Engineering, and Medicine, Strengthening U.S. Air Force Human Capital
Management: A Flight Plan for 2020–2030, National Academies Press, 2020.
National Institute of Standards and Technology Joint Task Force, Risk Management Framework for Information
Systems and Organizations: A System Life Cycle Approach for Security and Privacy, National Institute of
Standards and Technology, SP 800-37, Rev. 2, December 2018.
National Security Commission on Artificial Intelligence, Final Report, 2021.
O’Shea, Patrick Gavan, and Karrin E. Puente, “How Is Technology Changing Talent Management?” in David G.
Collings, Kamel Mellahi, Wayne F. Cascio, eds., Oxford Handbook of Talent Management, Oxford University
Press, 2017.
Osoba, Osonde A., Benjamin Boudreaux, Jessica Saunders, J. Luke Irwin, Pam A. Mueller, and Samantha
Cherney, Algorithmic Equity: A Framework for Social Applications, RAND Corporation, RR-2708-RC, 2019. As of
October 5, 2022:
Osoba, Osonde, and Willliam Welser IV, An Intelligence in Our Image: The Risks of Bias and Errors in Artificial
Intelligence, RAND Corporation, RR-1744-RC, 2017. As of October 5, 2022:
Oswald, Frederick L. Tara S. Behrend, Dan J. Putka, and Evan Sinar, “Big Data in Industrial-Organizational
Psychology and Human Resource Management: Forward Progress for Organizational Research and Practice,”
Annual Review of Organizational Psychology and Organizational Behavior, Vol. 7, No. 1, 2020, pp. 505–533.
Putka, Dan J., Adam S. Beatty, and Matthew C. Reeder, “Modern Prediction Methods: New Perspectives on a
Common Problem,” Organizational Research Methods, Vol. 21, No. 3, 2018, pp. 689–732.
Ransbotham, Sam, David Kiron, Philipp Gerbert, and Martin Reeves, “Reshaping Business with Artificial
Intelligence: Closing the Gap Between Ambition and Action,” MIT Sloan Management Review, September 6,
2017.
Robbert, Albert A., Anthony D. Rosello, C. R. Anderegg, John A. Ausink, James H. Bigelow, William W. Taylor,
and James Pita, Reducing Air Force Fighter Pilot Shortages, RAND Corporation, RR-1113-AF, 2015. As of
October 11, 2022:
34
References
Robson, Sean M., Maria C. Lytell, Matthew M. Walsh, Kimberly Curry Hall, Kirsten M. Keller, Vikram Kilambi,
Joshua Snoke, Jonathan Welburn, Patrick Roberts, Owen Hall, and Louis T. Mariano, U.S. Air Force Enlisted
Classification and Reclassification: Potential Improvements Using Machine Learning and Optimization Models,
RAND Corporation, RR-A284-1, 2022. As of October 7, 2022:
https://www.rand.org/pubs/research_reports/RRA284-1.html
Rose, Scott, Oliver Borchert, Stu Mitchell, and Sean Connelly, Zero Trust Architecture, National Institute of
Standards and Technology, SP 800-207, August 2020.
Rosenstock, Christian, Robert S. Johnston, and Larry M. Anderson, Maturity Model Implementation and Use: A
Case Study, Project Management Institute, 2000.
Ross, Ron, Michael McEvilley, and Janet Oren, Systems Security Engineering: Considerations for a
Multidisciplinary Approach in the Engineering of Trustworthy Secure Systems, National Institute of Standards and
Technology, SP 800-160, Vol. 1, November 2016.
Sanders, Nada R., and John D. Wood, “The Secret to AI Is People,” Harvard Business Review, August 24, 2020.
Schulker, David, Lisa M. Harrington, Matthew Walsh, Sandra Kay Evans, Irineo Cabreros, Dana Udwin, Anthony
Lawrence, Christopher E. Maerzluft, and Claude Messan Setodji, Developing an Air Force Retention Early Warning
System: Concept and Initial Prototype, RAND Corporation, RR-A545-1, 2021. As of November 22, 2021:
Schulker, David, Nelson Lim, Luke Matthews, Geoffrey E. Grimm, Anthony Lawrence, and Perry Firoz,
Can Artificial Intelligence Help Improve Air Force Talent Management? An Exploratory Application, RAND
Corporation, RR-A812-1, 2020. As of October 5, 2022:
Siroker, Dan, and Pete Koomen, A/B Testing: The Most Powerful Way to Turn Clicks into Customers, John Wiley
and Sons, 2013.
Snyder, Don, Funding Technology-Related Business Initiatives in the Department of the Air Force, RAND
Corporation, RR-A1198-3, 2022. As of August 16, 2022:
Tambe, Prasanna, Peter Cappelli, and Valery Yakubovich, “Artificial Intelligence in Human Resources
Management: Challenges and a Path Forward,” California Management Review, Vol. 61, No. 4, 2019, pp. 15–42.
U.S. Code, Title 5, Government Organization and Employees, Part I, The Agencies Generally, Chapter Five,
Administrative Procedure, Subchapter II, Administrative Procedure, Section 552, Public Information; Agency
Rules, Opinions, Orders, Records, and Proceedings.
U.S. Department of Defense, Summary of the 2018 Department of Defense Artificial Intelligence Strategy:
Harnessing AI to Advance Our Security and Prosperity, 2019.
U.S. Department of Defense, DoD Data Strategy, 2020.
Walsh, Matthew, David Schulker, Nelson Lim, Albert A. Robbert, Raymond E. Conley, John S. Crown, and
Christopher E. Maerzluft, Department of the Air Force Officer Talent Management Reforms: Implications for
Career Field Health and Demographic Diversity, RAND Corporation, RR-A556-1, 2021. As of November 16, 2021:
Welsh, John R., Jr., Susan K. Kucinkas, and Linda T. Curran, Armed Services Vocational Aptitude Battery
(ASVAB): Integrative Review of Validity Studies, Air Force Human Resources Laboratory, AFHRL-TR-90-22, July
1990.
Wright, Amy, Diane Gherson, Josh Bersin, and Janet Mertens, “Accelerating the Journey to HR 3.0: Ten Ways to
Transform in a Time of Upheaval,” IBM Institute for Business Value, October 2020.
Yan, Xiao, Jaewon Yang, Mikhail Obukhov, Lin Zhu, Joey Bai, Shiqi Wu, and Qi He, “Social Skill Validation
at LinkedIn,” Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data
Mining, 2019, pp. 2943–2951.
Yeung, Douglas, Elicia M. John, Jeannette Gaudry Haynie, James Ryseff, Bonnie L. Triezenberg, and Nelson
Lim, Implementing Technology-Enabled Human Resources Capabilities in the U.S. Air Force: Insight from the
Private Sector and Military Services, RAND Corporation, RR-A1198-1, 2022. As of December 2022:
35
PROJEC T A I R FORC E
T
he U.S. Air Force (USAF) and the other military services have a long history
of innovation in human resource management (HRM). The recent industry
boom in data-related technologies has prompted USAF leaders to sponsor
research on how these technologies could further improve HRM decisions.
This report describes the common theme of this research portfolio, which
is that adopting HRM practices that are technology-enabled could lead to more-effective
talent management. Of course, technologies exist on a spectrum, and the USAF, like
all other organizations, already rely on some technologies to perform HRM functions.
However, by pursuing the latest technological advances, the USAF can continue to
improve the efficiency and effectiveness of HRM processes.
To help policymakers understand the contrast between technology-enabled practices

and practices already in place that make use of rich data, this report describes industry
practices that fit under the umbrella of technology-enabled talent management and
presents a framework highlighting the distinctive features of those practices. The authors
focus principally on the use of artificial intelligence (AI) and other analytic techniques to
derive insight from data at speed and scale. The authors then present use cases in which
recent research has demonstrated technology-enabled practices in the USAF context,
discuss barriers to further implementation, and present an implementation structure for
moving toward greater adoption of these practices.
$22.00
ISBN-10 1-9774-1043-X
ISBN-13 978-1-9774-1043-6
52200
www.rand.org 9 781977 410436
RR-A1198-2

Rand Rra1198-2

Uploaded by

Copyright:

Available Formats

Rand Rra1198-2

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Rand Rra1198-2

Uploaded by

Copyright:

Available Formats

Research Report

DAVID SCHULKER, MATTHEW WALSH, NELSON LIM, AJAY K. KOCHHAR

How the U.S. Air Force

RAND Project AIR FORCE

A concerted approach to technology-enabled talent management coupled with an openness to think

Technology-Enabled Talent Management Becoming a Commercial

How the USAF Uses Data for Human Resources

Policy questions Data Analysis Recommendations New policies

Technology-Enabled Business Practices: The Future of USAF Talent

Leveraging New Types of Data to Enhance Decisions

Learn from decisions

Data Decisions Outcomes

Leveraging New Forms of Machine Intelligence to Enhance Decisions

Introducing Interactivity Among Machine Intelligence, Human Intelligence, Data,

USAF Well Positioned to Join These Organizations

Use Cases for Technology-Enabled

Human and machine

Tactical Recruiting Challenge: Helping Recruiters Manage Their Time

RAND Research Shows How Technology-Enabled Recruiting Could Improve

Recruiting as a leads funnel

Online job order

SOURCE: Lim, Orvis, and Hall, 2019.

Strategic Recruiting Challenge: Providing Enough Resources for

RAND Built a Model That Optimizes Recruiting Resource Levels

Specialty Classification Challenge: Matching Airmen to the Right Job

RAND Used Data Science Techniques to Match Airmen to Promising Specialties

Probability of completing IST Probability of completing term

Probability of reenlistment Probability of promotion

Actual pipeline Actual pipeline

Increase Decrease Same

SOURCE: Robson et al., 2022.

Development Challenge: Labor-Intensive Board Selection Processes

RAND Built an ML Model That Automatically Finds Officers with Good

Assignments Challenge: How to Best Use the New “Talent Marketplace”

RAND Found New Opportunities for Information Exchange

4 Unpublished RAND research by David Schulker and Matthew Walsh.

• Detailed job description

Officer Assignment team

• Optimal policy to boost

Promotion Challenge: Unpredictable Effects of Policy Changes

RAND Developed a Tool to Predict the Inventory of Officers

End strength Promotion

Projected inventory Aging

SOURCE: Adapted from Walsh et al., 2021.

Retention Challenge: Preventing Personnel Shortages or Surpluses

RAND Created a Retention Early Warning System

The Path to a Technology-Enabled Capability

Learn from decisions

Implementation Challenges for a Technology-Enabled Talent

Policies Can Hinder Application of New Technologies to Talent Management

Existing Organizational Practices and Culture Can Hinder Applications of New

Data Management Infrastructure Can Hinder Applications of New Technologies

A Way Forward for Technology-Enabled

Organizational and Policy Foundations

Data Curation, Data Management, and Data Services

Analysis Systems, Methods, and Services

Enterprise Integration and Deployment

To help policymakers understand the contrast between technology-enabled practices

www.rand.org 9 781977 410436