Trailblazers into Safety-II:

American Airlines’
Learning and Improvement Team
A White Paper Outlining AA’s Beginnings of a Safety-II Journey

June 2020

0
American Airlines’ LIT Leadership
Ron Thomas, Vice President, Safety
Al Madar, Managing Director, Operations Safety and Compliance
Captain Guy Mouton, Senior Manager, LOSA / SMS
First Officer James Kwasny, APA LOSA Coordinator
Ricardo L. Campos, Senior Investigator, Flight Safety
Gregory Scheidel, Senior Analyst, LOSA

American Airlines’ LIT Members

Captain Will Dismukes
First Officer Bogomir Glavan
First Officer Nicholas Peterson
Captain Michael Ford
First Officer Anand Vaswani

Safety Research Intern

Christine Jefferies, Cognitive Systems Engineering Lab (CSEL), The Ohio State University

Citation:
American Airlines’ Department of Flight Safety (2020). Trailblazers into Safety-II: American
Airlines’ Learning and Improvement Team, A White Paper Outlining AA’s Beginnings of a
Safety-II Journey.

1
Abbreviations
AA American Airlines

APA Allied Pilots Association; AA’s pilot union

AQP Advanced Qualification Program

ATC Air Traffic Control

CA Captain

CAMI Confirm-Activate-Monitor-Intervene; term to reference automation usage

DCT Data Collection Tool

FAA Federal Aviation Administration

FMS Flight Management System

FO First Officer

ICR Inter-Coder Reliability

LOSA Line Operations Safety Audit

OSU The Ohio State University

PF Pilot Flying

PM Pilot Monitoring

RAG Resilience Assessment Grid

SOP Standard Operating Procedure(s)

SMS Safety Management System

TEM Threat and Error Management

2
Content

ABBREVIATIONS ........................................................................................................................................ 2
CONTENT ..................................................................................................................................................... 3
EXECUTIVE SUMMARY .............................................................................................................................. 5
PROLOGUE .................................................................................................................................................. 6
INTRODUCTION ............................................................................................................................................ 6
KEY THEMES............................................................................................................................................... 7
ROADMAP ................................................................................................................................................... 7
UNION PARTNERSHIP................................................................................................................................... 7
ACADEMIC PARTNERSHIP ............................................................................................................................ 7
SAFETY MODELS AT AA............................................................................................................................ 8
SAFETY-I: THREAT AND ERROR M ANAGEMENT ............................................................................................. 8
AA’s TEM Model ................................................................................................................................... 8
LOSA ..................................................................................................................................................... 9
SAFETY-I TO SAFETY-II: LEARNING AND IMPROVEMENT TEAM ....................................................................... 9
Our Name .............................................................................................................................................. 9
Safety-II Within SMS ............................................................................................................................. 9
Building LIT’s Approach to Safety-II ...................................................................................................... 9
METHODS .................................................................................................................................................. 14
STRATEGY ................................................................................................................................................ 14
Proof of Concept ................................................................................................................................. 14
DATA COLLECTION .................................................................................................................................... 15
Observations ....................................................................................................................................... 15
Interviews ............................................................................................................................................ 17
DATA ANALYSIS ........................................................................................................................................ 18
Inter-Coder Reliability .......................................................................................................................... 18
Analysis ............................................................................................................................................... 19
LIT OBSERVERS ....................................................................................................................................... 19
Selection.............................................................................................................................................. 19
Training ............................................................................................................................................... 20
IMPLEMENTATION ...................................................................................................................................... 20
VALUE ........................................................................................................................................................ 20
DISSEMINATION ......................................................................................................................................... 21
Pilot Training ....................................................................................................................................... 21
Captain Leadership Training ............................................................................................................... 21
Formal and Informal Discussions ........................................................................................................ 21
Potential for Process Change ............................................................................................................. 22
TOWARDS ORGANIZATIONAL INTEGRATION ................................................................................................ 23
CONCLUSION ............................................................................................................................................ 24

3
APPENDIX A. EVOLUTION OF LIT MODEL: POTENTIALS ................................................................... 25
APPENDIX B. EVOLUTION OF LIT MODEL: PROFICIENCIES AND QUALIFIERS .............................. 29
APPENDIX C. KNOWLEDGE ELICITATION............................................................................................. 32
APPENDIX D. FIGURES ............................................................................................................................ 33
APPENDIX E. LIT DATA COLLECTION TOOLS ...................................................................................... 36
APPENDIX F. LIT “SHOP TALK” COGNITIVE INTERVIEW QUESTIONS ............................................. 38
REFERENCES ............................................................................................................................................ 40
GLOSSARY ................................................................................................................................................ 42
AUTHORS................................................................................................................................................... 43

4
Executive Summary
American Airlines’ Learning Improvement Team (LIT) has developed a new language and data
collection methodology to capture pertinent Safety-II data from line flight crews operating
regularly scheduled flights within their conventional domain. This document outlines the
numerous lessons learned, difficulties overcome, and insights realized during the first phase of
this project. This information is offered to various aviation and non-aviation organizations in the
hopes that the safety of all high-risk, mission critical systems might be improved.

The novelty of this process required the development of several original and previously untested
data collection tools that were created for the project. A number of iterative data collection tools
were developed along the journey to support narrative data capture and were helpful in defining
terms and organizing structure. Numerous data quality efforts were undertaken to validate the
process and resultant data standardization and consistency for analysis. A handful of observers
collected over 100 observations of narrative and tabulated data to build the initial data set for
analysis and learning. The growth and development of the language and process along this
journey was substantial and enlightening to the entire safety team. Providing a reliable data
stream to newly required training courses currently being developed to meet new FAA
requirements has become a central goal.

American Airlines’ LIT group has developed a revolutionary language, data collection tool, data
collection methodology and analysis process to better understand the relationship between
Work-As-Imagined and Work-As-Done as it relates to resilient performance of flight crews in
modern challenging systems. Intellectual aptitude and flexibility to accomplish similar gains
should not be under-estimated when endeavoring on a similar journey. The concepts and ideas
are novel and may not fit traditional data streams or collection methodologies.

Care should be taken to avoid forcing a Safety-II effort to fit within current programs. It is also
necessary that the organization has a solid understanding of its own organizational culture and
associated programs before intending to step into the realm of Safety-II. LIT is fortunate to have
the support of AA’s Safety Leadership in developing this Safety-II effort as its own unique data
stream to complement AA’s traditional Safety Management System (SMS). This investment in
human and material resources has proven valuable, as early analysis efforts have
demonstrated applicability of this LIT data to pilot mentoring and leadership courses in the
airline training curriculum.

AA’s LIT members have succeeded in demonstrating the successful development of language
and then applying this language to data collection for deeper understanding of how AA’s flight
crews adapt to changing circumstances during routine flight operations. The future opportunities
for this new stream of data to enhance an established SMS process are invigorating and
provide the boost and spark to move beyond current safety accomplishments using Safety-I
rationale and processes. Studying the complex and challenging environment of airline
operations from a Safety-II perspective not only provides a new avenue to accelerate learning
and development of hard-earned experience, but also engages a new source of pride and
ownership over our crews’ abilities to safely achieve organizational goals, one flight at a time.

5
Prologue

Introduction
In January 2018, AA embarked on a Safety-II journey. AA’s Safety Leadership became
interested in the concepts of Safety-II outlined in Dr. Erik Hollnagel’s multiple books and were
able to meet with Dr. Hollnagel to seek guidance in implementation. The AA Line Operational
Safety Audit (LOSA) Program was identified as the program of choice to explore this theory and
design a proof of concept to observe crews using this framework for safety and efficiency
improvements. Although this Safety-I framework within LOSA was a logical starting point, it was
strongly suggested by Dr. Hollnagel to separate the emerging Safety-II program from these
traditional methods.

Following delivery of a plan for implementation, Guy Mouton, AA Senior Manager LOSA, and
James Kwasny of the Allied Pilots Association (APA), recruited Captain Will Dismukes and First
Officer Bogomir (Bog) Glavan in November 2018 to lead the Safety-II effort and design a
program that could be implemented to capture Safety-II data. As professional flight crew, these
line pilots started by reading as much Safety-II literature as possible to better understand the
concepts and how these could be applied to AA operations. They met in Fort Worth, Texas, at
the AA Safety offices in December 2018 to outline program milestones and goals. The priorities
were to establish a language specific to aviation and AA and also to construct a basic
framework for how the data collection would be conducted. Recognizing the centrality of
learning, rather than compliance to the process of safety improvement and development, the
group chose to call itself the Learning Improvement Team (LIT) with the moniker ever to be a
reminder of that which must be emphasized in their work.

By describing the work of AA’s LIT team during their first two years of work, this document
seeks to accomplish three mutually reinforcing goals:

1. To document the actions taken and decisions made by a dedicated and enthusiastic
group of AA employees for the benefit of their organization.

2. To demonstrate the utility of Safety-II inspired data collection and analysis tools and
techniques.

3. To share insights recognized and obstacles overcome with others who may consider
similar paths in their efforts to improve operational safety.

A second report is anticipated in early 2021 which will share analysis and results from early
validated data.

6
Key Themes
Flight safety, Safety-I, Safety-II, resilience engineering, performance variability, resilient
performance, human performance assessment, human performance improvement

Roadmap
The LIT leadership has been self-motivated and proactive in fulfilling its mission to design and
implement a novel approach to safety at AA. With support and oversight from senior AA
leadership, and in conjunction with the APA, the LIT group outlined a robust 15-month timeline
divided into four phases:

1. Language and model development

2. Data collection and analysis
3. Solidify program structure and methods
4. Program implementation and data dissemination throughout AA and the aviation industry

Components within each phase are detailed in Figure D1 in Appendix D. At the time of this
report, Phase 3 was completed on schedule in December 2019, and Phase 4 is underway.

Union Partnership
The APA, AA’s pilot union, supports this initiative in conjunction with the other components of
AA’s existing SMS. LIT operates under the guidelines of the LOSA Memorandum of
Understanding between AA and APA, which outlines protections for both the pilot crews and for
the observers.

Academic Partnership
Following a daylong workshop with The Ohio State University (OSU) faculty and graduate
students in June 2019, AA decided to continue its partnership with OSU to ensure continuous
alignment with Safety-II core principles as well as rigorous data collection methods. A non-
disclosure agreement was secured for the confidentiality of AA data during use by OSU
researchers. One of the many benefits of the AA / OSU collaboration has been the opportunity
for AA to call on the cutting-edge theoretical developments underway at OSU and for AA to
share with the academic world the insights which can only be gained from working in a true
operational environment. In-depth discussions concerning the applicability of Work-as-imagined
versus Work-as-done occurred with the OSU CSEL team to further understanding and
applicability for the LIT members. “Work-as-imagined is both the work that we imagine others do
and the work that we imagine we or others did, do, or would do, in the past, present, or future.
Work-as-done is actual activity – what people do. It takes place in an environment that is often
not as imagined, with multiple, shifting goals, variable and often unpredictable demands”
(Shorrock, 2016).

7
Safety Models at AA

Safety-I: Threat and Error Management

AA’s TEM Model

AA’s Threat and Error Management (TEM) process has, as its primary goal, the management
and / or mitigation of the effects of threats and errors before they become operationally
consequential. With the understanding that perfect adherence to standard operating procedures
(SOPs) is unrealistic, and that SOPs alone cannot mitigate the myriad of situations that a flight
crew may encounter, the TEM model offers flight crews a reference for organizational values
and expectations during mitigation of unexpected and potentially hazardous situations
(American Airlines FOM, 2019).

The model consists of a TEM target (adapted from Van Drie, 2002) to visually depict the flight
crew’s available tools to effectively manage threats and errors as well as the mnemonic ABCs, a
framework onto which crews can scaffold their active and passive work during a flight.

Figure 1. AA’s TEM Model. (American Airlines FOM, 2019)

8
LOSA
TEM is the basis and language for LOSA data collection. Peer observers collect data on
Threats, Errors and Undesired Aircraft States from flightdeck jumpseat observations on
regularly scheduled flights with the goal to capture natural behavior as much as possible. This
data stream has proven to be very beneficial in the current AA SMS process and, since the
integration of LOSA to AA’s SMS program in October 2016, has become the data source of
choice for many recent major operational decisions at AA.

Other than their leadership teams, LIT and LOSA do not share resources, staff, or processes.
Nevertheless, for this pilot project, the foundational LOSA observation practices were adopted
for use by the LIT observers. This was especially true in terms of jumpseat protocols and
observer guidance. These specific provisions include the obligation of the observer to speak up
if they notice unsafe behavior that could greatly impact the safety of the flight. The observer also
assumes responsibility at the discretion of the captain to assist the flight crew in an emergency
situation.

Safety-I to Safety-II: Learning and Improvement Team

Our Name
The Learning and Improvement Team was named as such in order to emphasize learning and
continuous improvement on all levels within the organization: among individual team members,
between pilots in the flightdeck, and on the system level. While these aspirations reflect the
existing goals of many safety organizations, LIT chose to focus on a learning approach; LIT
views every interaction as having the power to generate new knowledge, engineer new ways to
capture, transform, and disseminate flexing of strengths, and enact expertise that might
otherwise fold into a pilot’s normal workday. With its own name, LIT seeks to embody two of its
primary premises: learning is central, and language matters.

Safety-II Within SMS

It is important to assert that Safety-II data collected within the LIT program is an additional data
stream to complement the existing data streams available at AA and does not replace any
traditional Safety-I data streams. Safety-II approaches are used in conjunction with AA’s SMS
framework and do not replace any current safety data programs. Additionally, the LIT model
does not replace AA’s TEM model nor AA’s current TEM approach to LOSA, Advanced
Qualification Program (AQP), or training philosophy. LIT has shown that it can complement the
current system because it approaches safety in a unique light.

Building LIT’s Approach to Safety-II

Greatly inspired by the work of renowned safety scientist Dr. Erik Hollnagel, especially his
Resilience Assessment Grid (RAG) model, LIT leadership reached out to Dr. Hollnagel directly
to discuss their interest and initial proof of concept for incorporating Safety-II into AA’s SMS. Dr.

9
Hollnagel met with LIT leadership in Copenhagen, Denmark, in August 2018, and made two
landmark recommendations:

1. Develop your own language based on your understanding of the how RAG will be used
in the cockpit. As the essence of the potential for resilient behavior only has context
within the trade space of the work being done, likewise the RAG model should be
adapted to meet the unique attributes of the work of piloting commercial airliners.

2. Devise your own data collection. Using data acquired via a data stream anchored in
“threats and errors” would be anathema to the appreciative mindset required for a
Safety-II approach. The recommendation was to begin the Safety-II program by creating
a separate, non-TEM driven data collection and analysis method.

The LIT group considered Dr. Hollnagel’s advice carefully, given AA’s safety leadership’s
intention to leverage LIT and LOSA data in the future. Most importantly, LIT recognized that
developing a thoughtful, deliberate program design would be imperative as they would be
blazing an entirely new trail within AA’s SMS as well as within the Safety-II and Resilience
Engineering communities.

Resilience Assessment Grid

Hollnagel’s RAG model consists of four capabilities: Respond, Monitor, Anticipate, and Learn
(Hollnagel, 2015). Hollnagel defines these capabilities as follows:

Knowing what to do, or being able to respond to regular and irregular variability,
disturbances, and opportunities either by adjusting the way things are done or by
activating ready-made responses. This is the capability to address the actual.

Knowing what to look for, or being able to monitor that which changes, or may change,
so much in the near term that it will require a response. The monitoring must cover the
system’s own performance as well as changes in the environment. This is the capability
to address the critical.

Knowing what to expect, or being able to anticipate developments, threats, and

opportunities further into the future, such as potential disruptions or changing operating
conditions. This is the capability to address the potential.

Knowing what has happened, or being able to learn from experience, in particular to
learn the right lessons from the right experience. This is the capability to address the
factual. (p. 279, author’s italics)

10
LIT Potentials
As its analog to Hollnagel’s capabilities, LIT designated the term “potential” to reflect the positive
and recurring action observed. In discussing the four RAG capabilities in the sharp-end context
of a pilot’s daily work, the same four plus two new LIT potentials emerged as interdependent
structures that interact in predictable ways. For example,

● Pilots monitoring notice alarms, anomalies, and situations that trigger their response.
● Slowing things down as a response mitigation strategy creates capacity for cognitive
processing of the situation, thereby facilitating effective learning.
● Lessons learned facilitate the ability to anticipate similar situations in the future.
● Anticipation of concerns, focus areas, and available options inform monitoring efforts.
● Leadership (or Lead) and Communication (or Communicate) seemed to potentiate and
facilitate success during the enactment of each of the other potentials.

From this point, the LIT model evolved in several ways to become what it is today (Appendix D).
In recognition of the fact that communication between pilots and other team members is only the
initiation of an interaction, and to reflect the forward-looking engagement that could be expected
with each potential:
● Coordinate was added to better capture the inherently cooperative aspects of safe
airmanship, both within the flightdeck and outside of it. It also reflects the understanding
that the LIT is rooted in the front-line work of pilot crews.
● Monitor was removed as a potential in its own right, as it was recognized as a vital
component of Coordinate.
● Adapt replaced Respond. Respond implies reactivity, a product behavior which is less
considered and less deliberate. Adapt more accurately describes pilot performance,
particularly as it relates to the active management of multi-factor trade-offs.
● Plan replaced Anticipate. While anticipation is an expert skill, recognizing the
proficiencies demonstrated by pilots who leverage anticipation to devise plans that
create resilient potential better encapsulates the work and mission of all pilots.

Early on, the team struggled with how to model Leadership and Communication, as they were
clearly essential among the crews who exhibited positive performance yet seemed difficult to
discretely observe or quantify. Eventually, the concepts of Lead and Communicate were
removed as directly observable potentials but retained as overarching principles that were
present in performance observations. The team continued to assert these concepts as critical to
the LIT project and envisioned them as the glue that held the four potentials together.

While the team considered whether the four core potentials might make sense as a linear
progression, it became clear that they were better envisioned each as a positive feedback loop
with the concept of resilience: their enactment created resilience and also received input from
other potentials via more resilient performance. In this way, the LIT model, with its four
potentials resting on an interconnected and interdependent foundation, visually reflects the
functional creation of the potential for resilient performance in any given situation. When further
envisioned as a three-dimensional model the complexity of the interconnectedness of the four

11
potentials and how they spark resiliency becomes evident. This model continues to evolve as
LIT better understands the relationships amongst the potentials.

Figure 2. AA’s LIT Model.

LIT Proficiencies
Hollnagel (2015) asserts that the details of each potential, that is, the specific functions that
enable a system to perform in a resilient manner, are more practical to address than the
potential as an idea or quality in and of itself. In essence, proxy measures are needed to
qualitatively assess the potential for resilient performance in a concrete, pragmatic way.

In an effort to use language unique to LIT while avoiding confusion or implication that these
proxy measures should be judged as best practices, the LIT group settled on the term
“proficiency” (rather than “behavior” or “competency”) to characterize the phenomena of resilient
performance. Early in the development of the LIT program, the LIT group surveyed and
interviewed more than fifty AA Check Airmen and LOSA Observers, recognized experts within
the pilot group who know what positive performance looks like in practice. It was initially decided

12
to focus on positive performance to better capture credible metrics in designing observable
criteria. Thematic analysis of this data provided content around which to explore LIT
proficiencies as they appear in normal work. Proficiencies themselves have undergone several
iterative revisions with the intent to optimize data capture, maximize inter-coder reliability, and
accurately reflect vital front-line work that contributes to success. LIT proficiencies as of
December 2019 can be found in Appendix C.

Leveraging LIT to Engineer System Resilience

In June 2019, the LIT group visited Dr. David Woods and Dr. Mike Rayo along with several
doctoral students at OSU’s Cognitive Systems Engineering Laboratory (CSEL). During the day-
long meeting, the team learned that recent developments in the field of resilience engineering
could be useful in offering additional theoretical grounding to LIT’s mission. Importantly, the LIT
group also re-centered on an essential tenet of resilience in practice: resilience is not a
component to be built into the organization, but the organization and its individual workers can
manage and foster the development of the potentials which will lead to resilient performance.

Resilience Engineering as a field seeks to 1) discover operational principles which allow

complex systems to adapt to the often-unforeseen conditions of changing worlds, and 2) design,
develop, and operate systems according to those principles. It recognizes that, though often
described in linear (or near-linear) terms, modern engineered systems are multi-scale, multi-
echelon, tangled, layered networks of interacting agents. The diversity of their components,
though often difficult to manage and coordinate, also offers the capacity to respond to much
wider range of events and challenges than more simple, homogeneous, or monolithic systems.

Recognizing that the variability of the world often means that systems (both engineered and
evolved) must successfully operate under a much wider and frequently more challenging set of
conditions than those for which they were created, resilience engineering begins with the
foundational expectations that such systems will be challenged at their operational boundaries.
Woods’ theory of graceful extensibility (Woods, 2018b) argues that those systems which
develop, manage, and retain the capability to flex, adapt, and grow at their limits in response to
challenges and changing conditions are more likely to be successful in their operational
missions and more enduring over time.

With this worldview in mind and using new language from resilience engineering, the LIT group
reexamined and refined their potentials and proficiencies, further adapted the data collection
tool for qualifying data, and cemented their determination to include robust narratives with each
data point in order to preserve the context of the observed situation.

13
Methods

Strategy
AA’s successful approach to Safety-II was tailored to both the aviation industry and specifically
the way AA conducts operations. As with all endeavors in AA Flight Safety, a dual-faceted
approach to leadership and oversight by AA and APA ensured mutual predictability and
direction for the LIT program.

Proof of Concept
AA’s robust, continuous LOSA program and its foundational TEM model served as the vehicle
for a Safety-II proof of concept. The LOSA data set contains threats, threats managed, and
errors captured by LOSA observers during routine flight operations. Threats managed, meaning
they did not lead to a crew error, indicate successful performance, or “what went right or well”
according to a Safety-II approach. Examining the LOSA data showed that over 80% of air traffic
control (ATC) threats were successfully managed and that more than 70% of all AA flights
encountered an ATC threat. ATC threats were selected as a Safety-II proof of concept data set.

A deep dive into this data set was performed by Lead Check Airmen (lead instructor pilots) from
each fleet, who extracted meaningful patterns within narratives from successfully managed ATC
threats during the previous 12 months. Three primary factors stood out among the LOSA
narratives: the crew’s experience level, their knowledge of the theater of operation, and,
perhaps most interestingly, the crew’s willingness to say ‘Unable’ in response to an ATC
instruction that was too challenging or placed the crew in a precarious situation. Exploration of
what factors positively and negatively contributed to this kind of open communication between
pilots readily yielded some teachable best practices. Satisfied that examining data for “what
went right or well” yielded leverageable insights, the LIT program was approved by AA’s Flight
Safety Leadership.

With the approval from the organization, LIT leadership sought the first of several academic
partnerships by initiating a conversation with Dr. Erik Hollnagel, whose works they were familiar
with. Heeding Dr. Hollnagel’s advice (discussed in the Safety Models section of this report), LIT
leadership chartered the first team members to create LIT from a blank slate, rather than
adapting from LOSA. Two seasoned LOSA observers were allocated paid time away from their
regular duties to research Safety-II concepts and apply them towards developing innovative
safety observations.

14
Data Collection

Observations
In March of 2019, the first LIT observations, informally called LIT rides, were conducted by two
former LOSA observers. Initially the observations were performed on pre-identified crews who
were selected due to their reputations as high performers as they operated during regular
scheduled flights. The intent of these early rides was to test whether flight observations
undertaken with an overtly positive, proactive approach could show measurable positive
resilient performance. Preliminary data as well as subjective feedback from crews and from LIT
observers themselves confirmed the test hypothesis, and the decision was enthusiastically
made to continue.

Data Collection Tool

The first observations, conducted in March and April 2019, were recorded in narrative form.
After approximately 10 observations the team determined that a more objective and data-
oriented approach would be more efficient for data capture as well as more useful for data
analysis. To meet this need, a unique AA’s LIT data collection tool (DCT) was developed within
Microsoft Excel. Rapid prototyping and testing of the LIT DCT produced no fewer than six
versions within eight weeks (Appendix E). Changes to the DCT fell into one of three categories:
enhancing the data input experience for the observers, adding or changing what data to collect,
or formatting changes to facilitate data extraction for database storage and future analysis.

Demographic data collected on each flight observation includes:

● Month and Year
● Aircraft type
● Crew base
● Departure and arrival ICAO airport designations
● Sequence day number of total days in that sequence
● Leg number of total legs that day
● Departure and arrival airport current weather
● Captain experience level, and whether they are a line or reserve pilot
● First officer experience level, and whether they are a line or reserve pilot
● Whether the captain or first officer was designated as the pilot flying or pilot monitoring

Space is allocated for narrative data per phase of flight, which may include special
circumstances or flight-related information related to the overall context of the flight.

LIT-specific data includes:

● Phase of flight
● Observed potential
● Observed proficiency
● Qualifiers
● Narrative description of the context in which the potential was observed

15
While Microsoft Excel spreadsheets are currently sufficient for data collection, eventually a built-
to-specification DCT is anticipated.

Qualifiers
In addition to observing potentials and proficiencies, LIT observers saw value in preserving two
notable artifacts from their experience as LOSA observers: measuring the connection of
observed proficiencies to each other and noting which if any outside influences were
encountered by the pilots in each situation. These qualifiers can be found in Appendix B, Table
B3.

Linkages
The interdependencies and interconnectedness of the potentials became apparent within the
first four months of conducting observations. The team struggled with how to best describe the
complexity of the relationships. Initial attempts noted if there was a linkage and then scored that
connection on a scale of 1 to 4 based on how complex the series of linkages were. It quickly
became evident that this was too subjective, so the collection method changed to listing which
potentials were linked by a labeling system that could be referenced in the analysis phase. Later
versions of the collection tool expanded to denote a “parent-child” relationship with the linkage
to better explain the interdependencies and complexity of day-to-day operations, and the current
version of the LIT model represents linkages with a backdrop upon which the four potentials
interact.

Preliminary data has shown approximately 15% of observed potentials appear in series with
other potentials. An example of this phenomenon is shown in Figure 3.

Figure 3. Linked potentials.

Pressures
In LOSA and TEM, a threat is any event that originates outside of the flightdeck that increases
operational complexity and must be managed to maintain the margins of safety (IATA 2013). In
recognition of this complexity as it relates to the potential for resilient performance, LIT includes
many of the same such events in the data collection. However, instead of considering them as

16
events to be managed, LIT views them as influences or pressures that impact the crew’s task
loading and prioritization, which can be found in the qualifiers in Table B3 of Appendix B.

Pressures are important contributing factors that drive timely, responsive decisions. In addition
to the pressure itself, the degree to which time is affected by the pressure is also noted. While
originally a 4-point scale which noted the pressure on a scale of simple to complicated and its
effect on time from no effect to significant effect, the LIT group sought to increase reliability and
objectivity by simply noting whether or not time was affected, i.e. the pressure’s impact on the
crew’s task loading and prioritization.

Radar Plot
A radar plot was included in each observation to give a subjective snapshot of how the observer
rated the crew’s resilient performance across the four potentials. The radar plot was inspired by
Hollnagel’s RAG model and served as an analogous feature to a subjective section of the LOSA
data collection tool. While there is no standard or norm against which to rate the potential for
resilient performance across the four potentials, each observer rates the potentials in
comparison to all of the observations previously completed. The LIT group is still considering
how this feature may be employed.

Version Control
With so many iterations of the DCT being used over time by LIT observers, it was necessary to
scrutinize the data as previous observations were re-coded. LIT observers took into account
known cognitive influences such as memory recall degradation as they re-coded their own
observations. The team’s research intern and data analyst reviewed and double-checked all re-
coding and version control efforts.

Additionally, throughout the DCT version changes, the team was continuously collecting data
with either the newest version or the most previous version. While continuity posed a challenge
in this situation, clear and timely communication among LIT group members quickly mitigated
any confusion that arose. The LIT data analyst maintained a Microsoft SharePoint database of
all original data collected from observers as well as re-coded observations, while serving as a
central point of contact for data consolidation.

Interviews
The team also expanded the collection method to include pilot interview sessions called “Shop
Talk”. These were conducted during downtime at recurrent simulator training or at other
scheduled periods. Approximately 30 minutes in length, they provided an opportunity to delve
into potentials deeper by examining how the crews analyzed and dealt with unique
circumstances such as diverting to another airport, approach go arounds, emergency situations
and other instances where the proficiencies could be observed. Following guidance from Klein
et al. (1989) Critical Decision Method and Flanagan’s (1954) Critical Incident Technique,
questions were designed as open-ended probes to spark dialogue to better understand how

17
pilots think and make decisions. Confidentiality of data and anonymity of participants was
ensured.

Interview Data Collection Guidance

Interviews with crews began as post-flight conversations during which observers asked
questions (Appendix F) to better understand how the crew thinks and adapts. LIT observers
were coached by their research intern in basic cognitive interviewing techniques1 such as asking
“how” as opposed to “why,” using non-confrontational prompts such as “walk me through your
thought process” as opposed to “why did you do that,” and simple techniques for increasing
accuracy of the interviewee’s memory recall surrounding a decision.

In an effort to be sensitive to the word “interview” as potentially confrontational, the LIT group
was careful to avoid the word and instead referred to these as “conversations,” “learning
sessions,” and “informal debriefings”. In addition to post-flight conversations, observers at the
Charlotte, North Carolina and Fort Worth, Texas training centers conducted “Shop Talk”
interviews with pilots who had arrived early for their recurrent training; these sessions were
usually conducted via convenience sampling in the training centers’ cafeterias.

Recording these conversations was discussed as a means to optimize efficiency and

interviewer engagement while minimizing interviewer recall error and inadvertent analysis or
bias during transcription from handwritten notes to typed reports. One LIT observer asked
several of his interviewees if they would mind being recorded for the above reasons, with the
assurance of confidentiality and anonymity, and each indicated their hearty consent and support
of recording. After transcribing the notes from the learning session, the observer destroyed the
recording to ensure that privacy concerns were met. Due to data privacy concerns regarding an
off-site, contracted service proposed for transcription of these recordings, no automated
transcriptions were made.

Data Analysis

Inter-Coder Reliability
Because this type of data analysis is susceptible to the introduction of data coding and
categorization errors, in August 2019, the observer team underwent an inter-coder reliability
(ICR) exercise to gauge the validity and reproducibility of their data collection methods.
Standardization amongst the three observers was necessary in order to develop a confidence
factor for the data that was to be presented. The exercise consisted of two parts: 1) each
observer coding one of their peer’s observations solely based on their peer’s description of each
data entry, and 2) each observer coding prose narratives of two flights that none of them had
observed. These two arms of the ICR facilitated simple agreement matrices and original
selection coding, respectively. The agreement matrix analysis showed that all three observers

1
See Fisher & Geiselman (1992), Klein (2001), Madans et al. (2011), Dominguez et al. (2016), and
others.

18
agreed 56% of the time while coding potentials, and two of the three agreed on the potential
95% of the time. When all three observers agreed on the coding of a potential, they agreed 53%
of the time on the proficiency as well. The team was encouraged by these results, following the
analysis with discussions of why each observer coded the data as they did.

Between August and December 2019, the LIT group initiated two additional agreement-matrix
ICRs as well as several proficiency language reviews. A deep dive into the first 100 flight
observations is planned for Phase 4 (Figure D.1., Appendix D), during which any outlying,
infrequently used, or poorly worded proficiencies will be reworked or discarded.

Analysis
Preliminary data analysis began concurrently with preliminary observations, thus facilitating the
rapid prototyping of the LIT DCT. Early analysis efforts focused on streamlining data collection
itself as well as formatting the data collection for maximum analysis capability in the future.

Data analysis of the first nine months of observations is encouraging: experienced AA pilots, AA
senior leadership, and aviation experts agree that the emerging patterns within LIT data are
logical and reflect their lived experiences as flight crew members.

This report deliberately excludes a robust explanation of results from these observations for
several reasons, most notably that the iterative changes to our data collection strategy required
re-coding several times, thus statistically diminishing the internal validity of the data regardless
of our diligence during re-coding as well as our efforts to improve inter-coder reliability.
Following a robust data review during Phase 4, we believe these concerns will be addressed
and future observations will contribute to an increasingly reliable data stream.

LIT Observers

Selection
As a result of their extensive experience with gathering data in a narrative format, LOSA
observers were recruited to join the team and participate in LIT data collection. While the first
three observers were hand-selected from within the LOSA program, the LIT group now hires
new observers via applications from current line pilots, review of the pilot’s record with AA, his
or her resume and cover letter, recommendations from senior colleagues, and a multi-step
interview process. The new hiring approach reflects the intent to add new team members who
are not entrenched in the compliance-focused mindset and TEM lens that are necessary for
effective LOSA observations. LIT sought recruits directly from the line pilot cadre to bring fresh
perspective and an open mindset to this new safety program. Additionally, the team believed
that the learning curve to successfully viewing safety through a more appreciative lens may be
less steep for new observers not previously trained in LOSA methodology. This hypothesis is
supported anecdotally by the LIT group’s most recently hired observers.

19
Training
The first two LIT observers trained the third didactically via explanation and discussion of LIT’s
mission, safety concepts, and strategies for data collection. The third LIT observer was then
mentored by the first two on several observation write-ups until the new observer felt competent
enough to perform solo observations.

The first formal training of new LIT observers occurred with the onboarding of the fourth and fifth
observers. Training included a review of the program’s inception and development, introduction
to LIT model and terminology of Safety-II, observation and jumpseat protocol, observation
schedule and Federal Aviation Regulation (FAR) Part 117 compliance (flight, duty, and rest
regulations), and practice with the LIT DCT. Trust was stressed as a central tenet of
observations: flight crew participation is always voluntary, confidentiality is assured at all times,
and anonymity is preserved. These attributes of the crew-observer relationship are familiar to
AA’s pilots due to the longstanding and well-received continuous LOSA program. The team
recently announced plans to add two new observers by the end of 2020. Training will follow the
same methodology noted above with improvements based on feedback from the newest
observers.

Implementation
Observations and interviews have continued without interruption for nearly ten months,
garnering support and interest from AA’s team members vertically and horizontally throughout
the organization. The team completed its first 100 observations and 10 interviews (post-flight
and Shop Talk sessions) in early 2020. Data analysis results will be the basis of a forthcoming
academic paper planned for release in 2021 or 2022.

Value
A primary aim of the LIT program is to increase organizational knowledge about work as it is
done on the flightdeck, with an eye towards how pilots think about safety. With robust data
collection, LIT anticipates being able to translate patterns in observed potentials and
proficiencies into opportunities for pilots to increase their clarity and ability to communicate their
mental models. LIT believes that building adaptive capacity among pilot teams is possible by
designing opportunities to foster and manage potentials for resilient performance. At this time,
these opportunities lie with training, informal and formal discussions at various levels of the
organization and exploring process change.

20
Dissemination

Pilot Training
Early brainstorming revealed the value of using LIT data to share these new safety concepts
with AA’s pilots. Examples of resilient performance, case scenarios observed during LIT rides,
and insights from LIT “Shop Talk” conversations were identified. Recurrent human factors and
quarterly distance learning are also targeted as future platforms to expand LIT principles.

Captain Leadership Training

Two Federal Aviation Administration (FAA) Advisory Circulars (AC) are currently in draft form
which will outline guidance for air carriers to develop and implement two training programs:
1. Leadership and Command Training for Pilots in Command, which presents guidelines for
developing and implementing leadership and command training for pilots in command
(PIC).
2. Air Carrier Pilot Mentoring, which presents guidelines for developing and implementing
mentoring training for PIC. This draft AC also presents guidelines for implementing a
Pilot Professional Development Committee (PPDC) to develop, administer, and oversee
a formal pilot mentoring program.

AA currently conducts a three-day Captain Upgrade Leadership training program. In order to

comply with the above two ACs, AA will revise this program by incorporating LIT data. The LIT
model provides a clear curriculum outline to meet the objectives of both mentoring and
leadership / command training.

Shop Talk questions, including those relating to captain leadership, can be found in Appendix F.

Formal and Informal Discussions

Part of the LIT group’s organizational charge is to inspire AA’s team members to think differently
about work and about the safety of their work. After LIT observations commenced in the spring
of 2019, news of the program has spread by word-of-mouth throughout the pilot corps. Pilots
engage eagerly with observers in the vast majority of LIT rides, with the remainder reporting
being pleasantly surprised after receiving more information and experiencing a LIT ride for the
first time. Acknowledging that organizational culture is universally difficult to change, the LIT
group and AA’s leadership are encouraged by the relative ease of adoption of this program by
the pilot group.

Safety Preflight is AA’s monthly internal flight safety magazine. LIT was invited to contribute to
the magazine under a new section for Safety-II. The first article appeared in the February 2019
issue which served as an accessible introduction to Safety-I and Safety-II, as well as the initial
efforts to extract examples of resilient behavior from within LOSA narratives. In May 2019, the
first official LIT article was published which explained the concepts of Safety-II and the early
language construct. The Learn potential was explained in an August 2019 article, and Plan was

21
explained in the January 2020 edition. Future follow up articles are planned on a quarterly basis
to include sharing examples of some of the best practices observed during line operations.

Additionally, several presentations internal to AA were given in August and September of 2019.
These outlined the early work of LIT and provided some examples of how the data may be
effectively used.

LIT also presented at and is invited to present at the following professional gatherings:
● 1st International Workshop on Safety-II in Practice with Erik Hollnagel, St. Petersburg,
Florida, February 2019
● 2nd International Workshop on Safety-II in Practice with Erik Hollnagel, Lisbon, Portugal,
September 2019
● IATA Issue Review Meeting (IRM), Luxembourg City, Luxembourg, September 2019
● 2019 Aviation Safety Infoshare, Seattle, Washington, October 2019
● AA SMS Industry Forum, Dallas, Texas, March 2020 (canceled due to COVID-19)
● Safety-II in Practice with Erik Hollnagel, Edinburgh, Scotland, October 2020

Potential for Process Change

Through the success of LIT’s post-flight interviews and Shop Talk sessions, much insight has
been gleaned from pilots of all experience levels and in both the captain and first officer roles. A
common theme throughout the interview responses was a sense that a semi-structured
conversation between the PF and PM would be beneficial to knowledge discovery and sharing,
sensemaking (individual and joint), and collegiality between roles. Since AA does not currently
formalize a post-flight debrief as other commercial airlines are known to do, there is an
opportunity to, at minimum, offer support and direction to pilots who wish to engage in a
professional, performance-reflective conversation either pre- or post-flight.

Below is a sample of the kinds of questions asked of pilots by LIT observers.

Do crews ever discuss with each other past events they have learned from? Do they brief how
they plan to handle the same event now? How do they implement these lessons learned?

Talk about debrief / learning culture. Do you have any feedback on how to improve the learning
culture at AA?

How would you feel about adding a debrief function to discuss any issues that come up during a
flight?

How would you create time and space for debriefs during everyday line operations?

As their name reflects, the LIT group believes that the Learn potential is of critical importance.
With the expected retirement of senior pilots over the next decade, harnessing opportunities to
learn from these vastly experienced professionals is a timely endeavor. Ideally, AA can turn
these lessons learned into valuable teaching tools for the anticipated influx of new talent. LIT
seeks to answer questions of where and how learning takes place on the flightdeck and how it
can be facilitated in a light, spontaneous manner, without tedious formalization.

22
Towards Organizational Integration
The LIT program is approaching completion of the first step towards demonstrating value to the
organization: connecting a dataset from which statistical analysis of coded data can achieve
statistical and practical significance. The team continues to identify meaningful patterns within
the data, though vigorous data review is needed before it is shared externally.

23
Conclusion
American Airlines’ Flight Safety and LIT Leadership group have endeavored down an
aggressive and meaningful journey into Safety-II foundations to learn more about pilot
performance, system complexities, and safety system design in large, modern, global,
commercial airline operations. The journey has not been simple or straightforward; the purpose
of documenting these efforts to date is for others to gain insight into the successes as well as
lessons learned. All indications are that senior leadership at AA will officially adopt and formalize
LIT as a Safety-II data stream and long-term contributor to AA’s SMS process within the next
fiscal year. This adoption will be a monumental success for the development and evolution of
the LIT effort at AA. The primary aim of LIT remains to learn and share expertise leveraged by
individuals, their interactions, and the overall system, towards the ultimate goal of improving
safety and efficiency during daily work on the flightdeck.

LIT leadership and core members are encouraged by preliminary analysis results of data from
the original 100 observations collected throughout 2019 and early 2020. The connections,
linkages and demonstration of the shared mental model concept by outstanding AA flight crews
is clearly evident and quantifiable through a newly developed language of Potentials and
Proficiencies. AA’s LIT members developed this new language upon recommendation from Dr.
Erik Hollnagel in early discussions on the program development planning. The learning and
introspection that occurred during the language development phase of the current process was
extremely valuable to the team members and to AA as a whole. Applying the lessons learned
and insight to the complex system that is modern worldwide airline operations is the next
challenge.

AA’s LIT Leadership team and senior management envision great promise in the introduction of
this new and enlightening data stream. The opportunities to share hard-earned lessons from
more experienced aviators to new-hire pilots over the next few years are exciting and even
paradigm-shifting for traditional training and Human Factor scenarios. The level of training and
indoctrination that is required at most U.S.-based modern airlines is daunting, and this new
knowledge and practical approach to accomplishing work is key to improving safety and
subsequently reap the benefits of a better and more supportive system within which to operate
aircraft.

The data currently being collected and analyzed by AA’s LIT program is ground-breaking and is
already providing enhanced knowledge and insight to facilitate AA’s transition to the next level
of safety. Continued analysis of this critical data stream is necessary to fully understand and
facilitate the potentials for resilient performance among modern flight crews in their own domain.

24
Appendix A. Evolution of LIT Model: Potentials
Table A1. Original LIT Model, May 2019.
RAG Capability AA v1.0 AA v1.0 Definition AA v1.0 Tagline Rationale
Capability* Definition* Potential

Respond Knowing what to Respond Effectively react to normal Act given current condition Mindset of TEM
do, being capable triggers, alarms, threats and
of doing it anomalies SLOW THINGS DOWN

Monitor Knowing what to Monitor Proactively watch for normal Watching now Daily work of a pilot
look for triggers, alarms, threats and
anomalies STAY ENGAGED

Anticipate Finding out and Anticipate What could happen? What ifs Thinking in future Resonates with newly adopted
knowing what to Threat-Forward Briefing
expect THREAT FORWARD- TALK

Learn Knowing what has Learn Active process of improving future What will I do/ change next Continuous quality improvement
happened performance. When I was last time? mindset as well as personal
here X happened, and this time experience
will do Y. SHARE EXPERIENCE

N/A N/A Leadership Professional and builds a team Professional Extrapolated from Check Airmen
effectively interviews and LOSA Observer
data as well as personal
experience

N/A N/A Communication Proactive and open environment. Open and Proactive Extrapolated from Check Airmen
CA sets the tone at start of interviews and LOSA Observer
sequence data as well as personal
experience
*Hollnagel, 2015.

25
Table A2. LIT Model Evolution.
v1.0 Potential v2.0 Potential Rationale for v3.0 Potential Rationale for v4.0 Potential Rationale for
May 2019 June 2019 Change September 2019 Change October 2019 Change
Respond Adapt Respond implies Adapt No change Adapt Updated
reactivity. Adapt graphic, no
Effectively react to Effectively react to normal more accurately Effectively react to normal Effectively react to normal change to
normal triggers, triggers, alarms, threats and describes pilot triggers, alarms, threats and triggers, alarms, threats language
alarms, threats and anomalies. performance. anomalies. and anomalies.
anomalies Removed
Recognize disturbances / Recognize disturbances / Recognize disturbances / “tagline” for
pressures to the plan as they pressures to the plan as pressures to the plan as professional
arise. Evaluate, respond, and they arise. Evaluate, they arise. Evaluate, audience
intervene to minimize impact respond, and intervene to respond, and intervene to
of the disturbance or redirect minimize impact of the minimize impact of the
Act given current to positive outcome.  disturbance or redirect to disturbance or redirect to
condition positive outcome.  positive outcome. 
Act given current condition
SLOW THINGS Act given current condition
DOWN SEMPER GUMBY
SEMPER GUMBY

26
v1.0 Potential v2.0 Potential Rationale for v3.0 Potential Rationale for v4.0 Potential Rationale for
May 2019 June 2019 Change September 2019 Change October 2019 Change
Monitor Coordinate Monitor joined Coordinate No change Coordinate Updated
Communicate to graphic, no
Proactively watch for Proactively build the team become Proactively build the team Proactively build the team change to
normal triggers, and establish a shared Coordinate. and establish a shared and establish a shared language
alarms, threats and mental model. Utilize all mental model. Utilize all mental model. Utilize all
anomalies available resources, both available resources, both available resources, both Removed
internal and external. internal and external. internal and external. “tagline” for
professional
The response to the altered The response to the altered The response to the altered audience
plan with other team plan with other team plan with other team
members. Ensure the crew members. Ensure the crew members. Ensure the crew
has shared a mental model has shared a mental model has shared a mental model
for getting the plan back on for getting the plan back on for getting the plan back on
track or normalized as quickly track or normalized as track or normalized as
as possible. Continue until quickly as possible. quickly as possible.
the plan has stabilized.  Continue until the plan has Continue until the plan has
stabilized.  stabilized. 
Watching now Get the team on the same
page Get the team on the same
page
STAY ENGAGED STAY ENGAGED
STAY ENGAGED
Anticipate Anticipate Expanded Plan Planning more Plan Updated
language for fully graphic, no
What could happen? What could happen? clarity. What could happen? encapsulates What could happen? change to
What ifs What ifs What ifs pilots’ work and What ifs language
mission
Create, discuss, initiate and Removed
monitor in action for “tagline” for
disturbances. Proactively act professional
Thinking in future to keep it on track. audience

THREAT- Thinking in future Thinking in future

FORWARD TALK
THREAT-FORWARD TALK THREAT-FORWARD TALK

27
v1.0 Potential v2.0 Potential Rationale for v3.0 Potential Rationale for v4.0 Potential Rationale for
May 2019 June 2019 Change September 2019 Change October 2019 Change
Learn Learn Expanded Learn No change Learn Updated
language for graphic, no
Active process of Active process of improving clarity. Active process of improving Active process of improving change to
improving future future performance. When I future performance. When I future performance. When I language
performance. When was last here X happened, was last here X happened, was last here X happened,
I was last here X and this time will do Y. and this time will do Y. and this time will do Y. Removed
happened, and this “tagline” for
time will do Y. Reflect on outcome of the Reflect on outcome of the Reflect on outcome of the professional
disturbance / pressure for disturbance / pressure for disturbance / pressure for audience
future planning. Review future planning. Review future planning. Review
materials / flight documents / materials / flight documents materials / flight documents
previous lessons learned / previous lessons learned / previous lessons learned
What will I do / before making the next plan. before making the next plan. before making the next
change next time? plan.
What will I do / change next What will I do / change next
SHARE time? time?
EXPERIENCE
SHARE EXPERIENCE SHARE EXPERIENCE
Leadership Leadership / Professionalism No change Leadership Foundational, Leadership Updated
not a Potential graphic, no
Professional and Professional and builds a Professional and builds a Professional and builds a change to
builds a team team effectively team effectively team effectively language
effectively
Communicate N/A Communicate Communicate Foundational, Communicate Updated
joined Monitor to not a Potential graphic, no
Proactive and open become Proactive and open Proactive and open change to
environment. CA Coordinate environment. CA sets the environment. CA sets the language
sets the tone at start tone at start of sequence tone at start of sequence
of sequence

28
Appendix B. Evolution of LIT Model: Proficiencies and Qualifiers
Table B1. Proficiencies.
Learn Apply what was previously learned
Demonstrates a positive interest in acquiring knowledge and improving
Used some sort of debrief to discuss what went well or could have been handled differently
Verbalize what was previously learned
Other
Coordinate Ask ATC for clearance verification or request relief of restriction
Ask other crew member for input or assistance
Briefs or gives new info / update to other member or adds info to build SA
Effective teamwork, delegate and divide tasks (Divide and Conquer)
Monitor automation, PM or PF deliberately references FMA or other aircraft system
PM or PF reviews, verifies and cross-checks other’s actions deliberately
Re-centers crew to SOP adherence when deviation observed
Update other pilot of change to plan to build shared mental model, ask if unsure
Used Jeppesen charts or other information to monitor route and follow along
Other
Adapt Address unanticipated new pressure
Adjust communication method or pauses based on the other pilot’s workload
Change automation level / mode / programming for changing condition
Delay task until more appropriate time
Heightened awareness or focus before non-standard or complex task (MEL, revised ATC clearance, runway or approach change)
PF or PM intervention for unwanted condition
Pilot assertive in addressing change in task loading (color coding language, heads up, etc.)
PM initiates action to decrease PF workload before being asked
Slow down pace or stop. Push back on external stimulus or create more time.
Other
Plan Conducts a thorough briefing
Develops “what if” scenarios and plans for contingencies
Discuss expected actions or plan beforehand (taxi route, clearance, STAR, approach, runway change, TPS, tailwinds, 10-7 info)
Establish countermeasures to pressures, enhance future SA (reminders, secondary flight plan, fixes)
Gather information from internal or external sources (ATC, EFB resources, etc.)
Prioritizes and schedules tasks
States expectation for flight with other crew members
Other

29
Table B2. Archived proficiencies.
Leadership Acknowledge and thank crew members for efforts
CA adjusts to experience level of crew
CA empowered FO to make decisions regarding flight
CA empowered the FO to speak up and created a more candid environment.
CA levels authority gradient to enhance open comms
CA recognized and complimented FO
CA sets positive and open communication tone
Change duties to adapt to workload or environment / situation
Conducts a thorough flight attendant briefing
Deals with adversity or stress in a courteous and respectful manner
Delegate where appropriate
Enforces sterile cockpit
Keeps crew and / or external stakeholders updated as plan or conditions change
Keeps entire crew and dispatch informed of changes to plan and schedule
Professionally explains operational philosophy and goals during initial introduction to other crew member
Provides tools for crew members to accomplish tasks
Shows enthusiasm at being a leader and staying on top of the situation
Solicits inputs from other crew members
States expectation crew will not be rushed and slows things down when pace begins to overwhelm the crew
States expectation for sterile cockpit
Stop interruptions to flightdeck duties by closing door, stopping FAs, pre- briefing no interruptions to briefs or checklists
Takes responsibility for mentoring and professionally developing other crew members
Talks to passengers frequently and clearly to inform and update
Uses EFB or other tools to enhance crew members' SA (FAs). Professional cabin crew brief
Communication Ask for clarification on item
Ask for info or prior experience from other crew member if unsure
Conducts a thorough flight attendant briefing
Ensure all pilots have the same shared mental model
Gives update or new info to group outside flightdeck (FAs, Maint or Dispatch)
Keeps crew and / or external stakeholders updated as plan or conditions change
Pilot assertive in addressing concern area or status (color coded language, heads up, etc.)
Tell other pilot the plan to build shared mental model
Work to build rapport with soft people skills. Demonstrates empathy, respect and tolerance for other people

30
Table B3. Qualifiers.
Phase of Flight Crew report to door closing
Pushback
Taxi out to runway
Takeoff and clean up
Climb to TOC
Cruise
TOD to FAF (STAR)
Approach and Landing
Taxi in and Park
Pressure Aircraft Mechanical
Airport
ATC
Automation
Cabin (Flight Attendants or Passenger)
Dispatch or Paperwork
Environment (air or ground traffic, terrain)
Ground or Ramp
Maintenance
Ops Pressure (Gate agent, CS, operational changes)
Weather Impact (Gusty winds, tailwind, convective, turbulence)
Time Pressure did not impact crew's task loading and prioritization
Pressure did impact crew's task loading and prioritization

31
Appendix C. Knowledge Elicitation
Table C1. Summary Responses from Check Airmen and LOSA Observers.
Question Summary Responses

What are the best Pilot monitoring behaviors and actions you PM follows along closely via FMS / charts / CAMI to “fly” along with PF
have observed other than those outlined in SOPs? Assertive backup that is spoken.
Always engaged.
Speaks up to ask PF what the plan is if unsure.
Anticipate threats and ATC needs before being asked by PF.

How do crews best anticipate what could happen before Leverage Experience.
departure, enroute and before arrival? Use engaged threat forward briefing.
Verbalize What ifs.
Constantly collecting info and processing with understanding of cause and effect relationships

If high performing crews communicate more effectively than Open communication.

others, what makes their communication more effective? CA sets the tone at the start.
What do they do best? Encourages assertiveness and creates an environment where speaking up is not seen as
criticizing nor second guessing pilot. Rapport is built.
PF tells plan and thinks out loud so PM knows when something is not going according to plan
and can intervene.

How do crews deal with feeling rushed and compressed for Create more time by stopping or slowing down.
time? How do they slow things down or other positive Use color coding language.
behaviors when task loading increases? Delegate.
CA sets tone to bring everyone back into green.
Be aware of task loading of other pilot.

What barriers have you seen regularly employed to prevent Tell external influence to wait.
distractions and interruptions? How do crews deal with Use nonverbal communication.
them? Create space and time.
Delegate or delay duties.

Do crews ever discuss with each other past events they have Opportunity to improve in this area but those who do show learned behaviors talk about
learned from How do they implement these lessons previous experiences during briefings or when asked by other pilots.
learned?

32
Appendix D. Figures

Figure D1. AA’s LIT Program Design and Implementation Schedule.

Figure D2. AA’s LIT Model, Version 1.0, May 2019.

33
Figure D3. AA’s LIT Model, Version 2.0. June 2019.

Figure D4. AA’s LIT Model, Version 3.0, September 2019.

34
Figure D5. LIT Model, Version 4.0, October 2019.

35
Appendix E. LIT Data Collection Tools
Table E1. LIT DCT Evolution.
Version Month Collection of Data Data Set Proactive Formatting of DCT

0 March Format facilitated natural thought Narrative, setup and description of start Formatting not considered.
progression of observer of sequence, phase of flight in which
behavior was observed

1.0 April Short narrative blocks Guidance to consider potentials: Microsoft Excel for ease of data
respond, monitor, anticipate, learn, analysis
leadership, communicate

2.0 May Listed interview questions Standardized interview questions

3.0 June Drop-down menus of phase of flight, Collect flight specifics and
potentials, and proficiencies environmental information

4.0 June New potentials: anticipate, monitor,

respond, learn

5.0 June New potentials: adapt, coordinate, plan,

learn
Removed leadership, communication as
potentials
New proficiencies per potential
Radar plot for subjective assessment of
potentials
Added pressures and pressure score
Added outcome score based on linkage
complexity

36
Version Month Collection of Data Data Set Proactive Formatting of DCT

6.0 July Alphabetized lists Added Automation to Pressures Hid cell outlines as an alternative to
Hid extracted-data tabs Deleted N/A options merging cells
Time and Outcome levels changed to Pressure score (4 options) changed to Asked observers to update tab name
be more intuitive. E.g. 0 = time not Time Affected (yes / no) with observation number
affected; 2 = 2 potentials used. Outcome score descriptions modified Created extracted-data tab for easier
Changed “competency” to “proficiency” manual database input
in this version Background calculation added to
extracted-data tabs to prevent
“0” from being recorded as “null”

7.0 August Significant interface redesign Set print areas

Changed file naming convention

8.0 October Combined preflight phases into crew Added experience input similar to LDCT Added proficiency label column to
report to door closing Removed outcome score aid with data extraction
Combined approach and landing Added reference to proficiency label for
phases into terminal phase linkage.

9.0 November Linkages changed to Parent-Child DataID reflects option for Parent-
Proficiencies revised after data review Child linkage

9.1 December Added notes for data entry assistance Minor changes to Coordinate
proficiencies

37
Appendix F. LIT “Shop Talk” Cognitive Interview Questions
Table F1. All forms of question probes, by question category. Not all questions were asked during each conversation.

Category Question

Adaptations Can you tell me about an event or incident you had where you had to deviate or adapt from a policy in some way?
Was there ever a situation at work where you had to adapt, or things didn’t go according to plan?
Tell me about a time you experienced a situation that required significant adaptation, or you could not follow the
textbook procedure to respond to the situation?
Can you recall a situation you and your first officer experienced that required significant adaptation?
Can you tell me about an event you had that required severe adaptation or your response was not textbook?

Captain What is the hardest part about being a CA?

What was most challenging for you about becoming a captain?
As you approach retirement, is there any advice you or knowledge from your experience as a captain that you can
pass on to new captains?
Knowing what you know now, is there anything you would have liked to have been told when you first upgraded?
Can you tell me about some of your favorite captains and what you enjoyed or liked about them?
Can you tell me about some captains that were poor leaders or ineffective?
What should new CAs know? How should we adapt training for them?
What, if any, leadership training would you like to see at the company?
What kind of company leadership training would be helpful to you when you upgrade?
What content would you like to be included in leadership training?
What could the company do better to prepare you for captain?

First Officer What’s the hardest part about being an FO?

Are there certain things your captains do that bother you or reduce overall TEM?
Are there certain things your captains do that increase or are beneficial to TEM?

High functioning crew What makes a high functioning crew?

Why are some crews higher functioning than others?

38
Category Question

Debrief How would you feel about adding a DEBRIEF function to discuss any issues that come up during a flight?
How would you feel about adding a post-flight debrief to discuss issues that come up during a flight?
How do you learn from your previous flights or outcomes?
Have you engaged in any form of debrief with the crew? How did it go? Do you have any suggestions on how to
implement that within our pilot group? How to create time and space for that during a typical line flight?
Understanding the potential negative aspect and cultural shift of adding a de-brief, or some mechanism for a
discussion, how would you incorporate that into everyday work?
How would you create time and space for debriefs during everyday line operations?
How would you create time to conduct a debrief?
If we were to add a debrief during the flight, how would you find time to have it?

Learning culture How do you think the company can improve the learning culture?
How would you characterize the learning culture at AA?

Operations Have you had any go arounds lately?

How do you deal with the clean ramp policy?
How do crews deal with feeling rushed and compressed for time?
How do you deal with feeling rushed or pressed for time?

39
References
American Airlines. (2019). “Threat and Error Management.” Flight Operations Manual, Revision
6, Chapter 19. Retrieved from AA Pilots’ internal website.

Berntzen, E. (2019). Normal people doing normal work. Wideroe Crew Training Department.

Busch, C. (2019). Brave New World: Can Positive Developments in Safety Science and Practice
also have Negative Sides? MATEC Web of Conferences, 273, (no page numbers).
https://doi.org/10.1051/matecconf/201927301003

Dominguez, C., Klein, G., Fallon, C., & Militello, L. (2016). Cognitive Systems Engineering
Workshop (slide deck). Retrieved from
http://www.cognitivesystemsdesign.net/Workshops/Dominguez et al Workshop Slides.pdf

Federal Aviation Administration. (2019). Advisory Circular (Draft): Leadership and Command
Training for Pilots in Command. Retrieved from Federal Aviation Administration website.

Fisher, R. P., & Geiselman, R. E. (1992). Memory-enhancing techniques for investigative

interviewing: the cognitive interview. Thomas.

Geiselman, R., & Fisher, R. P. (1988). The cognitive interview: An innovative technique for
questioning witnesses of crime. Journal of Police and Criminal Psychology, 4(2), 2–5.
https://doi.org/10.1007/BF02806548

Hollnagel, E. (2013). A tale of two safeties. Nuclear Safety and Simulation, 4(1), 9.

Hollnagel, E. (2015). Introduction to the Resilience Analysis Grid (RAG). Retrieved from
erikhollnagel.com.

Hollnagel, E. (2014). Safety-I and Safety-II. The Past and Future of Safety Management, CRC
Press. Boca Raton, FL.

Hollnagel, E. (2018) Safety-II in Practice. Developing the Resilience Potentials, Routledge,

Taylor & Francis Group, New York.

Hollnagel, E., Leonhardt, J., Licu, T., & Shorrock, S. (2013). From Safety-I to Safety-II: A White
Paper. Eurocontrol.

40
Hollnagel, E., Pariès, J., Woods, D., & Wreathall, J. (2011). Resilience engineering in practice:
A guidebook. Retrieved from http://ebookcentral.proquest.com/lib/ohiostate-
ebooks/detail.action?docID=615608

IATA. (2013). Evidence-Based Training Implementation Guide, 1st Edition. IATA.

Klein, G. (2001). A One-Day Workshop for Teaching Cognitive Systems Engineering Skills: Job
Aid for CDM Interviews. Cognitive Systems Designs.

Klein, G. A., Calderwood, R., & Macgregor, D. (1989). Critical Decision Method for Eliciting
Knowledge. IEEE Transactions on Systems, Man, and Cybernetics, 19(3), 462–472.

Lundahl, M. (2016). Runway Incursion Prevention, A Safety-II Approach. HindSight 24, 46–49.

Madans, J., Miller, K., Maitland, A., & Willis, G. (2011). Question evaluation methods:
contributing to the science of data quality. Wiley.

Mouton, G. (2019). LOSA Safety and a ’66 Corvette. American Airlines Safety Preflight
Magazine, 8–9.

Null, C. H., & Holbrook, J. (2018). NASA Engineering and Safety Center Technical Assessment
Report: Human Performance Contributions to Safety in Commercial Aviation. NASA.

Provan, D. J., Woods, D. D., Dekker, S. W. A., & Rae, A. J. (2018). Safety Differently
Professionals: How resilience engineering can transform safety practice (PhD Thesis). Griffith
University.

Shorrock, S. (2016). The Varieties of Human Work. Retrieved from

https://humanisticsystems.com/2016/12/05/the-varieties-of-human-work/

Swauger, S. (2018). Human Resilience in Flight Operations (HRFO). Southwest Airlines Pilots.

Thompson, A. (2019). The Crew Debrief (p. 9). United Airlines.

Van Drie, KD. (2002). Standards of Performance and Data Collection Using the Volant Model
(slide deck). Retrieved from https://documents.pub/document/volant-model-for-airline-crew-
resource-management.html

Woods, D. D. (2018). Resilience is a Verb. In B. Trump, M. Florin, & L. Linkov (Eds.), IRGC
resource guide on resilience (vol. 2): Domains of resilience for complex interconnected systems.
Retrieved from igrc.epfl.ch

Woods, D.D. (2018). The theory of graceful extensibility: basic rules that govern adaptive
systems. Environment Systems and Decisions 38(4), 433–457.

41
Glossary
Adapt Effectively react to normal triggers, alarms, threats and anomalies.
Recognize disturbances / pressures to the plan as they arise. Evaluate,
respond, and intervene to minimize impact of the disturbance or redirect
to positive outcome.  (AA’s LIT v4.0)

Anticipate Finding out and knowing what to expect (Hollnagel, 2015); What could
happen? What-ifs (AA’s LIT v1.0)

Coordinate Proactively build the team and establish a shared mental model. Utilize all
available resources, both internal and external. The response to the
altered plan with other team members. Ensure the crew has shared a
mental model for getting the plan back on track or normalized as quickly
as possible. Continue until the plan has stabilized. (AA’s LIT v4.0)

Error Any practice that deviates from a written policy or procedure, or deviates
from the crew's intention. (AA’s Flight Operations Manual, 2019)

Learn Knowing what has happened (Hollnagel, 2015); Active process of

improving future performance. When I was last here X happened, and
this time will do Y. Reflect on the outcome of the disturbance / pressure
for future planning. Review materials / flight documents / previous lessons
learned before making the next plan (AA’s LIT v4.0)

Monitor Knowing what to look for (Hollnagel, 2015); Proactively watch for normal
triggers, alarms, threats and anomalies (AA’s LIT v1.0)

Plan Create, discuss, initiate and monitor in action for disturbances.

Proactively act to keep it on track.

Potential Positive, recurring capability to adjust performance by responding to

changes disturbances and opportunities under actual operating
conditions in a flexible and timely manner

Pressure Influences or pressures originating outside of the flightdeck that impact

the crew’s task loading and prioritization

Proficiency Specific function whose presence serves as a proxy measure to resilient

capability (potential)

42
 Resilience Engineering Engineering what a system needs for its continued existence and growth,
hence addresses both safety and core business processes (productivity,
quality, and effectiveness). (Hollnagel, 2015); Resilience Engineering as
a field, seeks to 1) discover operational principles which allow complex
systems to adapt to the often-unforeseen conditions of changing worlds;
2) design, develop and operate systems according to those principles.

Resilient performance The ability to sustain required operations under both expected and
unexpected conditions by adjusting its functioning prior to, during, or
following events such as changes, disturbances, and opportunities
(Hollnagel, 2015)

Respond Knowing what to do, being capable of doing it (Hollnagel, 2015);

Effectively react to normal triggers, alarms, threats and anomalies (AA’s
LIT v1.0)

Safety-I Protection and prevention against harmful events (Hollnagel, 2015)

Safety-II Enhancing the system’s ability to function in a way that produces

acceptable outcomes (Hollnagel, 2015)

Threat An event, external to a pilot or flight crew, which increases operational

complexity and occurs outside the influence of the flight crew. (AA’s Flight
Operations Manual, 2019)

Authors
Christine Jefferies, The Ohio State University
Bogomir Glavan, American Airlines
Nicholas Peterson, American Airlines
James Kwasny, American Airlines
Guy Mouton, American Airlines
Gregory Scheidel, American Airlines
E. Asher Balkin, The Ohio State University

43