Office of Inspector General

Audit Report

ACTIONS ARE NEEDED TO STRENGTHEN

FMCSA’S COMPLIANCE, SAFETY,
ACCOUNTABILITY PROGRAM
Federal Motor Carrier Safety Administration

Report Number: MH-2014-032

Date Issued: March 5, 2014
 U.S. Department of
Memorandum
Transportation
Office of the Secretary
of Transportation
Office of Inspector General

Subject: ACTION: Actions Are Needed To Strengthen Date: March 5, 2014

FMCSA’s Compliance, Safety, Accountability
Program
Federal Motor Carrier Safety Administration
Report No. MH-2014-032

From: Joseph W. Comé Reply to

Attn. of: JA-40
Assistant Inspector General for Highway and
Transit Audits

To: Federal Motor Carrier Safety Administrator

Large trucks and buses were involved in over 125,000 reportable crashes in 2012.
To improve commercial motor vehicle safety, the Federal Motor Carrier Safety
Administration (FMCSA) launched its Compliance, Safety, Accountability
program (CSA) nationwide at the end of 2010. CSA is designed to target
enforcement interventions—such as roadside inspections and on-site reviews—on
motor carriers posing a greater safety risk to the traveling public. To identify
carriers with higher risks of unsafe behavior, FMCSA implemented the Carrier
Safety Measurement System (CSMS), which draws on data submitted by States
and carriers to assess carriers’ on-road safety performance.

During a September 13, 2012, hearing before the House Transportation and
Infrastructure Subcommittee on Highways and Transit, the FMCSA Administrator
stated that FMCSA has sufficient data to assess the safety performance of nearly
200,000 out of approximately 525,000 active carriers in at least one safety
category. According to FMCSA officials, these 200,000 carriers are involved in
approximately 93 percent of all crashes. However, the trucking industry and
Members of Congress expressed concerns about FMCSA’s implementation of
CSA, particularly the completeness and accuracy of CSMS data.

In October 2012, the Chairman and Ranking Member of the Subcommittee

requested that we evaluate FMCSA’s CSA program. Our objectives were to assess
FMCSA’s data quality controls and its enforcement intervention mechanisms.
Specifically, we determined whether FMCSA (1) has sufficiently strengthened its
 2

controls to ensure data quality, (2) addressed key challenges for timely and
effective implementation of CSA enforcement interventions nationwide, and
(3) followed system development best practices and controls when implementing
CSA.

We conducted our work between January 2013 and January 2014 in accordance
with generally accepted Government auditing standards. To conduct our work, we
evaluated data quality of the Motor Carrier Management Information System
(MCMIS) 1 tables from 2010 through 2012, which CSMS uses to generate rankings
intended to reflect carrier performance and risk. We also reviewed reports on
FMCSA’s data correction process and data quality program, and evaluated
FMCSA’s progress in implementing enforcement interventions. As the
Government Accountability Office (GAO) also has a standing request from the
Senate Committee on Appropriations to evaluate CSA, 2 we coordinated with GAO
to avoid duplicating work.

BACKGROUND
The goal of CSA is to reduce large truck and bus crashes, injuries, and fatalities by
focusing the resources of FMCSA and its State partners on higher risk carriers.
CSMS, a software algorithm, calculates percentile rankings for carriers’ on-road
safety performance in seven areas, called Behavior Analysis Safety Improvement
Categories (BASICs) and a crash indicator. 3 CSMS calculations rely on data
including State-reported crash and inspection data and carrier-reported census data
that include information on the company’s size and operations. FMCSA maintains
this data in MCMIS. Carriers and other parties can ask for corrections to State-
reported crash and inspection data if they believe it is inaccurate. CSMS generates
warning letters for carriers determined to be higher risk (based on BASIC and
crash indicator percentile rankings) and prompts interventions that Federal and
State enforcement officials could use to target those carriers, such as roadside
inspections and on-site reviews. Figure 1 illustrates the operational model for the
CSA program.

1
MCMIS contains FMCSA inspection, crash, compliance review, safety audit, and registration data.
2
GAO is evaluating the CSMS algorithm. Its announced audit objectives are to assess (1) How effectively does the
Compliance, Safety, Accountability program identify and evaluate carriers that pose the highest safety risk? (2) To
what extent do the interventions used under the Compliance, Safety, Accountability program improve motor carrier
safety? and (3) What challenges does FMCSA face in fully implementing the Compliance, Safety, Accountability
program?
3
The six BASICs are: (1) unsafe driving, (2) hours-of-service compliance, (3) driver fitness, (4) controlled
substances/alcohol, (5) vehicle maintenance, and (6) hazardous materials compliance.
 3

Figure 1. CSA Operational Model

Source: FMCSA CSMS Methodology, version 3.0.1 (August 2013)

The Volpe National Transportation Systems Center (Volpe) developed CSMS for
FMCSA and is responsible for testing, maintaining, and making changes to the
system. Because CSMS is a Department of Transportation (DOT) information
technology system, industry best practices and Federal internal control standards
are applicable to its development, testing, and validation. These best practices and
standards include DOT’s Integrated Program Planning and Management
Governance and Practitioners Guides (IPPM) 4 and guidance from the National
Institute for Standards and Technology (NIST) and GAO.

From 1999 to 2006, we issued four reports related to MCMIS data and/or CSA’s
predecessor, SafeStat. GAO issued five reports on motor carrier data and
enforcement from 1997 to 2011. In general, the recommendations from these
reports focused on addressing data quality problems. See exhibit B for a list of
these reports.

4
The IPPM provides a framework to ensure that DOT information technology programs and projects are effectively
planned and managed.
 4

RESULTS IN BRIEF
FMCSA has strengthened its controls to improve the quality of State-reported data
used to assess carriers’ safety performance, but the Agency has not fully
implemented planned improvements to its processes for reviewing data correction
requests and ensuring that information carriers are required to submit every
2 years is accurate. 5 Specifically, FMCSA enhanced its efforts to monitor and
correct State-reported data on crashes and inspections, and FMCSA reports show
that States’ data quality has generally improved. However, FMCSA has not yet
implemented planned actions to revise guidance for its data correction process. In
addition, FMCSA took limited action to address inaccurate and incomplete data
reported by carriers, despite our 2006 recommendations. Our current review
determined that only about 401,000 of the roughly 803,000 active interstate
carriers updated their data as required, 6 which can interfere with accurate
calculations of carriers’ safety performance. In November 2013, FMCSA issued a
policy to automatically deactivate USDOT numbers 7 for carriers who do not
submit required data, but the deactivations are not scheduled to begin until
March 2014.

FMCSA has not fully implemented the CSA enforcement intervention process
nationwide. Only 10 States had fully implemented CSA enforcement interventions
at the time of our report, and FMCSA provided no date when it expects to
complete implementation at all States. The remaining 41 States (including the
District of Columbia) are awaiting delivery of and training on the new software
required to assess and monitor the interventions. FMCSA expects to release this
software by May 2015. Because of the limited implementation of the enforcement
interventions to date, we did not assess the effectiveness of the interventions. 8
However, based on our initial observations, FMCSA faces 2 key challenges to
fully implement CSA interventions in the 41 remaining States: (1) developing and
deploying software training for the States in a timely manner and (2) working with
its Division Offices and their State partners to ensure that States apply the
interventions consistently.

FMCSA has limited documentation demonstrating that it followed information

technology system best practices and Federal guidance while developing and
testing CSMS. Specifically, industry best practices and Federal guidance
emphasize thorough documentation of information technology system components
5
Carriers are required to update their company’s census data every 2 years.
6
Between January 2011 and February 2013.
7
The USDOT Number serves as a unique identifier when collecting and monitoring a company’s safety information
acquired during audits, compliance reviews, crash investigations, and inspections. Companies that operate commercial
vehicles transporting passengers or hauling cargo in interstate commerce must have a USDOT Number.
8
In November 2013, the National Transportation Safety Board reported its concerns with the use of focused
compliance reviews for several motor carriers involved in fatal crashes. It recommended that the Department of
Transportation audit this area.
 5

and controls. While FMCSA documented how carrier percentile rankings are
calculated, its documentation of other important processes, such as validation and
testing, is incomplete. For example, FMCSA lacks documentation to show that it
conducted testing for four of the changes made to the system since its nationwide
implementation in 2010.

We are making a series of recommendations to strengthen CSA’s data controls,

address intervention challenges, and improve system documentation.

FMCSA STRENGTHENED DATA QUALITY CONTROLS BUT HAS

NOT FULLY IMPLEMENTED PLANNED IMPROVEMENTS
FMCSA strengthened data quality controls by enhancing efforts to monitor and
correct State-reported crash and inspection data. However, FMCSA has not fully
implemented planned improvements to its data correction process. In addition,
FMCSA took limited action to address inaccurate and incomplete data reported by
carriers, despite our 2006 recommendations. In November 2013, FMCSA issued a
new enforcement policy for carriers who do not submit required data but has not
yet begun using this process.

FMCSA Took Actions To Improve the Quality of State-Reported Data

Crash and inspection data that States regularly enter into MCMIS form part of the
calculation that CSMS uses to evaluate carriers’ percentile rankings. Prior audits
by our office and GAO identified the need for considerable improvement in State-
reported data. In 2004, FMCSA implemented the State Safety Data Quality
program (SSDQ), which evaluates States’ data reporting and assigns an overall
qualitative score based on ratings for nine SSDQ performance measures, such as
crash record completeness and inspection accuracy. 9 FMCSA has repeatedly
updated the SSDQ performance measures (see exhibit C for descriptions of the
performance measures and the years each was updated). For example, in 2010,
FMCSA added more stringent goals and two new measures for inspection report
completeness and accuracy.

FMCSA’s monthly reports from 2010 to 2012 show that States’ overall SSDQ
performance ratings have improved over time. According to the SSDQ ratings for
2012, 36 States’ data quality were considered “good” overall. In contrast,
31 States received this rating in 2010. States’ performance on some individual
SSDQ measures has also improved. In 2012, 47 States were rated “good” for the

9
Currently, the nine SSDQ performance measures are: (1) crash record completeness, (2) non-fatal crash completeness,
(3) fatal crash completeness, (4) crash timeliness, (5) crash accuracy, (6) inspection record completeness, (7) inspection
vehicle identification number accuracy, (8) inspection timeliness, and (9) inspection accuracy.
 6

crash record completeness measure, which means that these States reported
completed driver and vehicle information to FMCSA at least 85 percent of the
time. In 2010, only 44 States received a “good” rating for this measure.

Continued monitoring is important for FMCSA to promptly detect State data

quality issues. For example, we found that almost 19 percent of the
136,810 crashes that States reported in MCMIS for calendar year 2010 were
reported after the 90-day goal that FMCSA established for timely reporting. An
FMCSA official stated that FMCSA was aware of this issue and attributed it to a
software problem in some States. 10 Timeliness has improved in the last 2 years:
27 States were rated “good” for crash reporting timeliness 11 in every month of
2011, and 30 States were rated “good” for this measure in every month of 2012.

In addition to SSDQ, FMCSA uses other tools to monitor and support

improvement of State-reported data. For example, FMCSA regularly produces
detailed reports with additional analysis of State-reported data quality. These
reports, along with the SSDQ reports, help the Agency monitor the quality,
timeliness, and integrity of MCMIS crash and inspection data—two categories
used by CSMS to calculate carriers’ percentile rankings. In addition, FMCSA’s
SAFETYNET, a database management system, contains data quality controls in
the entry fields that help States and Divisions enter data correctly and identify
problems before submitting the data. FMCSA also provides SAFETYNET
guidance to promote best practices for consistent and accurate data entry.

FMCSA Has a Process for Correcting Inaccurate Data But Has Not
Implemented Planned Improvements
Using a process FMCSA developed, known as DataQs, carriers and other parties
sometimes challenge the accuracy of State-reported crash and inspection data in
MCMIS. The most common data challenge that results in a correction is the claim
that a State assigned a crash or inspection to the wrong carrier—an error that
would affect the carrier’s percentile rankings. The DataQs process allows carriers
or other parties to request corrections to State-reported MCMIS data. To request a
data review, a filer must submit an electronic data review request, with a
description of the inaccurate data, to FMCSA’s DataQs Web site. State DataQs
analysts 12 review these requests, along with any supporting documentation
provided, to decide whether to make data corrections. It is important to note that
only a small percentage of the crash and inspection data is challenged—about

10
We did not verify FMCSA’s statement that untimely reporting was due to State software problems, as our work did
not include assessments of reporting from each of the 51 States.
11
A “good” rating means the State reported 90 percent or more of its crash data in 90 days or less.
12
State DataQs analysts process nearly all DataQs requests, but FMCSA DataQs analysts review a small number of
requests related to FMCSA data.
 7

1 percent from 2012-2013. However, the American Trucking Association stated

that inaccurate State-reported data significantly impacted some of their members’
BASIC percentile rankings.

Since the crash and inspection data in MCMIS are State or locally generated,
FMCSA’s DataQs guidance gives State DataQs analysts considerable discretion
when deciding whether to make data corrections. For example, some filers
challenge whether carrier citations should be reported in MCMIS if they have been
dismissed in State court. FMCSA guidance does not prescribe a specific course of
action in these cases; instead, it recommends that DataQs analysts review these
dismissals on a case-by-case basis.

FMCSA data, which tracks the number of requests for data reviews closed with a
correction, shows it is possible that some States are stricter or more lenient than
others when making DataQs decisions. For example, the data shows that
California closed the highest proportion of its data challenges with corrections
(78 percent), and Connecticut closed the lowest proportion (26 percent). FMCSA
and State officials identified various factors that contribute to the variation in
correction rates. According to FMCSA, corrections are more likely if the filer
provides supporting documentation. For example, among data review requests
related to inspection data, 71 percent of requests that included supporting
documentation were closed with a correction, compared to 53 percent of requests
without supporting documentation. The variations in correction rates may also be
attributable to how accurately States input crash and inspection data in MCMIS.
We spoke with officials from Connecticut about why they closed the lowest
proportion of data changes with corrections. According to an official, the low
correction rate reflected the extensive training Connecticut provided to staff that
report violations. If the data are input correctly the first time, they require fewer
corrections later on.

FMCSA attempts to promote consistency among States’ data reporting by

providing written guidance and by tracking the number of data reviews closed
with corrections. According to FMCSA, it is updating the DataQs guidance to
clarify how data challenges should be reviewed and to provide additional
measures to ensure that data challenges are closed consistently. For example, the
draft guidance we reviewed provided clarification on what constitutes sufficient
evidence for making a correction. FMCSA also developed new reports to collect
additional information and better analyze the outcomes of data reviews but has not
begun using them.
 8

FMCSA Has Yet To Implement Planned Actions To Improve

Carrier-Reported Data
FMCSA safety regulations require carriers to routinely update their census data—
including information on carriers’ addresses, phone numbers, number of power
units (motor vehicles), and vehicle miles traveled—that goes into MCMIS. Certain
census data fields form part of the calculation that CSMS uses to evaluate carriers,
and missing or outdated data can lead to incorrect computations of carriers’
BASIC percentile rankings. Despite FMCSA’s efforts and our prior
recommendations, we found continued weaknesses in carrier-reported census data.

In 2006, we reported that approximately 192,000 (27 percent) of 702,277 existing

motor carriers did not update census data, and that inaccurate and incomplete
carrier-reported data hampered safety monitoring and enforcement activities.
Accordingly, we recommended that FMCSA take firm action to increase
compliance with the census data updating requirement by intensifying efforts to
fine motor carriers that resist compliance or by taking other measures that can be
demonstrated to be effective. 13 In its response to this recommendation, FMCSA
did not commit to any specific action to increase carriers’ compliance with this
requirement; therefore, the recommendation remains open.

In the 7 years since we first made the recommendation, FMCSA stated that it has
taken over 2,000 enforcement actions (such as levying fines) against carriers with
outdated census data. However, we determined that between January 2011 and
February 2013, only about 401,000 of the roughly 803,000 interstate carriers 14
active in MCMIS had updated their census data.

Lack of updated census data could impede FMCSA’s ability to effectively follow
through on its planned actions to assess higher risk carriers. At a
September 13, 2012, congressional hearing, the FMCSA Administrator stated that
FMCSA has sufficient data to assess nearly 200,000 out of approximately
525,000 active carriers in at least one safety category. According to FMCSA, these
200,000 carriers are involved in approximately 93 percent of all crashes. However,
we uncovered the following data quality issues when we analyzed the census data
for these 200,000 carriers:

• Approximately 5,000 carriers (about 2.5 percent) of the 200,000 carriers are
types that FMCSA does not oversee, such as inactive and/or intrastate non-

13
Significant Improvements in Motor Carrier Safety Program Since 1999 Act but Loopholes for Repeat Violators Need
Closing (OIG Report Number MH-2006-046), Apr. 21, 2006.
14
In September 2012, FMCSA testified there were approximately 525,000 active carriers. When we requested data on
these carriers in 2013, FMCSA provided a list of about 523,000 active carriers—fewer than the roughly 803,000 active
carriers we identified during our review. While we counted all carriers labeled “active” in MCMIS (to be consistent
with our prior reviews), FMCSA counted only carriers with a record of external activity (such as inspection, crash, and
vehicle registration) in the past 3 years.
 9

hazardous materials carriers. As a result, these carriers are not required to

submit census data in MCMIS and should not have been included in FMCSA’s
list of active carriers.

• Of the approximately 195,400 carriers remaining, about 9 percent have not

updated their census data in the last 2 years. While this is a marked
improvement compared to 2005 levels (approximately 27 percent), updated
census data is critical to accurate CSMS calculations of carriers’ percentile
rankings. For example, if a carrier does not update MCMIS to show that its
total vehicle miles traveled has decreased in recent years, then the carrier’s
BASIC percentile ranking could be inflated.

• About 13 percent of the carriers had zeros in a data field used to record the
number of their power units, which refers to the number of motor vehicles in a
carrier’s fleet. Power unit entry data is included as part of the denominator for
two CSMS calculations: the crash indicator and the percentile ranking for
unsafe driving. Zero power unit entries in the denominator can interfere with
these calculations, as numbers cannot be divided by zero. For example, it
would not be possible to calculate a crash indicator for a carrier with no motor
vehicles (zero power unit entries) because a crash, by definition, must involve
a vehicle.

To address this longstanding issue with carrier-reported data, FMCSA issued a

policy on November 1, 2013, that will automatically deactivate USDOT numbers
for carriers who do not submit required census data. According to the policy, 15
FMCSA plans to send warning letters to carriers and to provide a grace period for
them to update their census data before the Agency deactivates their USDOT
numbers. According to the policy, deactivations will begin in March 2014 for
carriers that fail to update their data by January 2014.

FMCSA FACES CHALLENGES IN FULLY IMPLEMENTING CSA

ENFORCEMENT INTERVENTIONS
CSA is designed to target enforcement interventions—such as roadside
inspections and on-site reviews—on higher risk motor carriers, as identified by
CSMS calculations of carriers’ percentile rankings. FMCSA phased in use of all
CSA enforcement interventions in some States 16 and focused its efforts on
ensuring consistent use of the interventions. As part of this process, FMCSA
issued its CSA intervention policy and a manual for enforcement personnel.

15
FMCSA memorandum MC-ECS-2013-0009.
16
The nine test States are Colorado, Delaware, Georgia, Kansas, Maryland, Minnesota, Missouri, Montana, and
New Jersey. According to FMCSA, Alaska also implemented all interventions.
 10

FMCSA reports that 10 States have fully implemented CSA enforcement

interventions and the other States have implemented most interventions except
off-site investigations and cooperative safety plans. FMCSA has a rough timeline
for full implementation of CSA interventions in the remaining 40 States and the
District of Columbia but has not set an implementation date. These remaining
States are waiting for FMCSA’s information technology office to deliver Sentri,
the enforcement intervention software, which will be used by all States and is
expected to be released in May 2015. The purpose of this software is to combine
FMCSA’s roadside inspection, investigative, and enforcement reporting, as well as
access to carrier and driver information, into one system for use by States and
FMCSA Division Offices. FMCSA plans to deliver training to these States 90
days after the software release but provided no date when it expects all States to
fully implement CSA’s enforcement interventions.

Because of the limited implementation of CSA enforcement interventions to date,

we did not assess the effectiveness of the interventions. However, based on our
initial observations, FMCSA faces 2 key challenges to fully implement CSA
interventions in the 41 remaining States: (1) developing and deploying software
training for the States in a timely manner and (2) working with its Division Offices
and their State partners to ensure that States apply the interventions consistently.

FMCSA HAS LIMITED DOCUMENTATION DEMONSTRATING IT

FOLLOWED INFORMATION TECHNOLOGY BEST PRACTICES
AND FEDERAL GUIDANCE FOR CSMS
Industry best practices and Federal guidance emphasize thorough documentation
of information technology system components and controls. While some of these
best practices and guidance are not requirements, they are advisable for high-
visibility systems, such as CSMS. 17 However, our review determined that, while
FMCSA documented how CSMS generates percentile rankings, it lacks complete
documentation on system components, validation processes, and system change
processes and related testing. Insufficient documentation impedes FMCSA’s
ability to demonstrate the actions already taken to support CSMS and to identify
the actions that will be needed to maintain effective control of the system in the
event of staff turnover and further changes to the system.

Documentation of System Components. NIST recommends that agencies have

an information system component inventory that accurately reflects the current
system. According to GAO, such an inventory helps to provide control over

17
As of November 2013, the DOT Chief Information Officer requires FMCSA and other Operating Administrations to
identify how IPPM processes and practices will be implemented for new and existing systems.
 11

system changes and mitigate system corruption risk. In March and April of 2013,
after we announced our audit, FMCSA updated and consolidated its CSMS
requirements document, which describes the functions the system is intended to
perform. However, the requirements document did not include a complete list of
the 51 MCMIS data fields that CSMS uses to calculate carriers’ BASIC percentile
rankings. Without a complete and documented list of the 51 data fields, it is
difficult for FMCSA to demonstrate the quality of the data it relies on to calculate
the percentile rankings. After several weeks of communication with officials from
FMCSA and Volpe, we were able to identify the 51 MCMIS data fields. Our
testing of the data revealed no significant problems, and we concluded that the
data were generally complete and accurate. 18

Documentation of Routine System Validation. FMCSA has not formally

documented its system validation processes in a single directive, policy, or manual
approved by management—as recommended in GAO guidance on internal
controls. 19 Although the CSMS requirements document mentions validation of
CSMS data, it does not provide details on the specific validation steps. According
to a separate, unsigned document that FMSCA provided, Volpe performs two
types of validations of CSMS results: (1) monthly validations to confirm that
CSMS calculations of carriers’ BASIC percentile rankings are correct and based
on complete and accurate MCMIS data and (2) validations performed after
changes are made to the system methodology, which are intended to reconcile
differences in CSMS coding. Without an approved document, FMCSA lacks
assurance that staff will properly carry out these validation processes.

Further, the results of CSMS validation steps are not completely documented,
despite GAO guidance calling for documentation of control activities performed.
For example, Volpe uses spreadsheets to record the results of its monthly
validations, but it does not collect them in a central file to provide a complete
validation record. In addition, Volpe does not formally record the results of its
validations after the system methodology is changed. In the event of staff turnover,
a well-documented process and records are necessary to ensure these validations
are properly performed and documented.

Documentation of System Change Processes and Associated Testing. DOT’s

Integrated Program Planning and Management Governance and Practitioners
Guides and NIST guidance 20 recommend documenting processes for making

18
We conducted basic reliability tests to identify blank or invalid entries among the 51 MCMIS fields used by CSMS
to calculate percentile rankings—as well as 8 supporting fields using data pulled directly from MCMIS from
January through April 2013.
19
GAO/AIMD-00-21.3.1, Standards for Internal Control in the Federal Government, Nov. 1999; GAO-01-1008G,
Internal Control Management and Evaluation Tool, Aug. 2001.
20
NIST Special Publication 800-53 Revision 3, Recommended Security Controls for Federal Information Systems and
Organizations, Aug. 2009 (includes updates as of May 1, 2010).
 12

changes to an information technology system in a configuration management

plan. 21 While FMCSA staff were able to describe the process used to make
changes to CSMS, FMCSA has not formally documented this process in a
configuration management policy, which would include a change management
process. In addition, FMCSA cannot demonstrate that its system change processes
have been followed. As table 1 shows, FMCSA lacks complete documentation for
changes made to the system since implementation in December 2010—including
documentation to show that it conducted testing during the four occasions the
system was changed. However, GAO’s Federal Information System Controls
Audit Manual 22 states that a system should be tested (and test results should be
recorded) when changes are made to the system.

Table 1. CSMS Testing Documentation

Evidence Documented
Date System Documented that testing Test results acceptance of
Changed test plan occurred recorded changes
December 2010
(nationwide system No Yes Yes No
implementation)

January 2012 No No No No

August 2012 No No No No

December 2012 Yes No No Yes

August 2013 No No No No

Source: OIG analysis of CSMS documentation.

CONCLUSION
FMCSA has made progress in moving toward a more data-driven, risk-based
approach to oversight of the motor carrier industry, as called for by CSA. Quality
data are critical to accurately identifying the highest risk carriers for enforcement
interventions. While FMCSA has strengthened quality controls for State-reported
data, more action is needed in key areas, including improving census data and
completing its roll out of CSA enforcement interventions. Given that CSMS is
such a high-visibility system within the motor carrier industry, FMCSA can also
enhance its documentation of system processes to better adhere to best practices
and Federal guidance. Without sustained management attention in these areas,

21
A configuration management plan outlines the processes required to ensure that changes to an information
technology system are controlled.
22
GAO-09-232G, Federal Information Systems Controls Audit Manual (FISCAM), Feb. 2009.
 13

FMCSA will be hindered in its ability to effectively implement CSA nationwide

and address the key concerns of industry stakeholders.

RECOMMENDATIONS
We recommend the Federal Motor Carrier Safety Administrator:

1. Issue updated DataQs guidance;

2. Implement the process for deactivating USDOT numbers when carriers do not
submit required census data, as described in FMCSA memorandum
MC-ECS-2013-0009;

3. Develop a comprehensive plan to fully implement CSA enforcement

interventions in the remaining 41 States. The plan should include an estimated
completion date and milestones for releasing Sentri software, developing and
delivering training, and using the enforcement interventions;

4. Update the CSMS requirements document to (a) specify all sources of CSMS
data, including each of the MCMIS fields used, and (b) fully describe CSMS
validation procedures;

5. Develop and implement a process for managing CSMS system documentation

that includes a central file for validation records and testing results; and

6. Develop and implement a configuration management policy that includes

documentation of system changes and associated testing for CSMS.

AGENCY COMMENTS AND OFFICE OF INSPECTOR GENERAL

RESPONSE
We provided FMCSA with our draft report on January 30, 2014, and received its
formal response on February 27, 2014. FMCSA’s response is included in its
entirety as an appendix to this report. In its response, FMCSA concurred with all
six of our recommendations and provided appropriate planned actions and
timeframes. Accordingly, we consider all six recommendations resolved but open
pending completion of planned actions.
 14

ACTIONS REQUIRED
FMCSA’s planned actions for all six recommendations are responsive, and we
consider the recommendations resolved but open pending completion of the
planned actions.

We appreciate the courtesies and cooperation of the Federal Motor Carrier Safety
Administration, the Volpe National Transportation Systems Center, and the U.S.
Department of Transportation during this audit. If you have any questions
concerning this report, please call me at (202) 366-5630 or Wendy Harris,
Program Director, at (202) 366-2794.

cc: DOT Audit Liaison, M-1

FMCSA Audit Liaison, MCPRS
 15

EXHIBIT A. SCOPE AND METHODOLOGY

We conducted our work from January 2013 through January 2014 in accordance
with generally accepted Government auditing standards. Those standards require
that we plan and perform the audit to obtain sufficient, appropriate evidence to
provide a reasonable basis for our findings and conclusions based on our audit
objectives. We believe that the evidence obtained provides a reasonable basis for
our findings and conclusions based on our audit objectives.

Our objectives were to assess FMCSA’s data quality controls and its enforcement
intervention mechanisms. Specifically, we determined whether FMCSA (1) has
sufficiently strengthened its controls to ensure data quality, (2) addressed key
challenges for timely and effective implementation of CSA enforcement
interventions nationwide, and (3) followed system development best practices and
controls when implementing CSA.

To verify whether FMCSA established adequate data quality controls, we

evaluated the MCMIS data used by CSMS to generate carriers’ BASIC percentile
rankings, including carriers’ census data and State-reported crash and inspection
data. We also evaluated FMCSA’s SSDQ performance measures and reviewed
SSDQ monthly reports that rated States’ performance from calendar year 2010 to
2012. Further, we reviewed FMCSA’s data correction process, known as DataQs,
and analyzed data challenges entered from February 1, 2012, to February 22, 2013.

To evaluate FMCSA’s implementation of CSA enforcement interventions, we

reviewed FMCSA regulations and guidance and policies related to the
enforcement interventions, and interviewed FMCSA officials. We limited work on
this objective because only 10 States have implemented the interventions.

To determine whether FMCSA met system development controls when

implementing CSA, we conducted a detailed review of FMCSA’s CSMS system
documentation. FMCSA provided much of this system documentation in
August 2013—7 months after we announced our audit. We also examined CSMS
validation and testing procedures. In addition, we worked with our information
technology specialists to review CSMS system security controls and FMCSA’s
information system change and approval processes.

Exhibit A. Scope and Methodology

 16

EXHIBIT B. OIG AND GAO REPORTS RELATED TO MOTOR

CARRIER DATA AND ENFORCEMENT

OIG Reports
Significant Improvements in Motor Carrier Safety Program Since 1999 Act but
Loopholes for Repeat Violators Need Closing (OIG Report Number MH-2006-
046), Apr. 21, 2006.

Report on Investment Review Board Deliberations on the Motor Carrier

Management Information System (OIG Report Number MH-2004-068),
June 29, 2004.

Improvements Needed in the Motor Carrier Safety Status Measurement System

(OIG Report Number MH-2004-034), Feb. 13, 2004.

Motor Carrier Safety Program, Federal Highway Administration (OIG Report

Number TR-1999-091), Apr. 26, 1999.

OIG reports are available on our Web site at: http://www.oig.dot.gov.

GAO Reports
Motor Carrier Safety: More Assessment and Transparency Could Enhance
Benefits of New Oversight Program (GAO-11-858), Sept. 29, 2011.

Motor Carrier Safety: The Federal Motor Carrier Safety Administration Has
Developed a Reasonable Framework for Managing and Testing Its
Comprehensive Safety Analysis 2010 Initiative (GAO-08-242R), Dec. 20, 2007.

Further Opportunities Exist to Improve Data on Crashes Involving Commercial

Motor Vehicles (GAO-06-102), Nov. 18, 2005.

Truck Safety: Motor Carriers Office Hampered by Limited Information on Causes

of Crashes and Other Data Problems (GAO/RCED-99-182), June 29, 1999.

Commercial Motor Carriers: DOT is Shifting to Performance-based Standards to

Assess Whether Carriers Operate Safely (GAO/RCED-98-8), Nov. 3, 1997.

Exhibit B. Recent OIG and G AO Report s Related To Motor Carrier

Data and Enf orcem ent
 17

EXHIBIT C. FMCSA’S STATE SAFETY DATA QUALITY PROGRAM

“GOOD” RATINGS

Year(s) Measure
SSDQ Performance
Description of “Good” Rating Implemented/
Measure
Changed
Overall State SSDQ Minimum of 1 Good Crash Measure, 1 Good 2004, 2006, 2007,
Rating Inspection Measure, and 0 Poor Measures 2010, 2011
Crash Record ≥85% completed driver and vehicle crash
2007
Completeness information reported to FMCSA
Non-fatal crash records are ≥ lower boundary of
Non-Fatal Crash 90% prediction interval and ≤ upper boundary of
2007, 2011
Completeness 99% prediction interval, based on data ranges
generated by a model.
≥90% of State-reported fatal crash records in
Fatal Crash MCMIS, compared to number of fatal crash
2004
Completeness records reported in NHTSA’s Fatality Analysis
Reporting System (FARS).

Crash Timeliness ≥90% of crashes reported within 90 days 2004, 2006, 2010

≥95% State-reported records are matched to a

Crash Accuracy 2004, 2006
company registered in MCMIS over 12 months
Inspection Record ≥85% completed driver and vehicle inspection
2010
Completeness information reported to FMCSA
Inspection Vehicle
≥85% completed and accurate VINs reported to
Identification Number 2010
FMCSA
(VIN) Accuracy
≥90% inspection records reported to FMCSA
Inspection Timeliness 2004, 2006, 2010
within 21 days over 12 months
≥95% inspection records reported by State over
Inspection Accuracy 12 months match to a company registered in 2004
MCMIS

Source: Based on FMCSA’s SSDQ methodology, dated January 2012.

Exhibit C. FMCS A’ s State Safet y Data Qualit y Program “G ood”

Ratings
 18

EXHIBIT D. MAJOR CONTRIBUTORS TO THIS REPORT

Name Title

Wendy Harris Program Director

Regan Maund Project Manager

Luke Brennan Senior Analyst

Doris Kwong Analyst

Peter Barber Analyst

William Savage Information Technology Specialist

Michael Marshlick Project Manager

Nicholas Coates Legal Counsel

Christina Lee Writer-Editor

Exhibit D. Major Contributors to This Report

 19

APPENDIX. AGENCY COMMENTS

U.S. Department
Memorandum
Of Transportation
Federal Motor Carrier
Safety Administration

Subject: ACTION: Response to Office of Inspector Date: February 27, 2014

General (OIG) Draft Report: Actions Are
Needed To Strengthen FMCSA’s
Compliance, Safety, Accountability Program

From: Anne S. Ferro Reply to: MC-P

Administrator

To: Joseph W. Comé

Assistant Inspector General
for Highway and Transit Audits

The Federal Motor Carrier Safety Administration (FMCSA) is committed to improving large
truck and bus safety and to ultimately reduce crashes, injuries, and fatalities that are related to
commercial motor vehicles. To assist in this effort, the Compliance, Safety, Accountability (CSA)
program provides motor carriers and drivers with attention from FMCSA and State Partners about
their potential safety problems.

FMCSA CONTINUES TO IMPROVE THE QUALITY OF THE DATA IT COLLECTS

The FMCSA closely monitors the quality of State reported data through the State Safety Data
Quality (SSDQ) program and offers assistance and guidance when potential problems are noted.
As of January 2014, 41 States had an overall rating of “good” 1 in the SSDQ performance
measures. Also, all States were rated “good” in crash record completeness, with only the District
of Columbia not scored due to insufficient data, and 38 States were rated “good” in crash
reporting timeliness.

On April 27, 2013, FMCSA released a redesigned DataQs 2 system for motor carriers, drivers,
other industry representatives, and the public to submit Requests for Data Review (RDRs). The

1
The State Safety Data Quality (SSDQ) program uses 4 ratings, Good, Fair, Poor, and Insufficient Data to assess the
quality of State reported data. For more information about the State Safety Data Quality (SSDQ) program please
reference http://ai.volpe.dot.gov/DataQuality/DataQuality.asp?redirect=intro.asp
2
DataQs is an online system for drivers, motor carriers, Federal and State agencies, and others to file concerns about
Federal and State data maintained in the FMCSA Motor Carrier Management Information System (MCMIS) and
released to the public.

Appendix. Agency Comments

 20

enhanced DataQs system improves the user experience and the quality of requests submitted. The
enhancements include the following features:

• New My DataQs dashboard;

• More easily accessible tools and resources;
• Step-by-step process for submitting a Request for Data Review (RDR) or Inspection
Report Request (IRR); and,
• Improved list of reviews requested with expanded status options allowing for easier
tracking and monitoring of requests.

In addition, new reports were developed to allow for more in-depth analysis of how RDRs are
processed. These reports provide the ability to identify potential areas of improvement from a
systemic level down to an individual officer.

A revision to the DataQs User Guide and Manual is under development to incorporate additional
guidance and best practices to assist the States in addressing RDRs. The manual will include new
guidance being developed for responses to RDRs concerning inspection violations with an
accompanying citation that goes through an adjudication process in the State within which the
violation was cited.

FMCSA has been proactive in encouraging motor carriers to submit biennial updates to their
registration data. In 2009, FMCSA implemented an automated reminder in the SafeStat system.
A reminder message would be sent to motor carriers whose Motor Carrier Identification Report
(FMCSA Form MCS-150) had not been updated within the previous 24 months as required. A
similar reminder was implemented in other systems, including DataQs where an additional
feature requires motor carriers to update their registration information before an RDR will be
processed. In addition, a reminder to submit a biennial update to registration data is also
incorporated in the on-line CSA Safety Measurement System (SMS), which is available to motor
carriers to monitor their safety performance data. Finally, information on the status of a carrier’s
registration data is displayed in all enforcement information systems employed by FMCSA and
its State partners to readily identify carriers with out-of-date registration data.

Effective November 1, 2013, FMCSA implemented a provision resulting from the Unified
Registration System (URS) final rule (78 FR 52608, August 23, 2013) that states that a motor
carrier that fails to comply with the biennial update requirement will be subject to civil penalties
and deactivation of its U.S. Department of Transportation (USDOT) number. In addition, one of
the new provisions in the URS final rule will prohibit a motor carrier without an active USDOT
number or without USDOT registration from operating a commercial motor vehicle (CMV) in
interstate commerce.

FMCSA IS COMMITTED TO SUCCESSFULLY DEPLOYING CSA INTERVENTIONS

NATIONWIDE IN A UNIFORM MANNER.

The first two phases of nationwide implementation of the SMS and “new interventions” elements
of the CSA program are complete. The first phase began in December 2010 with the use of SMS
to identify and prioritize high-risk carriers for investigations as well as several new interventions,
which include warning letters, focused investigations, red flag violations, and driver-based
sampling processes. The second phase of CSA implementation began in September 2011 with the
integration of the Safety Management Cycle into all investigations. This process provides
investigators with the tools needed to systematically identify truck and bus companies’

Appendix. Agency Comments

 21

underlying safety problems. It provides a step-by-step process that goes beyond just identifying
“what” the violations are to get at “why” the safety performance issues are occurring.

Deployment of the third phase of CSA will occur upon completion of the Agency’s new
investigative software. The Agency’s rollout of the Phase III process will include:

• Offsite investigations, which avoids the cost and disruption of deploying investigators to
a carrier’s place of business.
• Cooperative Safety Plans (CSP), which provides opportunities for truck and bus
companies to describe their plans for taking corrective actions.
• Serious Violations Follow-up Efficiency, which allows the agency to best use its
investigative resources while ensuring carriers properly address and correct serious
violations discovered during investigations.

FMCSA will mitigate risks to successful deployment by developing an implementation plan

covering policy, information technology (IT), training, and communications elements.

FMCSA IS COMMITED TO IMPROVE THE CARRIER SAFETY MEASUREMENT

SYSTEM (CSMS)

The FMCSA is updating the CSMS requirements document to specify all sources of CSMS data,
including each of the Motor Carrier Management Information System (MCMIS) fields used, and
to fully describe CSMS validation procedures. The FMCSA has been proactive in improving its
life cycle management of IT systems. The FMCSA now has a fulltime configuration manager and
a test lead which will facilitate the development and implementation of a process for managing
CSMS system documentation that includes a central file for validation of records and testing
results for IT systems. FMCSA will also develop and implement a configuration management
policy that includes documentation of system changes and associated testing for CSMS and other
FMCSA information systems.

RECOMMENDATIONS AND RESPONSES

Recommendation 1: “Issue updated DataQs guidance.”

Response: Concur. FMCSA is currently revising the DataQs User Guide and Manual.

Target Action Date: July 31, 2014

Recommendation 2: “Implement the process for deactivating USDOT numbers when carriers
do not submit required census data, as described in FMCSA memorandum MC-ES-2013-0009.”

Response: Concur. Starting March 1, 2014, FMCSA will begin deactivating USDOT
numbers for carriers that do not complete their census data updates as required. In
November 2013, FMCSA sent warning letters to all motor carriers required to submit a
census data update in January 2014. On March 1, 2014, any carrier who was required to
update its census data in January 2014, and fails to do so, will have its USDOT number
deactivated. Additionally, on the first day of every month thereafter, the next group of
carriers scheduled to complete their census data as required, and fail to do so, will have
their USDOT number deactivated. For example, on April 1, 2014, FMCSA will
deactivate USDOT numbers for carriers that were due to complete their census data

Appendix. Agency Comments

 22

update during February 2014 and did not complete their update as required. These
carriers received a warning letter in December 2013. On May 1, 2014, FMCSA will
deactivate USDOT numbers for carriers that were due to complete their census data
update during March 2014 and did not complete their update as required. The schedule
and table for the required census data update can be found in 49 CFR 390.19 (b)(2) & (3).

Target Action Date: March 31, 2014

Recommendation 3: “Develop a comprehensive plan to fully implement CSA enforcement

interventions in the remaining 41 States. The plan should include an estimated completion date
and milestones for releasing Sentri software, developing and delivering training, and using the
enforcement interventions.”

Response: Concur. FMCSA will develop an implementation plan to reflect recent

activity and future requirements.

Target Action Date: May 30, 2014

Recommendation 4: “Update the CSMS requirements document to (a) specify all sources of
CSMS data, including each of the MCMIS fields used, and (b) fully describe CSMS validation
procedures.”

Response: Concur. FMCSA will update the CSMS – System Requirements document to
include two new chapters: “Sources of CSMS Data” and “CSMS Validation Procedures.”
The latter chapter will address validation of both the monthly CSMS runs and CSMS
system changes.

Target Action Date: June 30, 2014

Recommendation 5: “Develop and implement a process for managing CSMS system

documentation that includes a central file for validation records and testing results.”

Response: Concur. In June 2013, FMCSA implemented a process for recording the
completion of the steps required to validate the SMS monthly runs in a centralized
location (via SharePoint). Based on the updated validation procedures referenced in
Recommendation #4, FMCSA will develop a more comprehensive centralized system
that will store important results and correspondence for each monthly run.

Target Action Date: June 30, 2014

Recommendation 6: “Develop and implement a configuration management policy that includes

documentation of system changes and associated testing for CSMS.”

Response: Concur. FMCSA will integrate CSMS system changes into the existing
FMCSA information technology (IT) configuration management policy and supporting
configuration management tool, the Electronic Change Request System (eReqs). The
eReqs process includes: internal FMCSA validation; executive oversight; impact
assessment; Software Development Life Cycle documentation, which describes the

Appendix. Agency Comments

 23

requirements, testing, and deployment; and, storage of the modification request

documentation and release artifacts to the centralized location.

Target Action Date: June 30, 2014

We appreciate the opportunity to offer our perspective on this report. Please contact William
Quade, Associate Administrator for Enforcement and Program Delivery, by telephone at (202)
366-4553 with any questions or requests for additional assistance.

Appendix. Agency Comments