See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/261309156

High Speed Video Inspection of Joint Bars Using Advanced Image Collection
and Processing Techniques

Conference Paper · May 2008

CITATIONS READS

24 591

4 authors, including:

Xavier Gibert
Google Inc.
15 PUBLICATIONS   446 CITATIONS   

SEE PROFILE

All content following this page was uploaded by Xavier Gibert on 03 April 2014.

The user has requested enhancement of the downloaded file.

A. Berry Page 1

High Speed Video Inspection of Joint Bars Using Advanced Image Collection and
Processing Techniques

Andie Berry, Boris Nejikovsky, Xavier Gibert

ENSCO, Inc. Ali Tajaddini,
Applied Technology and Engineering Federal Railroad Administration
Division Office of Research and Development
5400 Port Royal Rd 1200 New Jersey Ave SE
Springfield, VA 22151 Washington, DC 20590

ABSTRACT
Broken joint bars have been identified as one of the major causes of main line derailments in the
U.S. Currently, joint bars are inspected visually by railroad maintenance personnel during regular
track inspection from a moving hy-rail vehicle. Realistically, an inspector cannot see small defects
in joint bars while driving a hy-railer. Even large cracks and broken bars are often missed by this
method of inspection. Visual inspection on foot can provide better results, but this is a very slow,
tedious and labor-intensive process. Normal rail traffic and revenue services are often interrupted
when an on-foot inspection is performed.

In 2002, Federal Railroad Administration (FRA) Office of Research and Development (OR&D)
started a program to build a proof of concept prototype visual inspection system that can detect the
cracks on the joint bars. After successful demonstration of the proof of concept prototype, a
system was developed to detect the cracks in the joint bars on the field and gage side of each rail.
With new U.S. Federal mandates to increase joint bar inspection frequency, it has become evident
that the industry is in need of more effective and efficient inspection procedures. In 2004, the U.S.
National Transportation Safety Board (NTSB) investigated a series of derailments caused by joint
bar defects and concluded that current hy-rail inspection processes were inadequate. These
findings led the FRA to set forth an Interim Final Rule which requires the railroads to perform
frequent walking inspections based on the track speed and line density. In response to this
challenge, an image-based system for joint bar inspection was developed and tested with the goal
of capturing high quality digital video images of joint bars from a moving vehicle. The system,
developed with support from the FRA OR&D consists of high speed, high resolution line scan
cameras which capture images of the joint bars, a lighting system, and computers which process
images taken of the rail and report defects to the operator. The computers employ a series of
advanced image processing algorithms to detect joint bar defects. Joint bars are tagged with
milepost and footage information as well as precise GPS coordinates. All significant track features
are inventoried and stored in a permanent database.
A. Berry Page 2

The video inspection solution presented here greatly increases the speed of the inspection
process. The system operates up to speeds of 105 km/hr (65 mph) and can be mounted to either
a hy-rail or rail-bound vehicle. In addition to the defect detection features, the system also gives
the operator the ability to record all significant track features and their locations, such as joints,
bridges, switches and frogs in a comprehensive database.

DESIGN CONCEPT

The system is designed to acquire high resolution images of the rail and process these images in
real time using advanced computer algorithms. The system capitalizes on the capabilities of
computational image analysis in order to detect the presence of joints and their defects. The entire
system has been designed with optimization of image quality in mind. A lighting subsystem
provides optimal lighting conditions for the selected cameras. A mechanical mounting structure
positions the cameras and lights for the most favorable view of the rail web.

Once images are captured they are assembled by state-of-the-art computers. A series of image
processing algorithms are then applied to the images to determine the position of the joint bar.
Once the joint bar is located in the image, a set of classifiers are applied to the image to search for
cracks and other defects in the joint bars.

A user interface provides the operator with a platform to easily evaluate and interpret data.

Automated
Crack
Detection
Editor/Database
Image Storage/Viewing/Reports

Wheel Encoder

Data Collection Computer

Automatic Joint Detection
Assemble Images from Lines

Camera

Lighting

Figure 1: Visual Inspection System Concept

A. Berry Page 3

SYSTEM DESCRIPTION

The Automated Optical Joint Bar Inspection System (JBIS) consists of several major subsystems:
a lighting subsystem, an image acquisition system, image processing algorithms to detect joint
locations and defects, and a user interface that allows the user to manage and review the data.

Lighting Subsystem

The lighting system provides even and consistent illumination of the rail with high-powered Xenon
lights. It is designed to temper variations in illumination caused by ambient light while also
providing sufficient shadowing properties. The intensity of the light emitted is great enough to
eliminate the need for light shields and the wavelength of the light is optimized to operate within
the maximum sensitivity range of the camera. The lights are mounted on a beam exterior to the
car and the beam pattern emitted by the lights is aligned with the field of view of the line scan
cameras. The orientation and intensity of the lights is designed to illuminate the rail in such a way
that the image is optimized for both brightness and contrast.

Figure 2: Xenon Lights Illuminating the Rail

System Installation and Implementation

The lights and cameras are mounted on a mechanical beam. The beam can be designed for
installation on both rail-bound and hy-rail vehicles. The system is designed to fit within Plate C
Clearance when fully deployed (Figure 3). When installed on a hy-railer, the beam is designed to
be folded for highway passage (Figure 4). All components installed exterior to the car body are
shock mounted and weather proofed.
A. Berry Page 4

Figure 3: Beam Components Mounted on a Hy-Railer

Figure 4: Beam Folded for Highway Driving

Most of the interior system components are rack-mounted to optimize storage space and cable
routing. Rack mounted chassis provide conditioned power to the lights, cameras and camera
heaters. The power chassis for the lights was designed with LED indicators to alert the operator
when a light bulb is burned out. Image acquisition computers, data storage devices and networking
components are also rack mounted (Figure 5).
A. Berry Page 5

Figure 5: Rack Mounted Components

Image Acquisition System

The line scan cameras are positioned on the beam in a manner that optimizes the view of the rail
web while still falling within Plate C requirements. The system is composed of four cameras, one
each for both field and gage sides of the left and right rails. The cameras continuously image the
rail as it is traversed. A high resolution encoder supplies the cameras with a 0,5 mm (0.02 inch)
fixed distance trigger. Upon receipt of each trigger pulse, the camera acquires a grayscale line
scan of the rail from the top of the rail head extending to the ballast. The cameras then send the
images to a data collection computer for assembly and analysis.

Frame grabbers housed within the computers receive the images from the camera. High speed,
multiprocessor computers assemble the images from the cameras line by line. The images are
assembled and tagged with synchronization information supplied by a counter timer board.
A. Berry Page 6

Image Processing Algorithms

After the images are assembled, the first image processing algorithm evaluates the images for the
presence of a joint bar. The joint bar is detected when the algorithm identifies a specific joint bar
pattern within an image (Figure 6).

Rail head Detected

joint bar edges

Gap

Figure 6: Illustration of Joint Bar Detection Algorithm

Joint bar images are extracted from the rest of the data. GPS coordinates and milepost/footage
location are received through a serial port on the computer and the images get tagged with this
information.

Once a joint bar is detected, a second algorithm scans the image for cracks in the joint bar. The
algorithm identifies different areas on the bar where a crack is more likely to be present (Figure 7).
Anomalies in the pixilation of the image are acknowledged and rated on their likeness to the
characteristic pattern of a crack. The length, width, aspect ratio, contrast, defining shape, and
other features of the suspected defect are all considered by the algorithm (Figure 8). Based on
these criteria, the algorithm prescribes a probability rating to the defect. Images tagged with a high
defect probability are flagged and sent to the operator for consideration.

All image processing occurs in real time. A compression board compresses the images and sends
them to the user interface computer for operator evaluation and inventory reporting.
A. Berry Page 7

Figure 7: Center Crack Detection Region

Detail of center region

(raw data)

Result of detection filter

Result of thresholding and

connected components
search

Figure 8: Center Crack Detection Filter

Operator Interface and Software

All joint bar images, whether deemed defective or not, are sent to a database with corresponding
milepost and footage data as well as GPS coordinates. Detected defects are sent to the operator
interface and are accompanied by an on-board alarm. The image of the defect is presented to the
operator with the defect highlighted for easier consideration. The resolution of this image is very
high, enabling the operator to easily evaluate the defect. The image can be panned, zoomed and
scrolled, and the operator can add comments to the image, delete an image, or accept it as a
defect.
A. Berry Page 8

An automated report of the data may be generated. The operator can choose to generate defect
reports of confirmed cracks. Inventory reports of all joints present on a line may also be generated.
The inventory reports include GPS coordinates of each event as well as milepost and footage
location.

Track features such as switch points, frogs, bridges, and road crossings may also be entered into
the database via an operator console in order to create a complete inventory of all significant track
elements. The database and corresponding images can be accessed at any time in the future,
allowing for evaluation of track wear over time.

Figure 9: Operator Interface

INDUSTRY USAGE AND SYSTEM PERFORMANCE METRICS

The prototype system developed for FRA OR&D was tested on several major railroads through the
course of development. Once the system was deemed fully functional, it was tested on three
major railroads across the United States. The purpose of these tests was to acquire data to train
the algorithm with. After completion of each test, the crack detection algorithm was reviewed and
modified based on the real-time performance of the system during the test. Defects in the dataset
were used to refine the algorithm in order to decrease the false detection rate and increase the
number of true defects detected.

After high confidence in the system was achieved, ENSCO presented this technology under the
Trademark VisiRail for the North American market. The first production VisiRail system was
installed on a rail-bound geometry car owned by one of the seven Class 1 Railroads in North
America. This installation began in late 2006; the system was commissioned and went into full
time operation in October 2007. A second VisiRail system installed on a hy-rail vehicle was
A. Berry Page 9

commissioned in August 2007 and surveyed track for both Short Line Railroads and Class 1
Railroads.

FRA’s system is also operational and is used to collect data for research and development
purposes, as well as to augment testing by FRA Office of Safety. Additional VisiRail systems will
be deployed on both rail-bound and hy-rail vehicles in 2008.

Mileage

In total, the three systems that were in full operation by the end of 2007 surveyed approximately
2,850 miles of Continuously Welded Rail (CWR) and 1,315 miles of jointed rail for a total of 4,165
miles.

Joint Count

The system inventories all joints present on a line, and this information is stored in a
comprehensive database complete with GPS coordinates and milepost and footage information.
From the three operational systems, a total of 357,000 joints were inventoried by the system
(714,000 joint bars). This averages to 86 joints per mile.

The algorithm’s ability to correctly identify a joint depends largely on the quality of the image.
Under good track conditions, the joint detection accuracy is 98%, meaning that the algorithm will
detect 98% of the joints. With compromised image quality the ability of the system to identify a
joint is decreased to 85%. The largest contributor to compromised image quality is wet rail, which
darkens the image and decreases the contrast.

The false joint detection rate is also affected by image quality and track conditions. False joint
rates can either be presented as a percentage of total joints, or as a number of joints per mile.
Using the joint per mile statistic normalizes the false joint by accounting for the difference in joints
per mile on CWR versus jointed rail.

With clean, dry rail conditions, the false joint detection rate is 0.4 joints per mile. Under poor
conditions and water-streaked rail, the false joint rate is 1.6 joints per mile.
A. Berry Page 10

Automated Crack Detection Statistics

As with the joint detection statistics, the crack detection statistics depend largely on rail condition.
With clean and dry rail conditions, the system will accurately identify over 80% of the cracked or
broken bars present. Overall, the algorithms will flag ~5% of all joints inventoried as defective.
Typically, one in ten of the flagged bars will contain a true crack.

Wet rail conditions again pose the greatest challenge to the algorithms. When the rail is streaked
with water, the likelihood of a false joint detection increases. Often when a false joint is detected,
a false crack will also be detected on the image. Under these conditions, false joints account for
45% of the false cracks.

The sensitivity level of the crack detection threshold can be adjusted to increase the detection rate
of defects; however, adjusting this variable also reduces the filter for false positives. Therefore, as
the true detection rate is increased, the false detection rate will increase as well. At some point,
the value of an increase in the true positive detection rate is significantly compromised by the
corresponding increase in the false detection rate. This concept is best presented by a Receiver
Operating Characteristic (ROC) Curve. The Crack Detection ROC Curve (Figure 10) shows that
for a true positive detection rate of 80% or less, the false detection rate is reduced to around 3%.
However, for a 10% increase in the positive detection rate, to 90%, the false detection rate
increases by 20%.

The most common cause of a false detection is detection of a crack on an image which does not
contain a joint bar. This often occurs on water-streaked rail. Other common causes of false
positives are frayed bond wires, rust and grease spots, weeds, and debris.

Several examples of verified joint defects identified by the system are provided (Figures 11-14).
An example of an image that resulted in a joint bar being falsely identified as a defective joint is
also provided (Figure 15).

Crack Detection ROC

100
90
80
70
Detection Rate

60
50
40
30
20
10
0
0 20 40 60 80 100
False detection rate

Figure 10: Crack Detection ROC Curve

A. Berry Page 11

Images of Detected Defects

Figure 11: Tiny Hairline Center Crack Detected by Automated System

Figure 12: Hairline Center Crack Detected by Automated System

Figure 13: Center Break on Field and Gage Bars, Field Verification Image and Top View of Breaks
A. Berry Page 12

CONCLUSION
Advanced image processing techniques can greatly increase the speed and effectiveness of joint
bar inspection. The system presented in the paper demonstrates the ability to capture high
resolution images of rail at speeds up to 105 km/hr (65 mph). The system can also operate under
certain adverse weather conditions such as rain and light snow.
Through the course of development of this system, many capabilities and limitations of line scan
and image processing technologies have been discovered by the development team. Establishing
a lighting system which would provide optimal lighting conditions for the railroad environment was
a large part of the developmental process. Detailed image analysis pointed to further areas of
improvement as the system approaches maturity.
Further work is being done to improve the detection algorithms. Different classifiers are being
introduced to increase the likelihood of detecting a crack without impacting the false detection rate.
A filter specifically designed to detect the shape and location of bolts will eliminate many of the
false detections. Adding dynamic image correction features will also reduce the effects of uneven
illumination and poor rail conditions on the statistical performance of the system. Improvements to
the user interface will also speed the review process.
The capabilities of this system will be expanded in the next year to include missing bolt detection,
rail gap, and rail batter. The technology is also being applied to inspect other track features such
as fasteners, tie plates, switch points and frogs.

ACKNOWLEDGEMENTS
The authors would like to thank those in the industry who supported the development of this
system. Much of the work was funded by FRA OR&D. Mr. Gary Carr and Dr. Magdy El-Sibaie of
FRA OR&D assisted in the development of this system with their vision, leadership and knowledge
of the railroad industry. Ron Gagne and Mike Roney (Canadian Pacific) partnered with FRA
OR&D and ENSCO to assist in the refinement of the system. Field tests and demonstrations were
made possible with the assistance of Tom Toth (Union Pacific), Dave Thornton (Burlington
Northern Santa Fe), Steve Lakata (Norfolk Southern), Irv Taylor and Bruce Wigod (New Jersey
Transit), and Dan Pluta (Western Maryland Scenic).
 A. Berry Page 13

AUTHOR BIOGRAPHIES
Andie Berry is a Senior Engineer and Product Manager at Ensco, Inc., Applied Technology and
Engineering Division in Springfield, VA.
Ms. Berry works on the design and implementation of systems for track inspection and asset
management. She has worked to develop technology for visually based inspection systems
including the joint bar inspection system. She also manages both commercial product
development as well as governmentally funded technology research. Her areas of expertise
include electrical system design, circuit board design, semiconductors, and project management.
Ms. Berry received dual Bachelor of Science Degrees in Electrical Engineering and Biomedical
Engineering from Duke University in 2001. Her concentrated studies were in VLSI design,
biomedical instrumentation, and semiconductor physics.

Boris Nejikovsky is a Chief Engineer of ENSCO, Inc Applied Technology and Engineering Division.
He holds advanced degrees in electrical and civil engineering. His career has been dedicated to
advancing the state of the art in railroad and civil engineering instrumentation, measurement
techniques, and inspection technologies.
Mr. Nejikovsky has been with ENSCO, Inc. for 15 years. While at ENSCO, he managed
development of multiple track and vehicle inspection systems based on various technologies. He
has broad expertise in the areas of track geometry, ride quality, track strength, rail profile, and rail
corrugation measurements. He is also an expert in automated track video inspection, vehicle track
interaction monitoring, and vehicle testing and evaluation.
Mr. Nejikovsky is a member of the American Railway Engineering and Maintenance of Way
Association Committee 2 for Track Inspection Systems. He has authored numerous papers and
technical reports on track inspection and vehicle testing and holds several patents in the area of
instrumentation and railroad inspection technology.

Ali Tajaddini is a Program Manager in Office of Research and Development of Federal Railroad
Administration. He received his B.S. in Civil and Environmental Engineering in 1980 and M.S. in
Structural Engineering and M.S. in Engineering Mechanics in 1984 from University of Wisconsin-
Madison. He is registered as Professional Engineer in states Maryland and Wisconsin. He has
been working in FRA since 1999. Prior to working at FRA he worked as structural engineer and
analyst for Auto industry and he worked for 10 years at research and test Department of
Association of American railroads in Chicago Technical Center and Transportation Technology
Center and two years at ENSCO, Inc doing research in area of vehicle track interactions, Track
geometry and wheel Impact loads.

View publication stats