See

discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/224056595

Intelligent Parking System Design Using FPGA

Conference Paper · September 2006

DOI: 10.1109/FPL.2006.311249 · Source: IEEE Xplore

CITATIONS READS

8 600

7 authors, including:

Insop Song Fakhri Karray

University of Waterloo University of Waterloo
43 PUBLICATIONS 196 CITATIONS 427 PUBLICATIONS 4,609 CITATIONS

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Developing Next Generation Intelligent Vehicular Network and Applications (DIVA) View project

Multiple cooperative swarms clusering View project

All content following this page was uploaded by Fakhri Karray on 20 July 2014.

The user has requested enhancement of the downloaded file.

 INTELLIGENT PARKING SYSTEM DESIGN USING FPGA

Insop Song, Keith Gowan, Jason Nery, Henrick Han, Tony Sheng, Howard Li, Fakhreddine Karray

Department of Electrical and Computer - University of Waterloo

200 University Ave. W, Waterloo, ON Canada
{isong,khgowan,jnery,hahan,tlsheng,h23li,karray}@uwaterloo.ca

ABSTRACT To accomplish RTOS based multi-thread real-time applica-

tion, a CPU and memory are essential. In order to guarantee
In this research, we introduce FPGA based fuzzy logic con-
that all (or most) of the tasks’ deadline are to be met, a pri-
troller (FLC). The benefit of using FPGA based FLC com-
ori schedulability test should be performed, in case of hard
pare to software FLC is that the computation time reduction.
(or semi hard) real-time applications. In addition, sharing
Using this FLC, we design automated car back parallel park-
resource protocol and interprocess communication are also
ing system also with complete FPGA based controller. We
carefully designed and and tested to avoid deadlock and pri-
build a small-scaled robot car and test on a real environment
ority inversion. Our application scenario, a robot car con-
with VHDL code for wall following and parking. This paper
trol, does not require very high-speed interactions among
describes the background of fuzzy logic system, the design
tasks (threads); however, this does not mean that we can
of fuzzy logic system with FPGA and the experimental re-
be free from the above mentioned real-time design require-
sults.
ments: schedulability test and resource protocol test.
1. INTRODUCTION One of the common mis-concepts of real-time systems
For a novice driver, parallel parking is a very challenging are a real-time system runs very high-speed not slow-speed.
task. This will become even more so, since the number It is not; a real-time system is the system that should meet
of car grows annually, and parking space in urban area be- tasks’ deadline, either fast or slow. In this regards, our ro-
comes tighter. To assist drivers, industry and academia have bot car system is a real-time system, i.e. the robot should
researched parking assistance systems [1,2]. In this research, turn within certain time to avoid crash, a sensor data should
we present a fast hardware-based (FPGA-based) automated be processed periodically as examples. We did use a CPU
parking assistant system. We design (1) a hardware accel- and memory, and real-time design tests were performed in
erator (co-processor) for fuzzy logic controller that controls our previous research [2]; however, in this research we want
car movement and (2) a finite state machine that controls to carry out the same task using hardware-based method in
operation mode. order to use low-cost silicon chip and fulfill real-time re-
This research develops two important hardware-based lated design requirements with less efforts. These method-
components and accomplishes a challenging task, a robot ological reasons are the main reasons that we perform this
car parallel parking control. Both the hardware accelera- hardware-based research. Also hardware-based design re-
tor for fuzzy logic controller (FLC) and the FSM machine duces throughput time dramatically compared to software-
can be applied to other different types of applications. The based design.
hardware accelerator for FLC can be used many different Based on previous design experiences [2] of fuzzy logic
types of automatic controller as a supporting co-processor controller and parking expert knowledge, we re-design the
to a main CPU or as a stand-alone component. There are system with a complete hardware-based way in order to ac-
various potential applications of this FLC hardware acceler- complish more prompt and reliable system outputs, in addi-
ator, since numerous products use FLC, such as a rice cooker tion to the above mentioned methodological reasons. One of
to a steel pressing machine. Also a complicated FSM can be the important advantages of designing hardware-based sys-
used for many consumer and industrial devices, and it can tem is that it responses much faster then software-based sys-
replace a microcontroller. As a matter of fact, the automo- tem in most cases. It is because that computation is per-
tive industry started look at FPGA devices of its low silicon formed in parallel with logic gates on a silicon unlike the
cost and existing reference designs [3]. way of sequential computations in software-based system.
In our previous work, a software-based (C++ program) Hardware based system is built on FPGA (Field Pro-
real-time parking system was developed [2]. Multi-thread grammable Gate Array) chips with VHDL (Very high speed
application was designed based on real-time OS (RTOS). integrated circuits Hardware Description Language) language.

1-4244-0 312-X/06/$20.00 2006

c IEEE.
VHDL describes hardware logics in gate level circuits. We is not easy to exactly modeled by the various complexity of
use Altera’s Startix II FPGA board [3]. the system parameters. Thus, in such a complexity involved
The parking assistant system is fully autonomous and case, FLC is very suitable.
intelligent. It consists of two main components. A central At the beginning of designing fuzzy systems, we should
controller keeps track of external environmental information collect fuzzy IF-THEN rules from human experts or domain
as well as internal states. A Fuzzy Logic Controller (FLC) knowledge.
[4] controls car to drive straight with predetermined distance The fuzzy logic system shown in Figure 1 is a simple
from external objects, the FLC is used as a basic module for diagram of fuzzy system that is used in this research. Its in-
the central controller. puts and outputs are fuzzy sets; without loss of generallity,
A reduced-scale robot car is built for experimental test. the problem of this type is that most of the engineering sys-
The car is equipped with steering mechanism, speed con- tems have crisp-valued variables, not fuzzy value. Thus, we
trol mechanism, and distance measuring sensor data acqui- need fuzzifier that converts crisp input to fuzzy input. Then
sition systems. Thus, the car steers exactly as a full scale car fuzzy inferencing is performed using the fuzzified input val-
(with differential gear), and it moves forward and backward ues and fuzzy rule base. Afterwards defuzzifier finally con-
with different speeds. Five sensors and ADC measure the verts fuzzy value to crisp output. We use
distance between the car and external environment, such as Takagi-Sugeno-Kang (TSK, or also called Sugeno model)
other parked cars or curbs. proposed real-valued variables inputs and outputs [4] as shown
A FLC simulator is also designed on PC to develop FLC. a Weighted Average block in Figure 1. TSK fuzzy system
The simulator has two modules: first module is for FLC sim- can be defined as a weighted average of the values in the
ulation, and the second module is for FLC VHDL code gen- THEN parts.
erator. The simulator is designed to test FLC and parking In short, an important contribution of fuzzy systems the-
operation. Once we find proper FLC parameters based on ory is that it provides a systematic procedure for transform-
simulations, the simulator is able to generate FLC VHDL ing a knowledge base into a nonlinear mapping [5]. The
code based on predefined code template with corresponding applications of the fuzzy systems are countless, such as do-
parameters. The VHDL template is designed in advance, mestic appliances, an digital image stabilizer, automobile
and the code generator is used to expedite integration time. control systems, subway control systems, and manufactur-
In experimental test, first we develop the fuzzy logic ing control systems.
control for wall following, which maintains constant dis-
The fuzzifier maps from a crisp point, x ∈ U , in to a
tance from environments. We design two different types of
fuzzy set A
in U . A fuzzy rule base consists of a collection
wall following: short distance wall following and far dis-
of fuzzy IF-THEN rules as (2).
tance wall following for different environmental situations.
The defuzzifier maps from fuzzy sets in V to a crisp
We test FLC wall following on different shapes of wall:
point y ∈ V . There are many defuzzifiers, but followings are
straight, curved, or corner. Then a finite state machine (FSM)
three possible choices, such as Maximum, Center of gravity,
of the central controller is designed for back parallel park-
Center average, and Modified center average. Since we use
ing. We design and test back parallel parking with various
TSK type FLC, the defuzzifier is as in (4). The details of
sizes of parking spaces, and the car performs reliable and
this equation will be explained in the next Section.
fast parallel parking.
In this paper we consider a complete FPGA-based in- In this section, we briefly review the fuzzy system. Fuzzy
telligent parking system deign using FLC and central con- system composed as follows: Fuzzy rule base, Fuzzifier, and
troller. The organization of this paper is as follows: In Sec- Defuzzifier. The importance of fuzzy system is its incorpo-
tion II, we describe the background of Fuzzy system in de- ration with linguistic information in a natural and systematic
tail. In Section III, we explain how do we design FLC for
the robot car system. In Section IV, we show a FPGA de-
sign and the robot car design, and in Section V, we describe
experimental results. Finally, in Section VI we discuss the
results of the paper and provide possible future research ar-
eas.
2. BACKGROUND: FUZZY SYSTEMS
Fuzzy logic controller (FLC) can produces an approximate
control output for systems that are complex or non-linear,
which normally are difficult to modelled and solved analyt-
ically. In addition, FLC can use an inferencing method of Fig. 1. Basic configuration of fuzzy systems with fuzzifier,
human expert knowledge [4]. In our system, car driving task fuzzy rule base, and defuzzifier
 membership function

NEG Z POS

1.0

-8 -4 0 4 8

Fig. 2. Sensor location on the side of the robot car. Fig. 3. Fuzzy membership function of input variable.

way. Following section describes a the fuzzy controller de- to the side wall. f4 has this value; we obtain these zi values
sign and its experimental results. from expert knowledge and simulations.
R1 : IF x is ZERO and y is ZERO, T HEN θ is f1
3. INTELLIGENT PARKING SYSTEM DESIGN
R2 : IF x is P OS and y is ZERO, T HEN θ is f2
In this section, we describe FLC design of wall following R3 : IF x is N EG and y is ZERO, T HEN θ is f3
module, which is a main part of parking task. This wall R4 : IF x is ZERO and y is P OS, T HEN θ is f4
following task controls the robot car to maintain a given dis- R5 : IF x is P OS and y is P OS, T HEN θ is f5
tance from side objects using FLC. Then, in the next sub- R6 : IF x is N EG and y is P OS, T HEN θ is f6
section, a central controller is explained. The central con- R7 : IF x is ZERO and y is N EG, T HEN θ is f7
troller is a finite state machine that include driving experts R8 : IF x is P OS and y is N EG, T HEN θ is f8
knowledge of performing back parallel parking. R9 : IF x is N EG and y is N EG, T HEN θ is f9
3.1. Fuzzy Logic Controller Design for Wall Following (2)
After fuzzification and fuzzy inferencing using fuzzy rule
Fuzzy logic controller is designed and simulated to main- base, a final step has to be done, which is converting fuzzy
tain given distances from side objects, such as curbs or other values to crisp values in order to actually steer the robot to
cars. This wall following task is accomplished by measure follow wall, i.e. defuzzification step. Defuzzificaiton is per-
the side distance from two locations of the car: front side formed as in (4). fi are from fuzzy rule base and a weighting
and back side. As show in Figure 2, there are three sensors parameter wi is calculated from the membership operation
on the side of the robot, and there are two other sensors on shown in (3). wi is obtained by either “min” operation or
each front and the back of the robot car. The main sensors “product” operation shown in (3). θ, crisp value, is obtained
that are used in wall following task are D3 , front side sensor by (4).
and D4 , back side sensor. Dw is defined a given reference wi = min[µAi (x), µB i (y)] (3)
distance to be maintained as in Figure 2. ,where µAi (x) is the membership function of x and µB i (x)
is the membership function of y.
With these three values, we define two input variables
9
for FLC as in (1). x in (1) and in Figure 2 is a measure of i=1 wi · fi
θ=
9 (4)
the angle of the car reference to the side wall. It tells us that i=1 wi
how steep the robot is with the angle towards the side wall. Using the FLC wall following, the robot car can follow
On the other hand, y in (1) and in Figure 2 is a measure short distance as well as far distance wall following task.
of a distance shows how far the car is off from the given
reference distance, Dw . 3.2. Central Controller Design for Parallel Parking
To accomplish full automated parallel parking, we designed
x = D4 − D3
(1) a central controller. It is composed with FSM that checks
y = D4 − DW
corresponding sensor values and controls the car steering
Membership function is designed for two inputs, x and and direction. It is a very complicated FSM; however, we
y, as shown in Figure 3, horizontal axis is measured dis- try to simplify and show only important state operations and
tance. It is intentionally designed simple for hardware im- state transition conditions as shown in Figure 4.
plementation of fuzzificaiton, so we only use three fuzzy State 1 performs short distance wall following using FLC
variables: Z is zero, POS is positive, and NEG is nega- until a gap (a parking space candidate) is detected, which is
tive. These membership functions are simple triangular type transition condition A. Then in State 2, the robot changes its
membership functions. mode to far wall following also using FLC until the gap is
Using human expert knowledge, fuzzy rule base is de- ended, which is transition condition B.
signed as in (2). θ is a steering angle of the robot car. For After the gap is finished, in State 3, based on time taken
example, as in R4 if x, an angle measure, is ZERO and y, and the speed of the car, we can find out how big is the park-
distance measure, is P OS, then we can turn steering much ing size. Thus, if the gap size is big enough for parking, then
 Start C
If space is not
enough for parking

1 2 3

Short distance A Far distance B Measure

wall following If gap is wall following If gap is gap size
detected end

FLC is used FLC is used If space is

D enough for
parking

6 5
F Move back & E
Stop & move
steer straight
foward Back & side
Back sensor & left back sensor
triggerred triggerred
Front
sensor
FLC FLC is used 4 Fig. 5. A cut out picture of the robot car. Inside the robot,
is
triggered
G
used If too far
Move back & there are two boards: Altera board and custom made daugh-
from side
curb turn hard right ter board.
End H

sor (GP2D120 from Sharp) and another kind is a long range

Fig. 4. A very simplified finite state machine of central con- sensor (GP2D12 from Sharp). Three sensors are installed
troller. on side of the car, and two sensors are installed on front and
back of the robot car, respectively.
state moves to State 4. In State 4, the car moves back with 4.2. VHDL Design for Intelligent Parking System
hard right turn steering until it detects certain distances on There are two main VHDL modules: FLC and central con-
back sensor and back side sensor. State 5, then, has initiated troller. Also there are few other VHDL modules: ADC con-
with straight and then gradual left steering, which is kind of trol, direction control & speed control, steering conversion
wall following in backward movement. State 5 is finished from FLC output, and operation mode selection & display.
when the back sensor detects very close distance. In State We also add a moving average filter at sensor module to
6, the car moves forward using shorter wall following until eliminate noisy inputs.
front sensor is triggered. In Figure 6, controller and FLC can be seen, and ADC
In short, the automated parking task is accomplished by control module that feeds distance information to the con-
the combination of FSM and FLC, and these are designed troller and FLC. fi (i fi ) values is feed by using lpm constant
based on using human expert knowledge. for easy debugging.
4. SYSTEMS DESIGN WITH FPGA Due to the limitation of the space, we only show FLC
In this section, design of robot car and VHDL for fuzzy logic block in Figure 7. Two fuzzifiers are located in the bottom
control (FLC) and central controller are described. in Figure 7, and there are also min, multiply, adder, and
divider modules, which computes final steering output. It
4.1. Robot Car Design takes only 21 clock cycles to produce fuzzy output. This is
A RC (radio control) car frame is purchased and other addi- much reduced processing time compared to that of software
tional circuits are integrated as shown in Figure 5. based FLC in the previous research; it takes around hun-
A main control FPGA board is Altera Stratix II board dreds of cycles. Much more importantly, in hardware accel-
[3]. A daughter board is designed for power distribution and erated FLC, the processing time is fixed and deterministic
ADC & MUX circuits. unlike the software couterpart. The hardware-based system
Steering control is done by radio control servo controller, uses 8,900 logic elements. We use many debugging purpose
and its input is PWM (Pulse Width Modulation) signal. VHDL lpm constant blocks to modify parking parameters on the fly
code is designed for PWM generation. Direction and speed without recompile VHDL files. Also, we use SignalTap [3]
control is done by a built in speed controller that comes with for real-time monitoring of internal data. If we take out these
the car frame and body. It is also controlled by PWM signal, development related modules, the number of logic elements
so another PWM generation VHDL code is designed. will be reduced.
A 10-bit resolution high-speed parallel output ADC (ADC
10461 from National Semiconductor) is used to convert dis- 5. EXPERIMENTAL RESULTS
tance sensor signal. Since ADC10461 does not have enough In this section, experimental results are presented. First,
channels for five sensors, we use multiplexer (MC14051B). simulator and code generator are presented, and actual ro-
ADC read timing and multiplexer selection are also done by bot run is showed.
VHDL code, so that FLC and FSM can access distance val- 5.1. Simulation and VHDL code generation
ues from each sensors. With a simulator, a researcher can choose how many mem-
Five infrared distance measuring sensors are used. Two bership functions for input 1 and input 2, as well as their
kinds of sensors are used, one kind is a short range sen- shapes. After the parameters are set, a simulated robot can
 min_zero1_zero2 multiply
ADC_CNTL FLC z1 i_clk m1 i_clk
i_input1[9..0] o_min[9..0] i_input1[9..0] o_val[9..0]
i_int o_RD i_f1[9..0] to i_input2[9..0] i_input2[9..0]
to mux z2 i_f1[9..0] Adder:NumAdder
i_data[9..0] o_SH i_f2[9..0] o_steer[9..0] steering min_zero1_pos2 multiply
i_clk
i_clk m2 i_clk i_input1[9..0] o_val[31..0]
i_reset o_data[11.0] pwm z1 i_input1[9..0] o_min[9..0] i_input1[9..0] o_val[9..0]
i_f3[9..0] i_input2[9..0] i_input2[9..0]
i_input2[9..0]
i_clk o_mux[2..0] i_f4[9..0] p2 min_zero1_neg2
i_f2[9..0]
multiply
i_input3[9..0]
from i_clk i_clk
lpm_const
i_f5[9..0] z1 i_input1[9..0]
o_min[9..0]
m3
i_input1[9..0] o_val[9..0]
i_f6[9..0] i_input2[9..0] i_input2[9..0]
n2 i_f3[9..0]
controller min_zero2_zero1 multiply
i_f7[9..0] i_clk i_clk
z2 i_input1[9..0] m4
i_f8[9..0] o_min[9..0] i_input1[9..0] o_val[9..0]
i_input2[9..0] i_input2[9..0]
i_adc[9..0] i_f9[9..0] z1
min_zero2_neg1
i_f4[9..0]
multiply
Adder:NumAdder Adder:NumAdder
i_clk i_clk
i_reset o_distance[9..0] i_distance[9..0] z2 i_clk m5 i_clk i_input1[9..0] o_val[31..0] i_input1[9..0] o_val[31..0]
i_input1[9..0] o_min[9..0] i_input1[9..0] o_val[9..0] i_input2[9..0] i_input2[9..0]
i_clk o_angle[9..0] i_angle[9..0] n1
i_input2[9..0] i_input2[9..0]
i_input3[9..0] i_input3[9..0]
to i_f5[9..0]
min_pos1_pos2 multiply
i_param o_speed[9..0] direction i_clk p1 i_clk m6 i_clk
pwm i_reset i_input1[9..0] o_min[9..0] i_input1[9..0] o_val[9..0]
i_input2[9..0] i_input2[9..0]
p2 i_f6[9..0]
min_pos1_neg2 multiply
p1 i_clk m7 i_clk
i_input1[9..0] o_min[9..0] i_input1[9..0] o_val[9..0]
Fig. 6. VHDL modules for intelligent parking task. n2
i_input2[9..0]
min_neg1_pos2
i_f7[9..0]
i_input2[9..0]
multiply
Adder:NumAdder
i_clk
i_clk m8 i_clk i_input1[9..0] o_val[31..0]
n1 i_input2[9..0] i_input1[9..0] o_val[9..0]
o_min[9..0] i_input2[9..0]
i_input1[9..0] i_input2[9..0] division
p2 i_f8[9..0] i_input3[9..0] i_clk
min_neg1_neg2 multiply
i_num[9..0] o_val[9..0] FLC
be tested on a simulator. After few testing, the parameters i_clk
n1 i_input1[9..0]
o_min[9..0]
m9 i_clk
i_input1[9..0] o_val[9..0]
i_den[9..0] output

i_input2[9..0] i_input2[9..0] o_steer

are fixed, then FLC VHDL code is generated. This FLC n2 i_f9[9..0] [9..0]
Adder:DenAdder
VHDL code is integrated with other VHDL modules of the fuzzifier1
i_clk
i_input1[9..0] o_val[31..0]
m1
robot’s central controller. Figure 8(a) shows a screen capture i_clk o_neg[9..0]
i_data[9..0] o_zero[9..0]
n1 m2
z1 m3
i_input2[9..0]
i_input3[9..0]

of one of the parameter setting menus. Figure 8(b) shows o_pos[9..0] p1 Adder:DenAdder
i_clk
Adder:DenAdder
i_clk
i_distance
a screen capture of a robot run on simulator on a IBM PC. [9..0] fuzzifier2
m4
m5
i_input1[9..0] o_val[31..0]
i_input2[9..0]
i_input1[9..0] o_val[31..0]
i_input2[9..0]
n2 m6 i_input3[9..0] i_input3[9..0]
Figure 8(c) shows a screen capture of FLC VHDL code gen- i_clk o_neg[9..0]
i_data[9..0] o_zero[9..0] z2 Adder:DenAdder
o_pos[9..0] p2 i_clk
eration. i_angle m7 i_input1[9..0] o_val[31..0]
m8 i_input2[9..0]
[9..0] m9 i_input3[9..0]
5.2. Back Parallel Parking
First, we test wall following by FLC. In Figure 9(a), a short
Fig. 7. VHDL modules of FLC. Inside details of FLC block
distance wall following mode is activated as long as there
in Figure 6
is no big parking space is existed on the side, such as up to
number 1 in the Figure 9(a), the car maintains short distance
from the side curb using FLC. After passing number 1 in based design is much reduced compared to that of software
Figure 9(a), the car changes its mode to the far distance wall based design. It is because that, in hardware designs, most
following until it passes number 3, another close distance of the logics are synthesized in parallel; thus operations can
object. As shown in Figure 9(a), the car tries to maintain the be processed parallel. (2) real-time design requirements are
predefined distance. After the number 3 in Figure 9(a), the easier to met then ones with multi-thread software based de-
car calculates a parking space that just passed, if it is long sign under RTOS. In future work, we want to integrate the
enough for parking or not. FLC hardware accelerator (FPGA-based FLC), as a hard-
If it is long enough, the car changes its from state 3 to ware thread, with a software & CPU based automatic control
state 4 as in Figure 4. As in Figure 9(b), starting from num- (or other FLC reqired) systems. Then, both flexible and fast
ber 4, the car backs with hard right turn until it has objects response control system are expected, and less complicated
detected on side and back side sensors. Then it releases its real-time design requirements should be tested.
steering to straight and to soft left as at number 5 in Figure 7. REFERENCES
9(b). Then, FLC shorter wall following in backward direc-
tion is triggered and steers the car. It keeps backing up till [1] E. Batista, “Park without dings – or drivers,” Wired News,
an object is detected by the back sensor as at number 6 in August 25 2003. [Online]. Available: http://www.wired.com/
Figure 9(b). news/technology/0,1282,60154,00.html
As shown in Figure 9(c), the car moves forward till front [2] I. Song, F. Karray, Y. Dai, M. Masmoudi, and B. Ghaddar, “An
sensor is triggered. It stops in the middle of the parking intelligent car-like robot parking system design and implemen-
tation,” in Proceedings of Third IEEE International Confer-
space. This forward movement is also helped by shorter dis-
ence on Conference on Systems, Signals Devices, SSD, 2005.
tance wall following FLC.
[3] “www.altera.com.”
6. CONCLUSIONS [4] F. Karray and C. W. D. Silva, Soft Computing and Tools of In-
We design, simulate, and test FPGA-based automated park- telligent Systems Design: Theory and Applications. Addison
ing system with both a simulator and an actual robot car. Wesley, 2003.
The advantages of using hardware vs. software-based de- [5] L.-X. Wang, A Course in Fuzzy Systems and Control. Upper
sign are as follows: (1) the computation time of hardware Saddle River, New Jersey: Prentice Hall, 1997.
 2
1 3

(a) Simulation program screen shot. Rule base input menu.

(a) State 1, 2, 3.

5
6

(b) Simulation run screen shot. (b) State 4 and State 5.

7 8

(c) State 6.

Fig. 9. Overlayed picture of automated parking operation.

(c) FLC VHDL code generation screen shot.

Fig. 8. FLC simulation and FLC VHDL code generator

screen shots.

View publication stats