International Journal of Systemics, Cybernetics and Informatics (ISSN 0973-4864)

Real-Time Machine Vision Based Robot

G. Sathya G. Ramesh Chandra

Department of Electronics and Communication Engineering Research Engineer
VIF College of Engineering, Moinabad, Hyderabad Pentagram Research Centre Pvt Limited
Andhra Pradesh, India 201, Venkat Homes, MIGH – 59, Mehdipatnam
E-Mail: sathya_quake@yahoo.co.in Hyderabad – 500 028, Andhra Pradesh, India

Abstract
This paper discusses the practical implementation of Ø A USB to RS-232 converter (In case if the PC doesn’t
a real-time machine vision based autonomous mobile robot. have a serial port)
The relevant image processing concepts are implemented § Length of wire should be at least 2 meters
programmatically and the control of the servo motors is done Software requirements and their specifications
using the Basic Stamp 2 microcontroller. Some minor Ø Operating system
concepts regarding the serial-port or RS-232 communication, § Windows XP 32 or 64 bit
and block diagram are discussed. § Service Pack 2 or 3
. Ø Visual Studio .NET 2008 with MSDN
Keywords: AMR, Machine Vision, Target Tracking Ø DirectShow.NET API with AForge.NET wrapper
Ø Webcam device driver
1. Introduction Ø Driver for USB to RS-232 converter (In case if the PC
1.1 Experimental requirements doesn’t have a serial port)
The requirements for this project are divided into These are the minimum hardware and software requirements
two categories. One is the hardware requirements and other is as far as this robot is concerned. Use of advanced image
the software requirements. Both are equally important when it processing and pattern recognition techniques might require
comes to robotics based on machine vision. The requirements high end hardware to process each frame if the frame rate is a
are as listed below. matter of concern.
Hardware requirements and their specifications 1.2 The Block diagram
Ø Parallax Boe-Bot Robot Kit The block diagram illustrates the various stages and
§ Basic Stamp 2 microcontroller sub stages that are involved in the system. The block diagram
§ Board of Education also illustrates the closed loop nature of the system.
§ Robot’s aluminium body
§ Two Parallax servo motors
§ Four 1.5V AA sized battery
§ A RS232 serial cable
Ø A desktop or laptop PC
§ Processor: Intel Pentium4 or higher / AMD Athlon
4400+ dual core or higher
§ Hard-disk: 20 Gb or higher – preferably SATA drive
§ RAM: 512 Mb or higher with good bandwidth for data
transfer
§ Motherboard: A RS232 serial port and at least two
USB ports should be available
Ø A wireless or wired USB web camera
§ Resolution: 320x240 (minimum) or higher
§ Video frame rate: 10 – 15 frames/sec or higher
§ Focusing range: 30mm to infinity
§ Colour: 24 bit (RGB) or higher
§ Length of wire that connects the camera and PC should
be at least 2 meters in case of wired webcam
Copyright © 2009
Paper Identification Number: JAN09-10 Figure 1.2.1: Block diagram of the machine vision
This peer-reviewed journal paper has been robot system
published by the Pentagram Research Centre (P) The CCD (Charged Coupled Device) array is a
Limited. Responsibility of contents of this paper
rests upon the authors and not upon Pentagram
semiconductor device, which captures the incident light and
Research Centre (P) Limited. Copies can be creates corresponding current as the output. Since the output
obtained from the company for a cost. of the CCD video signal is very low, it is to be amplified.
Hence the amplifier strengthens the video signal that comes

61
Real-Time Machine Vision Based Robot

from the CCD. Since the video signal is purely analog in the object is on the left of the image or frame with respect to
nature, it is converted to digital by the analog to digital the reference co-ordinate and vice-versa when the error is
converter. All these three stages reside within the camera. positive. So the output of the comparator is only the error and
Hence a dashed box that encapsulates these three stages is its direction, which is a 1-D signal.
shown in the figure 1.2.1. All the image processing and tracking operations are
After the conversion of the analog video to digital, it performed on a computer programmatically which is
is time to process it. Since we have designed image processing represented by a dashed box in the figure 1.2.1 encapsulating
filters that are meant to be used in spatial domain, we need to both image processing and tracking stages.
convert the incoming digital video signal to a signal that can Once the servo controlling unit receives the error
be represented in spatial domain. Hence we introduce a digital signal from the computer (from comparator in other words), it
buffer memory that stores the incoming signal in each memory sends appropriate control pulses to the servo motors. Based on
location having their respective addresses. Each memory the direction, the desired servo motor will be activated. As
location represents a pixel’s location and intensity value. outlined earlier, the control pulses are nothing but PWM
Hence in this manner, the signal is now a spatial signal on pulses.
which spatial filters can now be applied. The buffer also acts 1.3 The Experimental Setup
as an accumulator for the video frames if the digital spatial The experimental setup deals with the arrangements
filters are no synchronous with the video camera. The spatial can connections that are made in this project. The three main
Gaussian filter is of 3x3 size and having an impulse response equipments in this project are the camera, the robot kit and the
sequence as discussed in chapter 4. Since first pass doesn’t computer. The assembling of the robot kit after its purchase is
prove to be effective on the noise which is Gaussian, the not mentioned as it is already covered in its manual. Figure
spatial signal is passed through another cascaded Gaussian 1.3.1 shows the setup required for object tracking.
filter that helps in reducing the amplitude of the noise further
and thereby making it negligible.
The colour segmentation spatial filter can be
compared to a bandpass filter for a 1-D signal. Each colour in
an image can be attributed to a unique frequency of light. For
example, colour red in the visible region of the
electromagnetic spectrum is of lowest frequency and the
colour violet has high frequency. The colours between violet
and red have frequencies that lie between the frequencies of
red and violet. The colour segmentation filter allows certain
frequencies of light that are of our interest and rejects the rest.
The range of colour values in this case belongs to
the object that is to be tracked. Since the light distribution over
the body of the object is not constant, a small range of values
are taken such that its variation is small. The word
segmentation implicitly tells that the object is discriminated Figure 1.3.1: Setup for object tracking
from the background on the basis of colour. Hence the output The camera is physically attached to the front side of
of this filter is a binary image or frame. The object in the the robot. However, in the figure 7.3.1 the camera is shown
image is represented by white pixels as foreground and the separately for our understanding purposes. The camera is
rest by black pixels as background. Hence we’ll be getting a connected to the PC using a special serial data bus called the
stream of binary images or frames from the output of the Universal Serial Bus (USB).
colour segmentation filter. The BS2 microcontroller that resides on the robot is
Next comes the dilation filter that operates on the interfaced to a serial RS-232 port. Hence the connection
incoming binary images. As discussed in chapter 3, dilation between the microcontroller and the PC is done using the RS-
filter grows the image. In this process of growth, the gaps or 232 serial communication cable. Again for the sake of
holes on the object which represent the background are understanding the connections, the microcontroller and the
eliminated so that the object looks like a continuous rigid servo motors are all shown separately in the figure 1.3.1
body. This makes tracking easier. Image processing ends here though they are fixed to the robot.
and the tracking stage comes next.
In the tracking stage we make use of a filter that 2. Image Processing And Tracking Using PC
extracts the foreground pixel co-ordinates. Next, the The video stream that comes from the camera is fed
foreground pixel co-ordinates which represent the object are to the computer through USB port. The video stream is first
averaged in the pixel averaging filter where the average of all decoded and decompressed digitally. The encoding depends
the co-ordinates of the pixels that represent the body is on the type of the camera uder use. Cheap cameras have low
calculated to give the centroid of the object. This is done for quality CCD compared to the expensive ones and thus the
each frame. compression that is used also would vary accordingly. The
The comparator compares of the centroid with a bandwidth of USB communication is 1.5 to 480 Mbps. The
reference and gives the error of the relative position of the bandwidth in turn depends on the type of USB controller used.
object with respect to the reference. Errors can be positive or Cheap USB controllers do not support high bandwidth.
negative along both x and y axes. But here, we consider only The calculations for a video of resolution 640X480
error along x-axis. If the error is negative, then the centroid of are given in table 2.1.

62
 International Journal of Systemics, Cybernetics and Informatics (ISSN 0973-4864)

Table 2.1: Calculation for 640X480 video resolution Interface which is a set of functions, classes, protocols,
Input Parameters interfaces etc, that are used by applications or programs. For
Pixel size: 640 x 480 example, the Windows API called WIN32 is used by almost
Frame rate: 30 FPS many user programs like Microsoft Word, Excel, Powerpoint,
Colour depth: 8 bit per colour etc., to create graphical user interface (GUI), access files in
Calculation memory and other user interactions. The user program simply
One frame calls the functions or classes present in the API and pass some
Uncompressed size: 921.6 KB variables or parameters, and get the job done.
Moving Image In the case of this project, I made use of third party
Pixel rate: 9.22 Mhz APIs called DirectShow.NET and AForge.NET because there
Uncompressed bitrates: 221.18 Mbps or 27.65 MBps is no managed API for DirectShow. It means that DirectShow
Required storage: 1 second: 27.65 MB is made only for C++ only and not for .NET. The interface
30 seconds 829.44 MB ISampleGrabber is used to grab the individual frames.
1 minute: 1.66 GB Because of the complexity involved in implementing
5 minutes: 8.29 GB DirectShow.NET, we made use of AForge.NET framework
1 Hour: 99.53 GB library that uses Directshow but exposes functions or methods,
From the above calculations, an uncompressed video classes or interfaces that can be easily used to capture the
needs large memory (frame buffer) for temporary storage and incoming frames and process them.
large data transfer bandwidth. If the resolution is high, then we The codes that are developed for carrying out
may expect more memory and bandwidth. Hence it is required various operations in this project are discussed in the form of
to compress the video stream rather than raw data. If the Pseudo-Code. Before we discuss the pseudo code, knowledge
compression algorithm is lossy, then we may lose some high of Bitmap Class is a must since it is used. A bitmap is a type
quality details. Some cameras support compressions like of memory organization or image file format used to store
MPEG4 and other do not. Usually, an uncompressed video is digital images. The term bitmap comes from the computer
of lower resolution. Hence, one has to go in for a trade off programming terminology, meaning just a map of bits, a
when it comes to compression or resolution. spatially mapped array of bits. Bitmap commonly refers to the
The memory that is used to temporarily store the similar concept of a spatially mapped array of pixels. Raster
frames is called Frame Buffer. Typically the frame buffer is images in general may be referred to as bitmaps whether
stored in the memory chips on the video adapter. In some synthetic or photographic, in files or in memory. When
instances, however, the video chipset is integrated into the coming to .NET, a Bitmap is class used to work with images
motherboard design, and the frame buffer is stored in general defined by pixel data. The library can be used under the name
main memory. The main memory is nothing but RAM present space in C#.NET as using System.Drawing.Bitmap Figure 2.2
inside the computer. Figure 2.1shows how RAM is used as shows a binary bitmapped image. The left half of this diagram
frame buffer. shows the bits in the bitmap, and the right half depicts what
would show on screen.

Figure 2.1: RAM used as frame buffer

Now that we know where the frames are stored, we
need to access it programmatically. Before we access it, we
need to make an event program that can generate events based
on the fact that a frame is ready. If one frame from the camera Figure 2.2: A binary bitmapped image
has arrived, it will be stored in the frame buffer, and from An example of a C# code to load a bitmapped
there, the frame will be rasterized on to the screen. (.bmp) image using the bitmap class is given below.
After the new frame being appraised, it has to be Bitmap image1;
converted to a bitmap image for application of different image private void Button1_Click(System.Object sender,
processing filters. This whole process is termed as Frame System.EventArgs e)
grabbing. A software based frame grabbing is used in this {
project instead of hardware. Frame grabbing can be done by try
using an API called DirectShow, which was developed by {
Microsoft and was included in earlier versions of a package // Retrieve the image.
called DirectX, which is a collection of APIs used for graphics image1 = new Bitmap(@"C:\Documents and Settings\All
and imaging. An API is abbreviated as Application Program Users\"

63
Real-Time Machine Vision Based Robot

+ @"Documents\My Music\sathya.bmp", true); Form1( ) //constructor for the form class

int x, y; Step1: Begin.
// Loop through the images pixels to reset color. Step2: Instantiate object named videocapture for acquiring the
for(x=0; x<image1.Width; x++) frames and assign all the frames from video capture device
{ Step3: Handle Exception by disabling the webcam option in
for(y=0; y<image1.Height; y++) menu if the method videodevice present in the class
{ videocapture throws an exception which means that no
Color pixelColor = image1.GetPixel(x, y); cameras are detected.
Color newColor = Step4: Assign the portname for the serialport
Color.FromArgb(pixelColor.R, 0, 0); Step5: Assign baudrate for the serialport
image1.SetPixel(x, y, newColor); Step6: Disable parity for serialport
} Step7: Assign stopbits as one for serialport
} Step8: Assign databits as 8 for serialport
// Set the PictureBox to display the image. Step9: Disable handshake for serialport
PictureBox1.Image = image1; Webcam Play event() /*Event invoked when user clicks
webcam option */
// Display the pixel format in Label1. Step1: Begin
Label1.Text = "Pixel format: Step2: Stop the other sources to access the buffer
"+image1.PixelFormat.ToString(); Step3: Assign some desired the frame rate
} Step4: Assign some frame size
catch(ArgumentException) Step5: Use the start method in the object so that the capturing
{ starts
MessageBox.Show("There was an error." + Step6: Create object that handles a new frame event and
"Check the path to the image file."); process the frame calling the new frame event handler()
} Step7: End
} New frame event handler()
What is Pseudo-code? Step1: Begin
Pseudo code is a kind of structured English for Step2: Instantiate two bitmap objects naming img1 and img2
describing algorithms. It allows the designer to focus on the Step3: Copy the entire frame on to the two bitmaps img1 and
logic of the algorithm without being distracted by details of img2
language syntax. At the same time, the pseudo code needs to Step4: Create a clone bitmap object of img2 call it img2cl
be complete. It describes the entire logic of the algorithm so /* Image processing starts here */
that implementation becomes a rote mechanical task of /* Gaussian Blur Starts */
translating line by line into source code. Step5: Lock the bitmap memory of both img2 and its clone
In general, the vocabulary used in the pseudo code Step6: Access the memories of both imag2 and its clone using
should be the vocabulary of the problem domain, not of the pointers such that the pointers point to a small portion of the
implementation domain. Pseudo code is not a rigorous memory in the form of a 3x3 matrix
notation and no universal "standard" exists for this Step7: Extract the colour values using pointers and to process
representation. them individually
Pseudo-codes for frame grabbing and Image Processing Step8: Multiply them with corresponding mask coefficients
Before we begin writing the codes, a new project that are stored in the convolution matrix 2 dimensional array
should be created in Visual Studio.NET IDE. Afterwards, Step9: Calculate the sum of all products
include the classes that are used for frame grabbing and image Step10: Divide the sum of products with a variable DIV and
processing, for instance DirectShow. Add the reference to the store in variable pixel
classes that has been added so that the compiler can identify Step11: If pixel > 255 then pixel = 255
trace out the library files. Next, two pictureboxes should to be Step12: If pixel < 0 then pixel = 0
drawn on the form in the form1 designer tab. Then, open Step13: Assign the value of pixel to the corresponding middle
form1.cs file in the editor window and start writing the pixel of the 3x3 matrix of the clone img2cl
required code for frame processing. Step14: Increment the pointer by 3 (Since B, G & R values are
The following are Pseudo-codes and should not be consecutive) and repeat from step 6 to 12 until the last pixel of
treated as actual codes for writing the program in C# in the bitmap
VS.NET editor. /* Gaussian Blur Ends */
Global Declarations /* Colour Filter segmentation starts*/
Declare the integers pixel, DIV=16, sum=0, count=0, pixpos, Step15: Access the starting colour pixel memory location of
centroid, diff the clone img2cl using pointer after it has been processed
Instantiate an object called serialport for RS-232 using Gaussian filter
Initialize the 2-D array convolution matrix = {{1,2,1}, Step16: If value at the address pointed by pointer is lying
//Gaussian between some range of colour values, then assign its value as
{2,4,2}, //Filter 255 else 0
{1,2,1}} //Coefts Step17: Increment pointer and repeat step13 until the last pixel
Initialize the 2-D array dilation matrix = {{0,255,0}, of the clone bitmap
// Structuring element // {255,255,255}, {0,255,0}} /* Colour Filter segmentation ends */

64
 International Journal of Systemics, Cybernetics and Informatics (ISSN 0973-4864)

/* Dilation operation starts */ § Baudmode is variable/constant/expression (0 - 7 on the

Step18: Create a clone bitmap img2cl2 of img2cl BS1, 0 - 65535 on all other BASIC Stamps) that specifies
Step19: Access the starting colour pixel memory locations of serial timing and configuration.
both img2cl and img2cl2 bitmaps using pointers such that a § Qualifier is an optional variable/constant (0 - 255)
3x3 matrix of locations are pointed indicating data that must be received before execution
Step20: Compare the pointed pixel locations of the bitmap can continue. Multiple qualifiers can be indicated with
img2cl corresponding to the elements of the 2-D array commas separating them.
Dilation Matrix. § Plabel is an optional label indicating where the program
Step21: If all are same as that of the structuring element assign should go in the event of a parity error. This argument
the middle pixel of the 3x3 pointer matrix of img2cl2 as 255 should only be provided if Baudmode indicates 7 bits,
else 0 and even parity.
Step22: Increment the pointers and repeat step17 to step20 § Timeout is an optional variable/constant/expression (0 -
until the last pixel location 65535) that tells SERIN how long to wait for incoming
/* Dilation operation ends */ data. If data does not arrive in time, the program will
/* Tracking starts */ jump to the address specified by Tlabel.
step23: Access the starting pixel memory of img2cl2 § Tlabel is an optional label that must be provided along
Step24: If the pixel value is 255 then assign pixpos = pixel with Timeout, indicating where the program should go in
coordinates and increment count the event that data does not arrive within the period
Step25: sum = sum + pixpos specified by Timeout.
Step26: Increment pointer and repeat step22 to step23 until the § InputData is list of variables and formatters that tells
last pixel SERIN what to do with incoming data. SERIN can store
Step27: centroid = sum/count data in a variable or array, interpret numeric text
Step28: Calculate the difference between the center of the (decimal, binary, or hex) and store the corresponding
frame or image and centroid value in a variable, wait for a fixed or variable sequence
diff = center of image - centroid of bytes, or ignore a specified number of bytes. These
Step29: If diff is negative, that is, diff<0 then actions can be combined in any order in the InputData
open serialport for communication list.
write charater “R” One of the most popular forms of communication between
close serialport electronic devices is serial communication. There are two
endif major types of serial communication; asynchronous and
Step30: If diff>0 then synchronous. The SERIN and SEROUT commands are used
open serialport for communication to receive and send asynchronous serial data. SERIN can wait
write charater “L” for, filter and convert incoming data in powerful ways. SERIN
close serialport deserves some lengthy discussion, below, since all this power
endif brings some complexity. The term asynchronous means "no
/* Tracking ends */ clock." More specifically, "asynchronous serial
Step31: Assign picturebox1 image as img1 communication" means data is transmitted and received
Step32: Assign picturebox2 image as img2cl2 without the use of a separate "clock" wire. Data can be sent
Step33: unlock the bitmap memory of img2cl2 using as little as two wires; one for data and one for ground.
Step34: End The PC's serial ports (also called COM ports or RS-232 ports)
use asynchronous serial communication. Note: the other kind
3. Servo Motor Control Using BS2 of serial communication, synchronous, uses at least three
The computer will be sending the control characters wires; one for clock, one for data and one for ground. RS-232
either ‘R’ or ‘L’ through the RS-232 serial communication is the electrical specification for the signals that PC serial
port. The Basic stamp 2 should be ready to receive and read ports use. Unlike normal logic, where a 5 volts is a logic 1 and
the incoming signal and activate appropriate motor either left 0 volts is logic 0, RS-232 uses -12 volts for logic 1 and +12
motor or the right one, depending on the characters ‘L’ or ‘R’. volts for logic 0. This specification allows communication
For this, we use the PBasic command SERIN. The syntax is over longer wire lengths without amplification.
given as below. BASIC Stamp 2 has a line receiver on its SIN pin
Syntax: SERIN Rpin {\Fpin}, Baudmode, {Plabel,} (Rpin = 16). The SIN pin goes to a PC's serial data-out pin on
{Timeout, Tlabel,} [InputData]. Its function is to receive the DB-9 connector built into BASIC Stamp 2 development
asynchronous serial data from RS-232 board. The connector is wired to allow both programming and
§ Rpin is a variable/constant/expression* (0 - 16) that run-time serial communication (unless we are using the Stamp
specifies the I/O pin through which the serial data will be 2 Carrier Board which is only designed for programming). For
received. This pin will be set to input mode. On the BS2, the built-in serial port set the Rpin argument to 16 in the
BS2e, BS2sx, BS2p, BS2pe, and BS2px, if Rpin is set to SERIN command.
16, the BASIC Stamp uses the dedicated serial-input pin BASIC Stamp 2 can also receive RS-232 data
(SIN, physical pin 2), which is normally used by the through any of their I/O pins (Rpin = 0 - 7 for BS1, Rpin = 0 -
Stamp Editor during the download process. 15 on all other BASIC Stamps). The I/O pins don't need a line
§ Fpin is an optional variable/constant/expression (0 - 15) receiver, just a 22 kΩ resistor. The resistor limits current into
that specifies the I/O pin to indicate flow control status the I/O pin's built-in clamping diodes, which keep input
on. This pin will be set to output mode. voltages within a safe range.

65
Real-Time Machine Vision Based Robot

one bit - or slightly more or less than one bit, that depends on
the modulation technique used. So the bit bate (bps) and baud
rate (baud per second) have this connection: bps = baud per
second x the number of bit per baud. The number of bit per
baud is determined by the modulation technique. Here are two
examples: When FSK ("Frequency Shift Keying", a
transmission technique) is used, each baud transmits one bit;
only one change in state is required to send a bit. Thus, the
modem's bps rate is equal to the baud rate: When we use a
baud rate of 2400, we use a modulation technique called phase
modulation that transmits four bits per baud. So: 2400 baud x
4 bits per baud = 9600 bps. Such modems are capable of 9600
Figure 3.1: RS-232 communication socket bps operation.
Table 3.1: Pin description of DB-9 Now that BS2 receives and reads the characters
DB-9 pin Description from the RS-232 port, it has to send PWM signals to the
1 Data Carrier Detect (DCD) corresponding servo motor. For this, we have to use the
PBasic command PULSOUT. The syntax is as given below.
2 Receive Data (RD)
Syntax: PULSOUT Pin, Duration. Its function is to generate a
3 Transmit Data (TD) pulse on Pin with a width of Duration.
4 Data Terminal Ready (DTR) § Pin is a variable/constant/expression (0 - 15) that
Signal Ground (SG) specifies the I/O pin to use. This pin will be set to output
5
mode.
6 Data Set Ready (DSR) § Duration is a variable/constant/expression (0 - 65535)
7 Request To Send (RTS) that specifies the duration of the pulse. The unit of time
8 Clear To Send (CTS) for Duration is described below.
PULSOUT sets Pin to output mode, inverts the state of that
9 No Connection (NC)
pin; waits for the specified Duration; then inverts the state of
the pin again; returning the bit to its original state. The unit of
Figure 3.1 shows the pinouts of the two styles of PC serial Duration is described above. The following example will
ports and how to connect them to the BASIC Stamp's I/O pin generate a 100 µs pulse on I/O pin 5:
(the 22 kΩ resister is not needed if connecting to the SIN pin). PULSOUT 5, 50 ‘generate a pulse on pin 5
Though not normally needed, the figure also shows loop back The polarity of the pulse depends on the state of the pin before
connections that defeat hardware handshaking used by some the command executes. In the example above, if pin 5 was
PC software. It should be noted that PC serial ports are always low, PULSOUT would produce a positive pulse. If the pin was
male connectors. Asynchronous serial communication relies high, PULSOUT would produce a negative pulse.
on precise timing. Both the sender and receiver must be set for If the pin is an input, the output state bit, won't necessarily
identical timing, usually expressed in bits per second (bps) match the state of the pin. What happens then? For example:
called baud. On BS2, SERIN requires a value called Pin 7 is an input and pulled high by a resistor as shown below.
Baudmode that tells it the important characteristics of the Suppose that pin 7 is low when we execute the instruction:
incoming serial data; the bit period, number of data and parity PULSOUT 7, 5 ' generate a pulse on pin 7
bits, and polarity. Figure 3.2 shows the sequence of events on that pin.
Difference between Bit rate and Baud rate Initially, pin 7 is high. Its output driver is turned off (because
The difference between the two is complicated and it is in input mode), so the 10 kΩ resistor sets the state on the
intertwining. They are dependent and inter-related. But the pin. When PULSOUT executes, it turns on the output.
simplest explanation is that a Bit Rate is how many data bits
are transmitted per second. A baud Rate is the measurement of
the number of times per second a signal in a communications
channel changes. Bit rates measure of the number of data bits
(that's 0's and 1's) transmitted in one second in a
communication channel. A figure of 2400 bits per second
means 2400 zeros or ones can be transmitted in one second,
hence the abbreviation "bps". Individual characters (for
Figure 3.2: Sequence of events on pin 7
example letters or numbers) which are also referred to as bytes
For BS2, the unit for pulse duration is 2µs and the
are composed of several bits. A baud rate, by definition,
means the number of times a signal in a communications maximum pulse width is 131.07ms. The entire schematic of
channel changes state or varies. For example, a 2400 baud rate the Servo control is as shown in figure 3.3. The right servo
means that the channel can change states up to 2400 times per motor is connected to PIN 13 and left servo is connected to
second. The term "change state", means that it can change PIN 12 of BS2.
from 0 to 1 or from 1 to 0 up to X (in this case, 2400) times Pseudo-code for servo control
per second. It also refers to the actual state of the connection, The following Pseudo-code is meant for programming in
PBasic programming language.
such as voltage, frequency or phase level). The main
Step1: Start
difference of the two is that one change of state can transmit

66
 International Journal of Systemics, Cybernetics and Informatics (ISSN 0973-4864)

Step2: Declare constants CON T2400 = 396, T9600 = 84,

Inverted = $4000, Baud9600 = T900 + Inverted
Step3: Declare variables VAR command = byte
Step4: Read the RS-232 com port at PIN 16 and store the
ASCII character in the variable command
Step5: If command = “R” then
Send PWM signals to PIN 12 which connects the left servo
motor
endif

Figure 4.1.2: Gaussian Blur Filter applied

to a recorded video stream
Our programs run only on frames stored in the video buffer. In
order to process AVI files, one has to uncompress the AVI file,
sequentially store frames in a buffer, process the buffered
frame, and sequentially display the processed frames on the
monitor.
Input specifications
Type of the video: AVI
Figure 3.3: Schematic of the Servo control Frame Rate: 25
Step6: If command = “L” then Colour Spec: 24-bit RGB
Send PWM signals to PIN 13 which connects the right servo
motor
endif
Step7: Repeat step3 to step5 forever until RST pin goes low
Step8: End

4. Results, Conclusions And Future Perspectives

4.1 Results
The results obtained while operating the robot are
presented in this section in the form of digital images frozen
from video streams. Each video frame is self explicable with
its caption.

Figure 4.1.3: Isolation of object based on colour information

Based on colour information (red), the object is
isolated after low pass filtering the video frames twice and
then thresholding it with a priori colour value (RGB) and
binarization.
Note that the isolated and binarized video frame
image is highly sparsed and disconnected at various places. In
such a situation, it is very difficult to fix the tracking point
(say, centroid) in the image.
A feasible solution to this problem is to dilate the
image using morphological dilation operator with the help of a
suitable structuring element. Bigger the size of the structuring
Figure 4.1.1: Gaussian Blur Filter applied element, better the dilation so that most of the disconnected
to a live video stream regions would get connected. The only disadvantage here is
The video stream is obtained from the web camera that the speed of processing the video would reduce
of the robot. Using Gaussian low pass filter, the image is considerably. Figure 4.1.4 shows the dilated version of the
blurred, which could be clearly seen in the figure 4.1.1. Note image shown in figure 4.1.3.
that the sharp edges are lost in the processed frame. The low Contouring is yet another operation used to solve
pass filter could be applied to recorded video stream too. different pattern recognition problems. Figure 4.1.5 shows a
Figure 4.1.2 shows a sample recorded video stream frame and recorded video frame and its contour map. Laplacian operator
its low pass filtered frame. The sharp edges could be seen to is used to obtain the contour map.
be lost in the processed frame.

67
Real-Time Machine Vision Based Robot

The Laplacian of a 2-D function a[i, j] is a second

order derivative defined as

Figure 4.1.6: Video frame with its edge map obtained

using Sobel operator
Many techniques are being in use to obtain the
centroid of an object image. Iterated thinning is one such tool
by which one can fix up the centroid.
Given an image one can apply thinning algorithm to
Figure 4.1.4: Dilation of disconnected regions it and remove its boundary by one pixel. The thinning process
repeatedly applied to the image till one ends up in a situation
Five Laplacian masks have been shown below for that there is no boundary left. The remaining point is called
ready reference and use. the centroid of the object in an image. Figure 4.1.7 shows a
live video from the web camera where in a colour object is
shown before the camera and the robot identifies its centroid
frame by frame. A sequential identification of centroids leads
to what we call as tracking.

Figure 4.1.7: Cross-hairs tracking the red object (refer to the

‘+’ mark on the red coloured object)
Figure 4.1.5: Video frame with its edge map In what follows, we provide the pictures of the entire real time
obtained using Lapalcian operator machine vision based tracking robot developed as the main
Another way of obtaining edge map is by applying product of this project.
Sobel operator to the video frame image. Sobel operators are
gradient operators. Gradient of a digital image a[i, j] at
location [i,j] is the vector ∇a is the transpose of the row vector
[Gx,Gy] which is the transpose of the row vector
[(∂a/∂x) (∂a/∂y)]. The edge of an image feature corresponds
to the magnitude of this vector which is given by [Gx²-Gy²]½.
Six different masks are shown below for reference and use.

Figure4.1.6 shows a frame of video from web

camera and its edge map obtained using Sobel operator. After
isolating the desired object one has to fix up the track point in
Figure 4.1.8
the image.

68
 International Journal of Systemics, Cybernetics and Informatics (ISSN 0973-4864)

Figure 4.1.9 Figure 4.1.13

Figure 4.1.14
Figure 4.1.10

Figure 4.1.15
Figure 4.1.11 Legend:
Figure Description
8.1.8 Rear view of the robot showing RS-232 connection.
8.1.9 Side view of the robot fitted with a web camera
Top view of the robot showing the BS2
8.1.10
microcontroller and the Board of Education
Bottom view of the robot showing two Parallax
8.1.11
Servo motors attached to the wheels
Robot connected to HP Pavillion DV-5 1106AX
8.1.12
laptop through serial port
8.1.13 Laptop screen shows the object tracking video frame
Red LED on the robot glows indicating that the
8.1.14
object is lost and hence no tracking
Green LED on the robot glows indicating that the
8.1.15
object has been found and the tracking has started
Figure 4.1.12

69
Real-Time Machine Vision Based Robot

4.2 Conclusions for it with an assumption that there will be no technological

To conclude, the output of my project work carried limitations in the future. As machine vision based systems are
out for the past 10 months (May 2008 to March 2009) has quantitative rather than qualitative, they can replace human
been presented in this report under 8 chapters. The first six beings in places where precision and hygiene are top priorities.
chapters deal with the concepts and tools that are required to Such a place can be a semiconductor industry. With high
carry out tracking by an autonomous mobile robot. The resolution cameras, the precision of the machine vision based
seventh chapter provides the state-of-the-art technology systems can be increased. Artificial intelligence which is
developed for this purpose. mostly based on probability and statistics, can give a big boost
Boe-Bot Robot Kit was purchased from its to machine vision systems such as a machine vision based
manufacturer Parallax. This robot consists of Basic Stamp 2 missile. The person who launches the missile can fire and
microcontroller, Board of Education, Robot’s aluminium forget about it.
body, two Parallax servo motors, four 1.5V AA sized battery
and a RS232 serial cable. A web camera was installed in the 5. References
robot for video capturing purposes. The camera was connected Note: Complete references have been given in the previous
to a computer for processing the video frames. paper JAN09-09. For clarifications, please contact Chief
"Vision is the most advanced of our senses, so it is Editor, IJSCI. [www.ijsci.org]
not surprising that images play the single most important role
in human perception". - an excerpt from book of digital image 6. About The Authors
processing by Gonzalez and Woods. So, it can be seen that
vision is the most important sensor which enables us to G. Sathya is the final year B.Tech
interact with the environment better than any other sensory
ECE student of VIF College of
perception. Hence it should make sense why machine vision is Engineering, affiliated to Jawaharlal
the most powerful tool for robot guidance. Robots have Nehru Technological University,
evolved from a simple mechanical clock to complex Hyderabad. His areas of interest
humanoids. But without proper guidance based on sensors, include Signal and Image Processing,
there will be no difference between a robot and a toy. Machine Robotics and Automation, Advanced
vision has proven to be an able guiding method which Communication Systems, Computer
involves various parameters. Animation and Graphics.
Applications of machine vision are many. Some of
useful applications are as given below. He is a Research Engineer in
Ø Machine vision based missile guidance can be a best Pentagram Research Centre Pvt
alternative for radar or laser based guidance. The reason Limited, Hyderabad, India. He is
why machine vision is so powerful in guidance is that it presently pursuing his PhD in image
we have got many characteristics in our hand to work processing in the Jawaharlal Nehru
with. For instance, characteristics such as colour, pattern, Technological University, Kukatpally,
edge, etc, can be used to scrutinize friendly and enemy Hyderabad, India. He had designed
targets, which cannot be done in conventional radar and
and implemented a compiler for Cyber
laser guided missiles.
Motion Technologies, Hyderabad,
Ø In industries where there is a requirement for inspection India and presently working on "Real-time Video Processing
of various things which are routine and which require System" and "Logical Pattern Generation System" as part of
quantitative analysis rather than qualitative. development team in Pentagram Research Centre. His areas of
Ø In security and surveillance where there is a need to interests are image processing, video processing, Pattern
recognize, track, and observe subjects. Recognition and Computer Vision. He presented few papers in
Ø Machine vision based robots can be sent to places where National and International Conferences.
there is a threat for human survival and places where the
robot should completely act on its own as the human
monitoring and control is out of scope. The best example A Humble Dedication
can be space exploration in which there will be a huge This paper and the previous one JAN09-09 are dedicated
communication gap between the machine that has been to our Guru Dr. E. G. Rajan who taught us the
sent to outer space and human beings on earth. fundamentals of Signal and Image Processing and who
4.3 Future perspectives motivated us to look into the possibilities of developing
There is a long way for to go for machine vision programming tools to process 3-D images and digital
based robot to exactly imitate human vision perception. video frames. We thank those peer review committee
Continuing efforts such as this project on machine vision field, members for their very valuable criticism and support in
can possibly accelerate its evolution and can make machine bringing out these two papers to this form. We gratefully
vision robots soon attain capabilities as close as possible to acknowledge the Managing Director of Pentagram
human vision. Although some machine vision based Research Centre, Mrs. G. Chitra, Ms. G. Prashanthi and
automated machine prototypes have been made available, they all Engineers and Scientists working in the company for
are not completely reliable and fault tolerant. Although the giving us the opportunity to carry out our research in the
applications are many, machine vision hasn’t been completely company for almost one year.
realized till date due to many technical limitations. As with G. Sathya, Manish Prateek and G. Ramesh Chandra
many other technologies, there are some future perspectives

70