1 Sidewalk Direction Recognition and

Multiple Color Sidewalk Area Recognition

Sidewalk Direction Recognition and

Multiple Color Sidewalk Area Detection
Junseong Kim Dongmin Kim
codertimo@gmail.com dmk9259@gmail.com

Korea Digital Media High School

Abstract Second, sidewalk direction recognition algorithm is the most

important part of our app. The purpose of this algorithm is
recognize the direction of sidewalk(e.g right or left corner,
Sidewalk area detection and direction recognition are rounded or curved sidewalk). There are some obstacle
important technologies in autonomous robot. In this paper, recognition and avoidance approaches for autonomous
We propose an algorithm which can recognize the area of robot, however there was no research for recognize the
the multiple color and pattern sidewalk in single RGB image sidewalk direction from front real-time video. So we need to
frame. Also we propose two algorithms which can recognize develop a new approach to this problem. There are lots of
the direction of sidewalk. Each results of two algorithms are input variables, edges which formed from Canny Edge
used in cross-evaluation system to get more accurate result. Detector, lines which formed from Hough Line Detector,
As a result, we could get 92% of accuracy. This algorithms corner point which formed from Harris Corner Detector and
are less accurate then Neural Network approach, however it Vanishing point etc. Among those variables, we decide to
is fast enough to use in real-time mobile devices. Using this use line for detect the tendency of degree, and finally
algorithm, we could develop real-time computer vision recognize the direction.
based walking assistance application for visually impaired.
Direction information is important for visually impaired,
1. Introduction however if we gave them a wrong information, we can put
them into a dangerous situation. To prevent this situation,
we decide to use cross-evaluate system with our algorithms
As the technology has been developed each day, convenient
result using two different algorithms which have the same
life for ours is also required. In our society, there are some
purpose(recognize sidewalk direction). Line’s degree
obstacles that impaired person have to go through in living
dependency based recognition and classified line’s graph
in dense urban. Walking in sidewalk, as a pedestrian, it’s so
pattern based recognition are those algorithms.
hard for them to show the way without navigator to help
recognizing sidewalk’s condition. For this problem, there are
Braille Blocks on road, but as you know, the installation of 2. Related Work
braille block are very low. So they need helper who can help
their walking. To solve this problem we want to develop The purpose and the method of our sidewalk area detection
some assistance application for them with the function of algorithm are similar with [1]Sidewalk following using color
giving sidewalk environment information(direction, obstacle histograms approach. However previous approach just only
etc.). To analyze the environment of sidewalk, we use input can detect the single color and simple patterns sidewalk.
image from camera, and analyze it with our computer vision
algorithm which proposed in our paper, Sidewalk Area and
Direction Recognition algorithm.

GPS Navigator is the best way to recognize the environment

of sidewalk, however GPS Navigator doesn't work in indoor.
We also wanted to assist visually impaired in indoor. That’s
the reason why we choose computer vision technique to
recognize the sidewalk environment.

Sidewalk area recognition algorithm make a base line data Figure 1. The Result of [1] sidewalk following using color histograms’s
for sidewalk direction recognition algorithm. In previous approach in multiple color sidewalk.
work, some papers made some approaches to this problem
but it could only worked on single color based sidewalk. It As you see Figure 1, it cannot fully detect the sidewalk area,
means that various pattern, multiple color and noise are not and Figure 1’s situation is common in South Korea and other
allow for previous work. In the case of South Korea, most of countries. This is our first problem to solve in this paper.
sidewalks are installed with multiple color and pattern. So
we need to solve this problem to use our app in various of And Sidewalk Direction recognition approach in real-time
environment and countries. front video doesn’t have related work. 

2 Sidewalk Direction Recognition and
Multiple Color Sidewalk Area Recognition

3. Sidewalk Area Detection Algorithm the ROI sector. Using the ROI, we can extract the samples of
the sidewalk colors and make target color for color
Sidewalk area detection is one of the most important detection.
technique in autonomous robots. So many researcher
struggle in this problem using Color-Detection, Neural
Network, etc. However, Supervised Neural Network
methods need lots of making datas which contain the area
of sidewalk to train their model. So it is not easy thing to do.
Also Neural Network methods are not enough fast to use in
real-time mobile devices. We decide to make an useful color
detection algorithm

Figure 5. Extract Result of ROI(Region Of Interest)’s Main Color Pattern

Figure 5. shows the two main colors, but it alway’s change in

the various situation. Usually the sidewalk is consisted with
1~6 colors. So we apply K-mean Clustering when it initialize,
and evaluate suitability to determine the K value. So even if
the colors are various, we could recognize the size of main
main color.

Figure 2. Process of Sidewalk Area Detection Algorithm

Figure 6: Compare the previous work and our algorithms result.

After we extract the main color form ROI, we could detect

Figure 3. Right Image show the result of Average Filter the area of sidewalk using And-OR calculation. Compare the
extracted main color and whole clustered pixel of image.
To reduce noises, we applied Averaging Filter. Figure 3 show Above image shows the result of our algorithm. As you can
you the differences between used or not. Second, we apply see, previous problem (single color only detectable) could
the K-Mean Clustering in Input Image. The reason why we be solved using our algorithm. Finally we could detect the
apply K-Mean Clustering is to group the similar color. Under total area of sidewalk very successfully.
Equation shows the equation of K-Mean Clustering.

Simple explain about the K-Mean Clustering, they made the

K Group and made cost function with “how much closer
with center point of each Group” and the final goal is
minimize the cost function. Handling three dimension matrix
(RGB Image) is more complicate to express in this paper, so
we will just comment under paper [2]Clustering and the Figure 7. It could successfully detect the whole sidewalk area even if has a
continuous k-means algorithm. multiple color. And second image use the Average Filter.

As you can see, even if the sidewalk has a multiple color or

pattern, our algorithm can detect the full area of sidewalk.
Using this algorithm, we could detect the vanishing point of
sidewalk more accurately(Vanishing point is just a sample,
not a important in after paper context).

However, there is some critical point of this algorithm. If

user is not located in the sidewalk, than ROI sector will
extract the wrong sample. It means that our algorithm only
Figure 4. RGB Images K-Mean Clustering Result. In First Image, the pink box can detect the area when the ROI is on the sidewalk area.
is the ROI (Rigon Of Interest) sector .
This is our homework, which has to make sidewalk
Next, we will extract the sidewalk’s color pattern in detection algorithm without ROI sector. And I think it could
ROI(Region of Interest) . In Figure 4, the pink rectangle is be possible with using Neural Networks method.

3 Sidewalk Direction Recognition and
Multiple Color Sidewalk Area Recognition

4-1 Sidewalk Direction Recognition calculate variance of matrix fill with data of loss functions. If
algorithm’s result are correct, its mount should be aroused.
Approach with tendency of lines
Also, we adopt linear regression method. We get implicit
derivatives of function by weighting parameter that adjusts
length factor to get minimum variance of loss function’s
result. Parameters are easily changed to improve its quality.
We coordinate the parameter to suit with input data.

We set the P as average of matrix A. P is related to

parameter and represent average of data in matrix A. So,
function S is shown at this form of by calculating deviation.

Figure 8. Process of Sidewalk Direction Recognition : Approach with tendency

of lines.
Now, we get implicit derivative of S by parameter. And
Sidewalk Area Detection’s result is our input data. First of
variables x, y, X, Y are shown as following equations.
all, we applied Canny Detection to find contours of the
images. Second, we used Probabilistic Hough Transform to
get lines of the images. Figure 9 shows preprocessing result.

Equation (7) present implicit derivative of loss function. To

get critical value of it, we can set parameter as related to
Figure 9. The Result of Canny Detector and Hough Line Detector. input data only. And if needed, we can change parameter to
get more quality. After calculating it, we can finally set the
First, we suggest that sidewalk’s border be refer with input loss function.
image’s tendency. As pedestrian is walking along road, the
view of road must have some tendency. And it is likely to Now, we should define function D to get result of S. While D
match with sidewalk’s border. Second, result lines should be get two function as variable, we can assume that two line’s
long and exclusive for each other. If slope of two lines are average angle is close to pedestrian’s direction and result of
two close, it means that there are no considerable linear regression method. So we define function D as this
possibilities of becoming border as two lines are form.
approximately same. Third, to become meaningful input,
line’s size should be big. If ones size is too small, it can be
actually some noise or another object’s line. So, we applied
some linear regression method that contains line’s slope and
length as variable.
In short, we search the slope of all lines by using linear
regression method. By adjusting ours, we can develop useful
loss function between two lines. This function’s parameter
are needed to be set before the function is called. But, the
parameters are calculated by linear regression method.
Although There are some necessary to be changed by
image’s width and height, we resize it’s size. So, in our
method weighting parameter beta is calculating like this.

Figure 10. Pair of line which was extracted from cost function.

The result of algorithm is pair of line. So, we need to convert

line’s reliability not only of them, but with some other line.
To give them more accuracy, we set some adopted
parameter and evaluated them. Now we set the loss
In the usual circumstances, we get the parameter beta
function of two line. In constructing loss function, we
between 0.015 and 0.020. After applying method that
4 Sidewalk Direction Recognition and
Multiple Color Sidewalk Area Recognition

calculate line’s reliability with tendency of image, we use 4-2 Sidewalk Direction Recognition
linear regression to reduce some trivial errors by simple
vibration of user. We use OLS(Ordinary Least Square) Approach with graph pattern
method and some RANSAC(Random Sample Consensus) to
get line’s tendency. In this process, we can get our direction Second method for recognizing direction is approach with
that can be assumed by our previous path model and know graph pattern. In this method, we will classify lines
whether our path is fit in sidewalk. according to slope of lines and watch the graph which
shows the changes of classified lines. Figure 13 shows the
process of our approach. Using all lines tendency and graph
pattern recognition are the differences between first
method and this one. Two method’s cross evaluation helps
to make more accurate classification.

Figure 11. Pair of lines which extracted from regression method in matrix
which accumulated pairs of lines.

The algorithm’s time complexity is mostly influenced by

situation which affect linear vector’s size. And it is estimated
to square of n by big O notation. By using some regression
method, we just obtain it’s time complexity mostly
manufactured by which regression models to take. If we
choose Theil-Sen Estimator or Gradient Descendant and so
on that refers to more complicated regression model, we
have to give up getting real-time processing. Thus, we Figure 13. The process of sidewalk direction recognition : approach with
graph pattern
choose just simple and strong model that compute our
algorithm.
First of all, we classified the lines according to slop. Input
lines are extracted from Hough Line Detector. And
classification condition followed the Figure 14 condition.

Figure 12. Left image shows the tendency of sidewalk direction which
extracted from final result of our algorithm Figure 14. Line Classification Condition

In real-time detection, there can be some unknown Figure 15. images shows the classification result. Green lines
situation that damage our result’s reliability. So, we finally mean right and left sidewalk border lines. And purple lines
use some linear regression method in combination of mean the direction vector. Lastly, white lines means horizon
direction and confidence of each result. Adjusting this lines. We used Polar System to express lines data. Polar
method, we assure that result show the direction of frame. System contains (r,theta) values not (x1,y1,x2,y2). So we can
There are some point that accuracy and performance time classify lines more easily.
of algorithm needed to be compromised. Especially in
realtime processing, performance time must be reduced to
get meaningful result. So, there are many ways to deal with
such that problems. But, we suggested that it is better to
use simple algorithm and integrate with regression or some
method is more efficient than using complex algorithm
structure to define each frame. It is more economical and
change it’s elapsed-time more easily and suitable.

Figure 15. Left image shows the hough line detection result, Right image
shows the classification result.

After classification proecces, we made some graph which

express the changes of each category. This graph
accumulate each classified lines size for 30 frames to see
the pattern. And we walked various of sidewalk
environment especially in corner situation.
5 Sidewalk Direction Recognition and
Multiple Color Sidewalk Area Recognition

So we made simple formula to describe our hypothesis. And

we applied these formulas in another test videos. Result of
the test was successful. Figure 19 shows the result of the test

Figure 16. The graph shows the changes which normal sidewalk move to right
vertical corner situation.

The images of Figure 16 show the normal and right vertical

corner and graph pattern with each direction situation. As
you can see there are some differences in graph. If the
normal sidewalk are change to the right vertical corner,
horizon lines(black lines) size was rapidly increased in graph Figure 19.The result of algorithm which applied the hypothesis. The direction
which was recognized from our algorithm are printed on the bottom of the
image.

5. Test
If the results of ‘approach with tendency of lines’ and
‘approach with graph pattern’ are same, we make a direction
detected event. To get accuracy of our algorithm, we used
specific estimate system. First, we gave the realtime video
and print the result of sidewalk direction recognition. Second,
Figure 17. After Figure 16 situation, right curved corner situation happened. we compare the result of algorithm with the real direction
And Right Direction Vectors are rapidly increased. Figure 20 shows our the evaluate system.
When we turned right to corner, you could meet the right
curved sidewalk. Then, the size of right direction vector are
rapidly increased and the size of horizon line are decreased.
Using this basics which was formed from experience, we can
make hypothesis. In vertical corner situation, horizon lines
are increase rapidly and the same direction vector are
increase also. And if curved corner situation, direction
vector increase rapidly and be a biggest in graph. This is just
hypothesis which constructed by experience. It could be
wrong. So we should proof our hypothesis with some
experiment.

Figure 20. Algorithm evaluate system. System shows the result of algorithm
and subject input the real direction.

As a result, we got 92% of accuracy in this test. We used 32

test videos, and 40 different circumstances of sidewalk.
Approach with line tendency got 83% of accuracy and
approach with graph pattern got 90% of accuracy. This result
are full enough to make the walking assistance application for
visually impaired which is our main goal.

Figure 18. Simple formula of hypothesis which can recognize the sidewalk
graph pattern
6 Sidewalk Direction Recognition and
Multiple Color Sidewalk Area Recognition

6. Conclusion
We made sidewalk area detection algorithm and sidewalk
direction recognition algorithm. Sidewalk area detection
algorithm solve the problem which cannot detect the full area
of the sidewalk which has multiple color. And we could made
the sidewalk direction recognition algorithm for the first time.

Especially, sidewalk direction recognition algorithm used two

different algorithm. If both of results are same, we made an
event that the direction information has detected.

Using evaluation system, we could get the accuracies of our

algorithms. As a result, our algorithm to 92% of accuracy .

We applied these algorithms in walking assistance application

for visually impaired. Also we hope that our research can help
the field of autonomous robot
7 Sidewalk Direction Recognition and
Multiple Color Sidewalk Area Recognition

Reference

[SENG08] Seng, John S., and Thomas J. Norrie. "Sidewalk

following using color histograms." Journal of Computing
Sciences in Colleges 23.6 (2008): 172-180.

[FABER94]Faber, Vance. "Clustering and the continuous k-

means algorithm." Los Alamos Science 22.138144.21 (1994).

[HE04] HE, YINGHUA AND WANG, HONG AND ZHANG, BO:

Color-based road detection in urban traffic scenes In Intelli-
gent Transportation Systems, IEEE Transactions on (2004),
IEEE, pp. 309–318.

[SENG08] SENG, JOHN S AND NORRIE, THOMAS J: Sidewalk

following using color histograms In Journal of Computing
Sci- ences in Colleges(2008), Consortium for Computing
Sciences in Colleges, pp. 172–180.

[CANNY86] CANNY, JOHN: A computational approach to

edge detection In Pattern Analysis and Machine Intelligence,
IEEE Transactions on(1986), IEEE, pp. 679–698.

[BAO05] BAO, PAUL AND ZHANG, LEI AND WU, XIAOLIN:

Canny edge detection enhancement by scale multiplication
In Pattern Analysis and Machine Intelligence, IEEE
Transactions on(2005), IEEE, pp. 1485–1490.

[KIRIYATI91] KIRYATI, NAHUM AND ELDAR, YUVAL AND

BRUCKSTEIN, ALFRED M: A probabilistic Hough transform In
Pattern recognition(1991), Elsevier, pp. 303–316.

[BARINOVA12] BARINOVA, OLGA AND LEMPITSKY, VICTOR AND

KHOLI, PUSHMEET: On detection of multiple object in-
stances using hough transforms In Pattern Analysis and
Machine Intelligence, IEEE Transactions on(2012), IEEE, pp.
1773–1784.

[SEBER12] SEBER, GEORGE AF AND LEE, ALAN J: Linear

regression analysis In Linear regression analysis(2012), John
Wiley & Sons.