Image Processing

COMP4421
Hao CHEN, 陳浩
Dept. of CSE and CBE
HKUST
Image Segmentation
• Goal: identify groups of pixels that go together
Image Segmentation
A process that partitions R into subregions R1, R2 ,..., Rn

Microsoft multi-class segmentation dataset

Image Segmentation Applications

Autonomous driving Medical image analysis

He et al., Mask R-CNN, ICCV, 2017.
4
Ronneberger et al., U-Net: Convolutional Networks for Biomedical Image Segmentation, MICCAI, 2015.
Image Segmentation
• Fundamentals

• Point, Line, and Edge Detection

• Thresholding

• Region Segmentation

5
Image Segmentation
• Fundamentals

• Point, Line, and Edge Detection

• Thresholding

• Region Segmentation

6
Fundamentals
Preceding conditions to satisfy for image segmentation:

𝑅: entire spatial region of image; 𝑅1 , 𝑅2 ,…, 𝑅𝑛 ,: subregions;

𝑄(𝑅𝑘 ): logical predicate defined over the points in set 𝑅𝑘 .
Symbol that represents a property or a relation

7
Fundamentals
• Segmentation algorithms for monochrome images generally are
based on one of two basic categories dealing with properties of
intensity values:
1. Discontinuity: edge-based algorithm
2. Similarity: region-based algorithm

8
 Fundamentals
Edge detection algorithm
(gradient) a b c
(a) Image of a constant intensity d e f
region.
(b) Boundary based on intensity
discontinuities.
(c) Result of segmentation. Edge based

(d) Image of a texture region.

(e) Result of intensity discontinuity Region based

computations.
(f) Result of segmentation based on
region properties.
For each pixel, calculate value that’s based on Q Calculate the standard deviation
(predicate) => ∞
(>0: white)

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 9

Image Segmentation
• Fundamentals

• Point, Line, and Edge Detection

• Thresholding

• Region Segmentation

10
Point, Line, and Edge Detection
Image characteristics with sharp, local changes in intensity:
• Edge pixels: pixels at which the intensity of an image changes abruptly.
• Edges: sets of connected edge pixels.
• Line: thin edge segment in which the intensity of the background on either side of
the line is either much higher or much lower than the intensity of the line pixels.
• Isolated point: a foreground (background) pixel surrounded by background
(foreground) pixels.

11
Background
Finite differences: derivatives of a digital function.

How to get first-order derivative of a digital image?

Approximate with Taylor series.

12
Background
Set ∆x=1, then

Retain only linear terms, we get forward difference:

13
Background
Set ∆x=-1, then

Retain only linear terms, we get backward difference:

14
Background
Subtracting backward difference from forward difference, we can get central
difference with first-order:

Which is more accurate? => Central

Has lower error than the forward and backward difference.

15
Background
Higher order derivatives based on a central difference:
Second:

Third:

Fourth:

16
 Background
First four central digital derivatives (finite differences)

For two variables, we apply the derivative to each variable independently, e.g.,

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 17

 Background
a b
c

(a) Image.

(b) Horizontal intensity profile that

includes the isolated point
indicated by the arrow.

(c) Subsampled profile.

Forward difference
Central difference
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 18
 Background
Observations: a b
c
(1) First-order derivatives generally
produce thicker edges.
(2) Second-order derivatives have a stronger
response to fine detail, such as thin lines,
isolated points, and noise.
(3) Second-order derivatives produce a double-
edge response at ramp and step transitions in
intensity.
(4) The sign of the second derivative can be used
to determine whether a transition into an edge
is from light to dark or dark to light.

Forward difference
Central difference
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 19
 Background
Computing the first and second derivatives at
every pixel using spatial convolution:

A general 3x3 spatial filter kernel,

with 9 kernel coefficients.

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 20

 Detection of Isolated Points

Laplacian:

Second-order finite difference:

Laplacian kernel (an

extension including diagonal
terms) for isolated point
Implementation: detection.

21
 Detection of Isolated Points
a b c

(a) X-ray image of a turbine blade

with a porosity manifested by a
single black pixel.

(b) Result of convolving the

Laplacian kernel with the image.

(c) Result of thresholding.

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 22

 Line Detection using Laplacian
a b
c d
(a) Original image of a wire-bond mask.

(b) Laplacian image after scaling

(magnified part shows positive/negative
double-line effect).

(c) Absolute value of the Laplacian.

(d) Positive value of the Laplacian.

Insensitive to directions!

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 23

 111
000

Line Detection with Specific Directions 111

Detects vertical line X

Line detection kernels with different directions. The preferred direction of each kernel is
weighted with a larger coefficient (i.e., 2) than other possible directions.
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 24
 Line Detection with Specific Directions
a b c
d e f
(a) Image of a wire-bond mask.
(b) Result of processing with +45°
line detection kernel.
(c) Zoomed view of top-left region
of (b).
(d) Zoomed view of bottom-right
region of (b).
(e) Negative values in (b) set to zero.
(f) The points in (e) that are larger
than T=254.

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. Ringe???? also detected 25
Edge Models
Three types:

Step Ramp Roof edge

26
 Edge Models

Step

Three types of edges in

one image.

Ramp Roof edge

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 27

 Edge Models—An Ideal Ramp Case
Derivatives – Ideal Case
a b

(a) Two regions of constant

intensity separated by an
ideal ramp edge.

(b) Detail near the edge,

showing a horizontal
intensity profile, and its
first and second derivatives.

28
 But, in practice …. always need to do smoothing

A ramp edge corrupted by additive Gaussian noise with zero mean

and standard deviation σ.
Original First-order Second-order Original First-order Second-order

σ=1
σ=0

σ=10
σ=0.1

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 29

Edge Detection in Practice

Thus, three steps are performed typically for edge detection:

(1) Image smoothing for noise reduction;
(2) Detection of edge points;
(3) Edge localization. Second order diff zero crossing

30
Basic Edge Detection

Gradient:
(Vector image)

Magnitude:
(Gradient image)

Direction of gradient:

31
 Basic Edge Detection

Prewitt Operator used

(Right - Left) Magnitude: Direction:
Gradient:

The direction of an edge is orthogonal to the direction of the gradient vector at

the point. The gradient vector sometimes is called the edge normal.
32
 Basic Edge Detection
Forward difference:

1-D kernel for implementing forward difference.

For diagonal edge direction:

2-D kernel for implementing 3x3 region.

diagonal edge.
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 33
 Basic Edge Detection
Prewitt Operator:

Sobel Operator:

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 34

 Basic Edge Detection
a b
(a) An image with intensity values c d
scaled to range [0, 1].

(b) 𝑔𝑥 , the component of the

gradient in x-direction, obtained
using Sobel kernel.

(c) 𝑔𝑦 , the component of the

gradient in y-direction, obtained
using Sobel kernel.

(d) The gradient image, 𝑔𝑥 + 𝑔𝑦 .

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. Gradient magnitude: strong response along horizontal and
35 vertical
 Basic Edge Detection
Areas of constant intensity in this
image indicate that the direction
of the gradient vector is the same
at all the pixel locations in those
regions.

Gradient angle image.

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 36

 Basic Edge Detection
a b
Same sequence as before but c d
add 5x5 averaging kernel prior
to edge detection for noise
reduction.

Thicker edge
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 37
 Combining Thresholding
Edge detection + Thresholding Smoothing + Edge detection + Thresholding

thicker eage
eage
broken :

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 38

 The Canny Edge Detector
An advanced edge detector with three basic objectives:
fake

(1) Low error rate. All edges should be found, and there should be no spurious responses.
(2) Edge points should be well localized. The edges located must be as close as possible to
the true edges.
(3) Single edge point response. The detector should return only one point for each true
edge point. That is, the number of local maxima around the true edge should be minimum.

Canny edge detector: expressing the three criteria mathematically, then find
optimal solution.

39
 The Canny Edge Detector

Gaussian smoothing

Gradient magnitude

Angle

40
 The Canny Edge Detector
a b
Nonmaxima Suppression to thin ridges. c

(a) Two possible orientations of a horizontal

edge in a 3x3 neighborhood.

(b) Range of values (shaded) of 𝛼, the

direction angle of the edge normal for a
horizontal edge.

(c) The angle ranges of the edge normals for

the four types of edge directions in a 3×3
neighborhood. Each edge direction has two
ranges, shown in corresponding shades.
41
 The
The Canny
Canny Edge
Edge Detector
Detector
Nonmaxima Suppression to thin ridges.
Let 𝑑1 , 𝑑2 , 𝑑3 and 𝑑4 denote the four basic edge directions for a 3×3 region: horizontal, −45°,
vertical, and +45°, respectively. Nonmaxima suppression for a 3×3 region centered at an arbitrary
point (x, y) in 𝛼:
1. Find the direction 𝑑𝑘 that is closest to 𝛼(x, y).
2. Let K denote the value of ||∇𝑓𝑠 || at (x, y). If K is less than the value of ||∇𝑓𝑠 || at one or both of the
neighbors of point (x, y) along 𝑑𝑘 , let 𝑔𝑁 𝑥, 𝑦 = 0 (suppression); otherwise, let 𝑔𝑁 𝑥, 𝑦 = K.
When repeated for all values of x and y, this procedure yields a nonmaxima suppressed image
𝑔𝑁 𝑥, 𝑦 .
Suppress to thin the edges

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 42

 The
The Canny
Canny Edge
Edge Detector
Detector
Hysteresis (double) thresholding to reduce false edge points: using two thresholds, a low
threshold 𝑇𝐿 and a high threshold 𝑇𝐻 (with ratio in the range of 1:3 to 1:2 usually).
First, Strong edge pixels
G_N: nonmaxima
suppressed image

Then, Weak edge pixels

(a) Locate the next unvisited edge pixel, p, in 𝑔𝑁𝐻 (𝑥, 𝑦).
(b) Mark as valid edge pixels all the weak pixels in 𝑔𝑁𝐿 (𝑥, 𝑦) that are connected to
Repeat, p using, say, 8-connectivity. P is from strong edge pixels
(c) If all nonzero pixels in 𝑔𝑁𝐻 𝑥, 𝑦 have been visited go to step (d). Else, return to
step (a).
(d) Set to zero all pixels in 𝑔𝑁𝐿 (𝑥, 𝑦) that were not marked as valid edge pixels.

Finally, append all the nonzero pixels from 𝑔𝑁𝐿 (𝑥, 𝑦) to 𝑔𝑁𝐻 (𝑥, 𝑦).
43
 The Canny Edge Detector

To summarize, the Canny edge detection algorithm consists of following steps:

(1) Smooth the input image with a Gaussian filter.
(2) Compute the gradient magnitude and angle images.
(3) Apply nonmaxima suppression to the gradient magnitude image.
(4) Use double thresholding and connectivity analysis to detect and link edges.

44
 The Canny Edge Detector
a
b c

(a) An image with intensity values

scaled to range [0, 1].

(b) Thresholded gradient of the

smoothed image.

(c) Image obtained using the Canny

algorithm.

45
 The Canny Edge Detector
a
b c

(a) A head CT image with intensity

values scaled to range [0, 1].

(b) Thresholded gradient of the

smoothed image.

(c) Image obtained using the Canny

algorithm.

The edge broken !

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 46

Image Segmentation
• Fundamentals

• Point, Line, and Edge Detection

• Thresholding

• Region Segmentation

47
Thresholding
a b
Intensity histograms that
can be partitioned (a) by a
single threshold, and (b) by
dual thresholds.

Segment with
thresholding:

• Global thresholding: T is a constant applicable over an entire image.

• Multiple thresholding: More than one types of object to segment.
• Variable thresholding: The value of T changes over an image.
• Dynamic thresholding (special case of variable thresholding): T depends on the spatial
coordinates (x,y).
48
Thresholding

Observation: the success of intensity thresholding is related directly to

the width and depth of the valleys separating the histogram modes.

Key factors affecting the properties of valleys:

• the separation between peaks.
• the noise content in the image.
• the relative sizes of objects and background.
• the uniformity of the illumination source.
• the uniformity of the reflectance properties of the image.

49
 Thresholding
a b c
The Effect of Noise d e f

(a) Noiseless 8-bit image.

(b) Image with additive Gaussian

noise of mean 0 and standard
deviation of 10 intensity levels.

(c) Image with additive Gaussian

noise of mean 0 and standard
deviation of 50 intensity levels.

(d)-(f) Corresponding histograms.

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 50

 Thresholding
a b c
The Role of Illumination and Reflectance d e f

(a)Noisy image.

(b) Intensity ramp in the

range [0.2, 0.6].

(c) Product of (a) and (b).

(d)-(f) Corresponding
histograms.

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 51

Basic Global Thresholding
How to estimate the threshold T ?

• Select an initial estimate for the global threshold, T.

• Segment the image using T by producing two groups of pixels.
• Compute the mean of these two groups of pixels, say 𝑚1 and 𝑚2 .
• Compute a new threshold value midway between 𝑚1 and 𝑚2 :
1
𝑇 = (𝑚1 + 𝑚2 )
2
• Repeat step 2 through step 4 until the difference between values of T in
successive iterations is smaller than a predefined value.

52
Basic Global Thresholding
1. Select an initial estimate for the global
threshold, T.
2. Segment the image using T by producing two
groups of pixels
3. Compute the mean of these two groups of
pixels, say m1 and m2.
4. Update the threshold T=(m1 +m2)/2
5. Repeat Steps 2 through 4 until
convergence

m1 T m2
53
Basic Global Thresholding
1. Select an initial estimate for the global
threshold, T.
2. Segment the image using T by producing two
groups of pixels
3. Compute the mean of these two groups of
pixels, say m1 and m2.
4. Update the threshold T=(m1 +m2)/2
5. Repeat Steps 2 through 4 until
convergence

m1 T m2
54
 Basic Global Thresholding

Noisy fingerprint Histogram Segmented result using

global threshold T=125.
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 55
Otsu’s Algorithm

56
Optimum Global Thresholding using Otsu’s Method

Philosophy: Maximize between-class variance

Some variables:
{0,1,2,…,L-1}: set of L distinct integer intensity levels
M x N: image size 𝑛𝑖 : number of pixels with intensity i

MN= 𝑛0 + 𝑛1 + 𝑛2 + ⋯ + 𝑛𝐿−1 𝑝𝑖 = 𝑛𝑖 /𝑀𝑁 (Normalized histogram)

𝐿−1

෍ 𝑝𝑖 = 1 𝑝𝑖 ≥ 0
𝑖=0

57
Optimum Global Thresholding using Otsu’s Method
Suppose we threshold an image into two classes, 𝑐1 and 𝑐2 with threshold
[0, k] [k+1, L-1]
• Then, probability of a pixel assigned to class 𝑐1 occurring:

• Probability of of a pixel assigned to class 𝑐2 occurring:

• Compute mean Bayes’ rule

intensity value of
pixels in 𝑐1 :

• Compute mean
intensity value of
pixels in 𝑐2 : 58
Optimum Global Thresholding using Otsu’s Method
cumulative mean up global mean:
to level k:
From preceding and
results, we have: _

점
= I -
P
Pr
,

( 앞장 ,
• Calculate global m
: 금
m .

variance: 증명 직접 해볼 것

• Calculate between-class
variance:
'

Reintroduce k

59
Optimum Global Thresholding using Otsu’s Method
A normalized measure of
separability at level k: Constant

So we optimize which means

How to obtain 𝑘 ∗ ?
Just evaluate all integer values of k (subject to condition 0 < 𝑃1 𝑘 < 1).

Finally, we get the mask:

60
Optimum Global Thresholding using Otsu’s Method
To summarize:

• Compute the normalized histogram of the input image. Denote the components
of the histogram by 𝑝𝑖 .
• Compute the cumulative sums, 𝑝1 (𝑘).
The farther apart the means the
• Compute the cumulative means, 𝑚(𝑘). 𝟐
• Compute the global mean, 𝑚𝐺 . larger will be 𝝈𝑩 (𝒌)

• Compute the between-class variance term, 𝜎𝐵2 (𝑘).

• Obtain the Otsu threshold, 𝑘 ∗ , as the value of k for which 𝜎𝐵2 (𝑘) is maximum. If
the maximum is not unique, obtain 𝑘 ∗ by averaging the values of k corresponding
to the various maxima detected.
• Compute the global variance, 𝜎𝐺2 , and then obtain the separability measure, 𝜂 ∗ .
This is a measure of how easily separable the classes are.
61
 Optimum Global Thresholding using Otsu’s Method
a b
c d

(a) An optical microscope image of

polymersome cells.
(b) Histogram.
(c) Segmentation result using the
basic global algorithm (T=169).
(d) Result using Otsu’s method
(T=182, 𝜂 ∗ =0.467).

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 62

 Using Image Smoothing to Improve Global Thresholding
a b c
d e f
(a) A noisy image.
(b) Corresponding
histogram.
(c) Result obtained using
Otsu’s method.
(d) Smoothed using 5x5
averaging kernel.
(e) Corresponding
histogram.
(f) Result of thresholding
using Otsu’s method.

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 63

 Relative sizes of objects and background ..

Using Image Smoothing to Improve Global Thresholding

Between Class Variance - brighter class small area: a b c
Effect on small foreground region: fail! Why? it’s contribution to histogram insignificant.
d e f
(a) A noisy image.
(b) Corresponding
histogram.
(c) Result obtained using
Otsu’s method.
(d) Smoothed using 5x5
averaging kernel.
(e) Corresponding
histogram.
(f) Result of thresholding
using Otsu’s method.

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 64

Multiple Thresholds
Extending two classes to three classes
Between-class variance:

where and

So we optimize

Finally, the mask: Separability

measure:
65
Multiple Thresholds
K classes

Between-class variance:

where and

To obtain the K-1 thresholds that separate K classes:

66
 Multiple Thresholds

Image segmented into

Image of an iceberg. Histogram.
three regions using
dual Otsu thresholds.

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 67

 Variable Thresholding
A basic approach to variable thresholding is to compute a threshold at every
point in the image based on one or more specified properties in a neighborhood
of (x, y):

Two categories of variable thresholding:

• Local Image Properties
• Moving Averages
68
 Variable Thresholding
Local Image Properties based

A general form:
Parameters computed using
A predicate
pixels in neighborhood 𝑆𝑥𝑦

An example based on local mean and standard deviation:

69
 Variable Thresholding
a b
Local Image Properties based c d

(a) Image of yeast cells.

(b) Image segmented using the dual
thresholding approach.
(c) Image of local standard deviations
with neighborhood size 3x3.
(d) Result obtained using local
thresholding. Std and mean

70
Variable Thresholding
Moving Averages based
Purpose: For fast processing during scanning by computing moving
averages along scan lines.
Moving average:

• n is number of points used in computing average

• 𝑧𝑘+1 is the intensity of point encountered in the scanning sequence
at step k+1
• 𝑚 1 = 𝑧1
71
Variable Thresholding
Moving Averages based

An example:

where c is positive scalar, 𝑚𝑥𝑦 is the moving average at point (x, y).

72
 Variable Thresholding
Moving Averages based
Correct the shading pattern

Text image Result of global Result of local

corrupted by spot thresholding using thresholding using
shading. Otsu’s method. moving averages. 73
 Variable Thresholding
Moving Averages based

Text image Result of global Result of local

corrupted by thresholding using thresholding using
sinusoidal shading. Otsu’s method. moving averages. 74
Image Segmentation
• Fundamentals

• Point, Line, and Edge Detection

• Thresholding

• Region Segmentation

75
Region Segmentation by Region
Growing

76
 Region Segmentation by Region Growing
Segmentation by Region Growing: group pixels or subregions into larger regions based
on predefined criteria for growth. 𝑓(𝑥, 𝑦): input image
S(𝑥, 𝑦): a seed array containing 1’s at the locations of
A basic pipeline based on 8-connectivity: seed points and 0’s elsewhere 𝑄 : a predicate

1. Find all connected components in 𝑆(𝑥, 𝑦) and reduce each connected

component to one pixel; label all such pixels found as 1. All other pixels in S are
labeled 0.
2. Form an image 𝑓𝑄 such that, at each point (𝑥, 𝑦) , 𝑓𝑄 𝑥, 𝑦 = 1 if the input image
satisfies a given predicate, Q, at those coordinates, and 𝑓𝑄 𝑥, 𝑦 = 0 otherwise.
3. Let g be an image formed by appending to each seed point in S all the 1-valued
points in 𝑓𝑄 that are 8-connected to that seed point.
4. Label each connected component in g with a different region label. This is the
segmented image obtained by region growing. 77
 Region Segmentation by Region Growing
a b c
Segmentation by Region
d e f
Growing g h i
(a) X-ray image of a defective weld.
(b) Histogram.
(c) Initial seed image.
(d) Final seed image.
(e) Absolute value of the difference between
the seed value (255) and (a).
(f) Histogram of (e).
(g) Difference image thresholded using dual
thresholds.
(h) Difference image thresholded with the
smallest of the dual thresholds.
(i) Segmentation result obtained by region
growing.

78
Region Segmentation by Region
Splitting and Merging

79
 Region Segmentation by Splitting and Merging
Segmentation by Region Splitting and Merging

The pipeline:
1. Split into four disjoint quadrants any
region 𝑅𝑖 for which 𝑄(𝑅𝑖 ) = FALSE.

2. When no further splitting is possible,

merge any adjacent regions 𝑅𝑗 and 𝑅𝑘
for which 𝑄(𝑅𝑗 ∪ 𝑅𝑖 )=TRUE.

3. Stop when no further merging is (a) (b)

possible. Partitioned image. Corresponding quadtree.
R represents the entire image region.

80
 Region Segmentation by Splitting and Merging
a b
Segmentation by Region Splitting and c d
Merging

(a) Image of the Cygnus Loop supernova,

taken in the X-ray band by NASA’s
Hubble Telescope.
(b) Results of limiting the smallest allowed
quadregion to be 32x32 pixels.
(c) Results of limiting the smallest allowed
quadregion to be 16x16 pixels.
(d) Results of limiting the smallest allowed
quadregion to be 8x8 pixels.

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 81

Region Segmentation by K-means
Clustering

82
 Region Segmentation by K-means Clustering
Segmentation using K-Means Clustering

Basic idea: To iteratively partition a set, Q, of observations into k clusters, and each
observation assigned to the cluster with the nearest mean (also called prototype).
Is there optimal solution? X
we can’t solve with linear time

Objective: It’s NP-hard!

where z is a vector observation (sample), 𝑚𝑖 is mean vector (or centroid), and 𝐶𝑖 is a cluster.

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 83

K-means Clustering: Step 1

Initialize the k-center

randomly

84
K-means Clustering: Step 2

Assign the data to

the nearest center
to form a cluster

85
K-means Clustering: Step 3

Re‐estimate the 𝑘
cluster centers, by
assuming the
memberships found
in the previous step
are correct.

86
K-means Clustering: Step 4

Assign the data to

the nearest center.

87
K-means Clustering: Step 5
Divide and Conquer

Re‐estimate the 𝑘
cluster centers, by
assuming the
memberships found
in the previous step
are correct.

88
K-means Clustering: Step …

Iterate between
step 4 and step 5
until convergence.

89
 Region Segmentation by K-means Clustering
Segmentation using K-Means Clustering: A heuristic algorithm

How to determine the K?

• prior knowledge about K good

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 90

 Region Segmentation by K-means Clustering
Segmentation using K-Means Clustering a b

(a) Original image.

(b) Image segmented using

the k-means algorithm with
k=3.

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 91

Region Segmentation Using
Graph Cuts

92
Region Segmentation Using Graph Cuts
Segmentation using Graph Cuts
Basic idea: Pixels as nodes of a graph, then find an optimum partition (cut) of the
graph, where members within a group have high similarities and the ones across
groups have low similarities.

Images denoted as graphs: 𝐺 = (𝑉, 𝐸)

where 𝐺 is a graph, 𝑉 is a set of nodes, and 𝐸 ⊆ 𝑉 × 𝑉 is a set of edges (undirected).
Symmetric bc it is undirected.

To characterize undirected edges, we use 𝑾, which is a symmetric matrix (i.e.,

𝑤 𝑖, 𝑗 = 𝑤(𝑗, 𝑖)).
Could be distance, intensity value, …

𝑤 𝑖, 𝑗 : similarity measure between 𝑉𝑖 and 𝑉𝑗 .

93
Region Segmentation Using Graph Cuts
Segmentation using Graph Cuts
Set 𝑉 can be partitioned into subsets by cutting the graph 𝐺.

A cut of a graph is a partition of 𝑉 into two subsets A and B by removing the edges
connecting subgraphs A and B, where

Two key aspects of using graph cuts for image segmentation:

(1) How to associate a graph with an image.
(2) how to cut the graph in a way that makes sense in terms of partitioning the image
into background and foreground (object) pixels.

94
 Region Segmentation Using Graph Cuts
Segmentation using Graph Cuts: A simple example

The weights 𝑾 for the edges usually are formed from

spatial relationships and intensity measures, consistent
with the similarities between pixels.

For example,
𝑤 𝑖, 𝑗 = 1/(|𝐼 𝑛𝑖 − 𝐼(𝑛𝑗 )| + 𝑐),
where 𝐼 𝑛𝑖 and 𝐼(𝑛𝑗 ) are intensities of nodes i and j,
respectively, and c is a constant to avoid division by zero.
Thus, the closer the intensities, the higher the similarities.

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 95

 Region Segmentation Using Graph Cuts
a d
Segmentation using Graph b c
Cuts: A simple example

(a) A 3x3 image.

(b) A corresponding graph.

(c) Graph cut.

(d) Segmented image.

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 96

 Region Segmentation Using Graph Cuts
Segmentation using Graph Cuts: Minimum graph cuts
After constructing a graph, the next step is to cut the graph into two
or more subgraphs.

Minimum cut: The optimum partition of a graph is the one that minimizes this
cut value.

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 97

 Region Segmentation Using Graph Cuts
Images can be viewed as graphs

Nodes: pixels

Edges: Constraints between

neighboring pixels
 Region Segmentation Using Graph Cuts
Graph-view of intelligent scissors:

1 2 1 1. Assign weights (costs) to edges

2 1 1 9
5 4 4

3 3 3 6
1 1 1
 Region Segmentation Using Graph Cuts
Graph-view of intelligent scissors:

1 2 1 1. Assign weights (costs) to edges

Start
2. Select the seed nodes
2 1 1 9
5 4 4

3 3 3 6
1 1 1
End
 Region Segmentation Using Graph Cuts
Graph-view of intelligent scissors:
1. Assign weights (costs) to edges
1 2 1
Start
2. Select the seed nodes

2 1 1 9 3. Find shortest path between

5 4 4 them

3 3 3 6
1 1 1
End
 Region Segmentation Using Graph Cuts
Graph-view of intelligent scissors:
1. Assign weights (costs) to edges
1 2 1
Start
2. Select the seed nodes

2 1 1 9 3. Find shortest path between

5 4 4 them
What algorithm can we use to find
the shortest path?
3 3 3 6
1 1 1
End
 Region Segmentation Using Graph Cuts
Graph-view of intelligent scissors:
1. Assign weights (costs) to edges
1 2 1
Start
2. Select the seed nodes

2 1 1 9 3. Find shortest path between

5 4 4 them
What algorithm can we use to find
the shortest path?
3 3 3 6 • Dijkstra’s algorithm (dynamic
1 1 1 programming)
End
 Region Segmentation Using Graph Cuts
Segmentation using Graph Cuts:
Minimum graph cuts

The problem of min-cut: favor

cutting small sets of isolated
nodes in a graph, leading to
improper segmentations.

Here, the similarity is defined

as the spatial proximity.
An example showing how a min cut can lead to a
meaningless segmentation.
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 104
 Region Segmentation Using Graph Cuts
Segmentation using Graph Cuts: Minimum graph cuts
A solution for min-cut issue
Normalized cut (Ncut): work with a measure of “disassociation” that computes
the cost as a fraction of the total edge connections to all nodes in the graph.

where

Assoc(A, V) is the sum of the weights of all the edges from the nodes of
subgraph A to the nodes of the entire graph, and similarly for Assoc(B, V).

Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 105

 Region Segmentation Using Graph Cuts
Segmentation using Graph Cuts: Minimum graph cuts

Ncut(A, B) will no longer have small values for the cut that partitions
isolated points.

Image segmentation using graph cuts is to find a partition that minimizes

Ncut(A, B). However, this is an NP-complete computational task.

An approximate discrete solution minimizing Ncut(A,B) by formulating it as a

generalized eigenvalue problem, for which numerous implementations exist
(see Shi and Malik [2000]).

Shi J, Malik J. Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence. 2000 Aug;22(8):888-905. 106
Region Segmentation Using Graph Cuts
Specifying weights for graph cut segmentation
As we mentioned, intensity values or distances can be used to specify
edge weights. A more practical approach is to use both:

where 𝑛𝑖 and 𝑛𝑗 denote two nodes, 𝐼 𝑛𝑖 is the intensity of 𝑛𝑖 , 𝜎𝐼2 and 𝜎𝑑2
are constants determining the spread of the two Gaussian-like functions,
𝑑𝑖𝑠𝑡(𝑛𝑖 , 𝑛𝑗 ) is the distance between the two nodes, and r is a radical
constant that establishes how far away we are willing to consider similarity.
107
Region Segmentation Using Graph Cuts
Segmentation using Graph Cuts Actually min cut

Original image. Image smoothed with a Graph cut segmentation obtained

25x25 box kernel. by specifying two regions.
108
 Advanced Approaches for Image Segmentation

• General-purpose image segmentation is far from being well

solved using traditional image processing techniques.
• It is still a research problem that is being investigated by many
researchers.
• Image segmentation with Graphic Models (MRF, CRF, etc.).
• Semantic Segmentation with Deep Learning like Segment
Anything Model (SAM).
Kirillov, Alexander, et al. Segment anything. arXiv:2304.02643 (2023).
Summary
• Fundamentals

• Point, Line, and Edge Detection

• Thresholding

• Region Segmentation

110