Recovery of Watermarks from Distorted Images

Neil F. Johnson2, Zoran Duric1 , and Sushil Jajodia2

Center for Secure Information Systems
George Mason University
Fairfax, VA 22030-4444
http://ise.gmu.edu/ csis

fnjohnson,zduric,jajodiag@gmu.edu

Abstract. Many techniques for watermarking of digital images have

appeared in numerous publications. Most of these techniques are sensitive to cropping and/or ane distortions (e.g., rotation and scaling).
In this paper we describe a method for the recovery of original size and
appearance of images based on the concept of identication marks (\ngerprints"); the method does not require the use of the \original" image,
but only a small number of salient image points. We show that, using our
method, it is possible to recover original appearances of distorted images.
The restored image can be used to recover embedded watermarks.

1 Introduction
Interest in digital watermarking techniques is growing, motivated by the need to
provide copyright protection to digital works, such as images, audio, and video,
has increased over the last few years. Much of the interest has been driven by
the growth of the Internet and the development of compression techniques and
compression standards that make possible fast transmission of large volumes of
information. These advances have made it easy to copy almost any song, image,
video, or multimedia object that is available in digital form.
Watermarking can be used to identify owners, license information, or other
information related to the digital object carrying the watermark. Watermarks
may also provide mechanisms for determining if a work has been tampered with
or copied illegally. In the domain of video and satellite broadcasts, watermarks
are used to interfere with recording devices so copies of a broadcast are somewhat
corrupt. Much of the focus has been on digital watermarking of images; this
paper deals with images, although some of the discussion can be equally applied
to other digital works.
Many proposed watermarking techniques are sensitive to image compression
and transformations such as smoothing, rotation, scaling, cropping, and so on, or
even printing and scanning. This interferes with the readability of the watermark.
Publicly available tools can be used to distort images and eectively disable their
1
2

Also with Computer Science Department, George Mason University.

Also with Information and Software Engineering Department, George Mason University.

watermarks. Given the volume of data on the internet, a watermark is eectively

hidden if it cannot be found using a computationally simple and fast procedure.
Attacks on watermarks may be accidental or intentional. Accidental attacks
may be the result of standard image processing or compression procedures. Illicit
attacks may include cryptanalysis, steganalysis, image processing techniques, or
other attempts to overwrite or remove existing watermarks or confuse the reader
as to the authenticity of the watermark [7, 8].
Many owners of watermarked works do not want the watermark to interfere
with the use of the work by others; they therefore require that the watermark
be imperceptible to the human visual system. This requirement works against
the robustness of a watermark. Nevertheless, watermark users usually advertise
the fact that a watermark exists.
In images, various image processing techniques and transformations are commonly employed to develop and apply digital watermarks. These methods can
also be used to attack and disable watermarks. Even with advances in watermarking technology, watermarks may be forged or overwritten; for example, multiple watermarks may be placed in an image and one cannot determine which of
them is valid [1]. Current watermark registration services are \rst come, rst
serve", and someone other than the owner of a digital work may attempt to
register a copyright rst.
Attacks on watermarks may not necessarily remove the watermark, but only
disable its perceptibility. If watermarks are used to locate images, how can an
image be located or the watermark veried after it is disabled? To begin to
understand this issue, we can ask: what features of an image are unaected
by (or invariant to) the processing that disables the watermark? Finding such
features is key to reliably locating an image when an embedded watermark has
been disabled.
We propose alternative methods for image recognition based on the concept
of identication marks (id-marks or \ngerprints"), which can be used to locate
corrupted copies of images [3, 6]. Such a ngerprint can be used as a code to
recognize an image and once an image is found, then the amount of change that
occurred in the process of disabling the watermark can be determined. This
same ngerprint can be used in calculating the inverse transform based on these
changes over the image to recover the original appearance (scale and rotation)
of the image. In doing so, we are also able to recover aspects of the embedded
watermark, as we will see later.
The remainder of this paper is organized as follows: Section 2 provides formal
descriptions of image processing techniques that we use in the recognition and
recovery processes. Section 3 provides a brief description of our image ngerprinting and recognition methods based on invariant image properties. It shows
how images can be prepared before their release to allow their easier recognition.
Section 4 describes our method of recovering image parameters and appearance,
and presents experimental results on images. Conclusions and suggestions about
future research are presented in Section 5.
2

2 Ane Transforms and Displacement Fields

In this section we provide a mathematical background for our work. In Section 2.1
we formally dene ane transforms and give an expression for displacement elds
under ane transforms. In Section 2.2 we give the expression for normalized
cross-correlation and explain its geometric meaning; we use it later in this paper
to establish point correspondences in images. Finally, in Section 2.3 we introduce
normal displacement elds, which we used for ne tuning of image registration
parameters [3].

2.1 Ane Transforms

Let (x; y) be the image coordinates of a pixel in an image I (x; y) and let the
image center be at (0; 0). An ane transform of I (x; y) is given by

 x0   a b  x   e 
+
0 =
y

cd

(1)

where (x0 ; y0 ) are image coordinates in the transformed image I 0 (x0 ; y0 ) and a , f
are the transform parameters.
If we subtract the vector (x y) from both sides of equation (1) we obtain an
expression for the displacement (x; y) of the point (x; y) due to the transform:
T

 x   x0 , x   a , 1 b  x   e   a b  x   e 
 0
=
+

+
:
y ,y

y

c d,1

c d1

(2)

2.2 Normalized Cross-correlation

Let w = I (x + i; y + j ) and w = I (x + i; y + j ), i = ,W; : : : ; W; j =
,W; : : : ; W be two square image windows centered at locations (x ; y ) and
(x ; y ) of images I and I , respectively. The normalized cross-correlation of w
and w is given by
NCC (w ; w ) = k(ww ,, ww )k
k(ww ,, ww k)
(3)
where w and w are treated as vectors. (a
b stands for the inner product of
vectors a and b, a for the mean value of the vector elements and kak for the
2-norm of vector a.) For two windows whose pixel values dier by a scale factor
1

only NCC will be equal to 1; if the windows are dierent NCC has value lower
than 1. For two non-zero binary patterns which dier in all pixels NCC is ,1.
Normalized cross-correlation corresponds to the cosine of the angle between w1
and w2 ; as this angle varies between 0 and 180, the corresponding cosines vary
between 1 and ,1.
3

2.3 Normal Displacement Fields

Let { and | be the unit vectors in the x and y directions, respectively; r =

{x + |y is the projected displacement eld at the point r = x{ + y|. If we
choose a unit direction vector n = n { + n | at the image point r and call it the
normal direction, then the normal displacement eld at r is r = (r
n )n =
(n x + n y)n . n can be chosen in various ways; the usual choice (and the
one that we use) is the direction of the image intensity gradient n = rI=krI k.
Note that the normal displacement eld along an edge is orthogonal to the
edge direction. Thus, if at the time t we observe an edge element at position r,
the apparent position of that edge element at time t +
t will be r +
tr .
This is a consequence of the well known aperture problem. We base our method
of estimating normal displacement eld on this observation.
For an image frame (say collected at time t) we nd edges using an implementation of the Canny edge detector. For each edge element, say at r, we resample
the image locally to obtain a small window with its rows parallel to the image
gradient direction n = rI=krI k. For the next image frame (collected at time
t0 +
t) we create a larger window, typically twice as large as the maximum
expected value of the magnitude of the normal displacement eld. We then slide
the rst (smaller) window along the second (larger) window and compute the
dierence between the image intensities. The zero of the resulting function is at
distance u from the origin of the second window; note that the image gradient
in the second window at the positions close to u must be positive. Our estimate
of the normal displacement eld is then ,u , and we call it the normal ow.
r

3 Fingerprinting Images
The task of recognizing images can be dened as matching invariant features.
These features may be salient parts of images or they may be articial additions to them. In digital watermarking the information is typically embedded
into images to facilitate identication of images. The embedded information is
susceptible to attack through ltering and transformations [7, 8]. To make this
information robust enough it is usually necessary to distort the images to the
point of making the embedded information visible.
Another approach to recognizing images is to use salient features of images
as registration patterns or identication marks. In this way perceptually important image features are used for image identication. Removing these features
is not possible without destroying the image. An alternative is to transform the
image so that it cannot be easily recognized. Such transforms include rotating,
cropping, resampling, etc. Most of these operations can be classied as ane
transforms (see Section 2). They are included in widely available image processing software and are easy to perform.
In this section we introduce our methods for using salient image features in
recognizing images that have been distorted by unknown ane transforms. The
image features used for recognition are groups of points. [lines (edges) are also
considered, but these can be considered as collections of points]. Isolated points
4

are not sucient for image recognition since they are not necessarily invariant,
as diering images may contain similar points. However, groups of points tend
to exhibit uniqueness, For example, ratios of areas enclosed by triples of points
are invariant to ane transforms [3, 6].
We have two basic approaches to our recognition process. One is based on
nding strong, relatively unique points along image edges of high gradient magnitude. The other is based on nding corners with strong gradient magnitude
within the image.
Regardless of which apporach is taken, our image recognition method consists
of two parts. First, for each image we select a set of representative feature points
at multiple resolutions. Second, we use these points for recognizing images. These
methods are described in [3] and [6].

3.1 Selecting Feature Points

In general, our approach is based on nding unique points in each image at multiple resolutions. The points are represented by small rectangular neighborhoods.
We use neighborhood sizes ranging from 3  3 to 11  11 pixels. For each resolution we identify the unique points separately. The selected points usually dier
when resolution changes. In this section, we will summarize our two techniques
for identifying image feature points used to construct an \image ngerprint."
The rst approach we will discuss is based on nding relatively unique points
along edges within an image. The second method is a further renement of the
rst approach by considering corners of edges and less emphasis on uniqueness
of the individual points.

Edge-based Feature Points This method of choosing unique feature points

consists of several steps. First, we compute the image gradient rI over the image.
We identify the image points that have large values of the gradient magnitude
krI k. Note that these points typically correspond to edges (see Figure 1). We
consider all the points that have gradient magnitude larger than one third of the
highest gradient magnitude in the entire image. In doing so we insure that the
second selection step operates on a smaller number of image points.
Second, for each of the selected points (x ; y ) we compute the similarity of
its neighborhood, centered at (p; q), to the neighborhoods of other points in the
image. In the remainder of this paper we will use the term image point to represent the point and its neighborhood. We use the normalized cross-correlation
(see Section 2.2) as the similarity measure. For an image point (p; q) we obtain
the similarity function s (x , p; y , q). This function has a local maximum at
s (0; 0) = 1 since the value at (0; 0) corresponds to the similarity of the point
with itself. If the point is unique, i.e. there are no other points in the image that
are similar to it, s (0; 0) is the global maximum of s as well. If the point is
unique we consider the sharpness of the peak at (0; 0) and the next highest value
of s to decide if the point is a feature point.
Figure 2 shows three examples of feature point selection. The points on the
left and right are good feature points; their similarity functions computed over a
i

p;q

p;q

p;q

p;q

p;q

(a)

(b)

Fig. 1. Identifying likely feature points: (a) an original image and (b) the image points
with large values of gradient magnitude.

6060 pixel window (lower row) have sharp peaks at the center (cross-correlation
with itself), while all other similarity values are below 0:5. The center point is
not a good feature point; it can be seen that its similarity function (the middle
of the lower row of Figure 2) does not have a sharp peak at the center, and there
are multiple other points with similarity values around 0:8 in the 60  60 pixels
window centered at the point.

Corner-based Feature Points We rely on the cornerness measure for salient

point selection. Consider the spatial image gradient [E ; E ] , computed for all
points (x; y) of an image area (neighborhood) A. The matrix M is dened as
x

2
M = EEE EEE2
x

(4)

where the sums are taken over the image neighborhood captures the geometric structure of the gray level pattern of A. M is a symmetric matrix and
can therefore be diagonalized by rotation of the coordinate axes, with no loss of
generality, we can think of M as a diagonal matrix [4]:

M = 01 02
(5)
where 1 and 2 are eigenvalues of M . We can choose 1 as the larger eigenvalue so that 1  2  0. If A contains a corner, then we expect 1  2 > 0,

and the larger the eigenvalues, the stronger (higher contrast) their corresponding
edges. A corner is identied as two strong edges; therefore as 1  2 , a corner
is a location where 2 is suciently large [10].
Given this The method of choosing feature points based on corners consists of
several steps. First, we identify the corner points that have cornerness measure
larger than some threshold. We consider all the points that have cornerness
6

0.8
0.6
0.4
0.5

0.5
0.2
0
0.2

0
0.4
0.6
0.8

0.5

1
0

50

100
0

10

20

30

40

50

60

0.5
0

50

100
0

10

20

30

40

50

60

50

100
0

10

20

30

40

50

60

Fig. 2. Selecting feature points. Upper row: 9  9 neighborhoods of three image points.

Lower row: corresponding similarity functions shown from an 80 viewing angle. The
two outside points are good feature point candidates, while the center point is not.

Fig. 3. Selected feature points (based on edges) for 1=2, 1=4, and 1=8 resolutions; each
point is shown with its 9  9 neighborhood.
7

(a)

(b)

Fig. 4. Identifying likely feature points: (a) an original image. (b) The image points

with large values of gradient magnitude and image points (in red) with strong cornerness measures.

measure larger than one-twentieth of the highest cornerness measure in the entire
image (see Figure 4). Also, values for 2 must be signicantly large and 1 <
2:52 .

Fig. 5. Selected feature corner points for full, 1/2, and 1/4 resolutions; each point is
shown with its neighborhood.

Second, for each of the selected corner points we compute the similarity of its
neighborhood to the neighborhoods of other corner points in the image. In the
remainder of this paper we will use the term image point and feature point to represent the point and its neighborhood. We use the normalized cross-correlation
as the similarity measure (see [3] for details). This process is applied at multiple
resolutions. Figure 5 shows selected feature points at three dierent resolutions.
Notice that the number of feature points goes down as the resolution decreases.
8

Also, image points that are not unique at higher resolutions may become prominent at lower resolutions. This is to counter scaling and blurring so the image
can still be recognized. These sets of feature points provide a compressed representation of the original image and can be use to identify variations of the same
image.
The processes described above for feature point selection are applied at multiple resolutions. Typically, as resolution changes selected feature points also
change. Figure 3 shows selected feature points for the edge-based method at
three dierent resolutions. Figure 5 shows selected feature points for the cornerbased method at three dierent resolutions. Both of these gures illustrate that
the number of feature points goes down rapidly as the resolution decreases. Also,
image points that are not unique at higher resolutions may become prominent
at lower resolutions.

4 Recovering Watermarks
In this section we describe our method of recovering the original size and aspect
of a distorted image given the feature points identied in Section 3.1. Following
the recovery process watermarks that may have been embedded in the image
can be retrieved.

4.1 Estimating Transform Parameters

In this section we describe how the image size and aspect can be recovered by
using the correspondences between image points in the original (I ) and transformed (I 0 ) images.
Let (x ; y ); i = 1; : : : ; N be image points in the image I and let (x0 ; y0 ); i =
1; :::; N be the corresponding points in the image I 0 , respectively. From (1) we
have
 x0   a b  x   e 
(6)
y0 = c d y + f ; i = 1; : : : ; N:
i

We can rewrite equations (6) as

0 x y 1 0 0 01
0 x0 1
BB x y 1 0 0 0 CC 0 a 1 BB x0 CC
BB ...
C
B.C
BB x y 1 0 0 0 CCC BBB eb CCC BBB x..0 CCC
BB 0 0 0 x y 1 CC BB c CC = BB y0 CC :
BB 0 0 0 x y 1 CC B@ d CA BB y0 CC
BB .
CC f BB . CC
@ ..
A
@ .. A
1

0 0 0x y 1
N

Equation (7) can be written as

Au = b
9

(7)

y0

(8)

(a)

(b)

Fig. 6. Recognizing images: (a) A distorted version of the image from Figure 1a. (b)
A cropped version of image (a).

where A, u, and b are dened by comparing equations (7) and (8).

We seek u that minimizes kE k = kb , Auk; the solution satises the system [9]
A Au = A b = d:
(9)
We observe that the problem can be further simplied if we rewrite (8) as
T

 A 0  u   b 
=

(10)
0 A u
b
where A , u , u , b , and b are dened by comparing equations (7) and (10).
1

Equation (9) thus separates into two equations:

A A u =A b ;
T

A A u =A b
T

(11)

We solve these systems using the Cholesky decomposition [9]. Since the matrix
A1 A1 is a positive denite 3  3 matrix there exists a lower triangular matrix L
such that LL = A1 A1 . We solve two triangular systems Le1 = d1 = A1 b1 and
L u1 = e1 for u1 and similarly for u2. Note that we need only one decomposition
for both systems.
The computed u may be inaccurate due to various geometrical and numerical
factors, to be discussed below. Given the estimate u, based on point correspondences between images I and I 0 , we use equation (6) to obtain the inverse ane
transform of I 0 ; we call this corrected image I (1) . [The inversion of (6) is obtained
implicitly. For each pixel position (x; y) of I (1) we compute the pixel position
(x0 ; y0 ) in I 0 (note that x0 and y0 may be non-integers). We obtain the gray level
for (x; y) by interpolating the gray levels of transformed image I 0 .]
Figure 6 shows images derived from the \original" images in Figure 1a and
Figure 4. Figure 6a was created by applying an ane transform (cropping, scaling, and rotating) to the original. Figure 6b was created by cropping the image
in Figure 6a.
T

10

(a)

(b)

Fig. 7. Recovering image size and aspect: (a) The recovered image from Figure 6a. (b)
The recovered image from Figure 6b.

Figure 7 shows results of recovering the size and aspect of a distorted image
using our method for the images in Figure 6. The estimated ane transform
parameters a , f for the image in Figure 6a were (1:082 0:004 , 0:015 1:015 ,
8:51 2:32) ; the corresponding inverse transform applied to the image resulted
in the image shown in Figure 7a. Similarly, the estimated ane transform parameters for the image in Figure 6b were (1:084 0:004 , 0:014 1:015 , 20:85 ,
34:81) ; the corresponding inverse transform applied to the image resulted in
the image shown in Figure 7b. Note that the recovered parameters are very similar for both the uncropped and cropped images. Note also that the embedded
watermark [2] has been successfully detected in the recovered images.
In this section we discussed various factors that may contribute to inaccuracies in estimating u = (a b e c d f ) . However, it is possible to iteratively
improve on the computed solution of the system (11) using the normal ow. The
technique for the renement of image recovery based on normal displacement is
explained in [3]. We show experimental results in Section 4.2.
T

4.2 Additional Experiments

We have experimented with the images shown in previous sections and with the
images shown in Figure 8 using both a commercially available watermarking
tool [2] and the watermarking technique described in [5]. A demo of the commercial watermark is available with Adobe Photoshop. The technique described
in [5] embeds a watermark that corresponds to a logo or a text image into the
original/cover image. We have successfully recovered the watermark in all cases.
In some instances we had to go through the renement phase to recover the
watermark. An example of watermark recovery is shown in Figure 9.
Figure 9 shows an example of a mask-based watermark and recovery after
attack. The image is watermarked using a mask to produce the watermarked
image (see Figure 9a). The watermark is not visible, but the enhanced image
dierence reveals it (see Figure 9b). An attack on the watermark is conducted
11

Fig. 8. Additional images used in our experiments.

12

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 9. Recovery of a commercial watermark [2] for an image in Figure 8. (a) The

watermarked image. (b) Detection of the commercial watermark embedded in (a). (c)
The distorted image. (d) Error message when attempting to read the watermark from
(c). (e) The recovered image. (f) The watermark is once again available and can be
read from (e).

13

by applying Stirmark against the watermarked image (see Figure 9c). Figure 9d
shows the enhanced dierence between the original image and the distorted
image; the watermark is not visible. The ane transformation parameters were
estimated as (1:0255 0:0012 ,0:0049 1:0045 0:9685 1:0939) . The recovered
image is shown in Figure 9e. Finally, Figure 9f shows the enhanced dierence
between the recovered image and the original image, revealing the watermark.
Figure 10 shows the results of rening the size and aspect of a distorted image
using our method. [This method is applied when the distortion of an image is
small; this typically happens after applying the recovery method described in the
previous section.] Figure 10a shows the original image. Figure 10b shows the normal displacement eld between the original image and the distorted image (not
shown). The ane transform parameters estimated from the normal displacement eld using the method described in this section are (0:9993 , 0:0002 ,
0:0029 1:0002 , 0:7565 , 0:842) . Figure 10c shows the recovered image that
was obtained by applying the inverse ane transform to the distorted image.
Finally, Figure 10d shows the normal displacement eld between the images in
Figure 10a and Figure 10c. The ane transform parameters estimated from this
normal displacement eld are (1 , 0:0002 0:0001 1:0001 0:0356 , 0:0206) .
Since the transform is small (the induced normal displacement eld is < 0:5
everywhere) no further renement is needed.
T

5 Conclusion and Future Directions

Digital works are subject to illicit copying and distribution. May owners wish
for some means of of identifying ownership and copyright. Digital watermarks
can fulll this role and provide a means to identify and track digital works.
However, embedded watermarks may fail to be recognized due to accidental
corruption or attack by cropping and/or ane distortions (e.g., rotation, scaling,
and blurring) [7, 8]. This hampers the ability to locate and identify watermarked
images over distributed networks such as the Internet.
In this paper, we introduced alternative methods for image recovery, based
on inherent features within images that can be used to "ngerprint" images.
These identication marks can be applied to locate images and recover image
size and aspect from distorted images. Our methods do not rely on embedded
information and can be used to recover images distorted by various geometric
transformations [3, 6].
We provided examples showing that it is possible to recover the original
appearances of distorted images. In many cases doing so results in the recovery of
embedded watermarks. We continue to investigate ways to make these methods
more ecient for image recognition and recovery.

References
1. Craver, S., Memon, N., Yeo, B., Yeung, N.M., \Resolving Rightful Ownerships
with Invisible Watermarking Techniques: Limitations, Attacks, and Implications,"
IEEE Journal on Selected Areas in Communications, 16(4):573{586, 1998.

14

2. Digimarc Corporation, PictureMarcTM , MarcSpiderTM , http://www.digimarc.com

3. Duric, Z., Johnson, N.F., and Jajodia, S., \Recovering Watermarks from Images,"
submitted to IEEE Transactions on Image Processing, 1999.
4. Jahne, B.: Digital Image Processing, 4th ed. Springer-Verlag, 1997.
5. Johnson, N.F., Duric, Z., and Jajodia, S., \A Role for Digital Watermarking in
Electronic Commerce," ACM Computing Surveys, 1999.
6. Johnson, N.F., Duric, Z., and Jajodia, S., \On \Fingerprinting" Images for Recognition," to appear in Fifth International Workshop on Multimedia Information
Systems (MIS'99), Indian Wells, California, 1999.
7. Johnson, N.F., Jajodia, S., \Steganalysis of Images Created using Current
Steganography Software," in Aucsmith, D., (ed.), Second Information Hiding
Workshop, Portland, Oregon, Lecture Notes in Computer Science, Vol. 1525.
Springer-Verlag, 1998.
8. Petitcolas, F., Anderson, R., Kuhn, M., \Attacks on Copyright Marking Systems,"
in Aucsmith, D., (ed.), Second Information Hiding Workshop, Portland, Oregon,
Lecture Notes in Computer Science, Vol. 1525. Springer-Verlag, 1998.
9. Stewart, G.W. Introduction to Matrix Computations. Academic Press, New York,
1973.
10. Trucco, E. and Verri, A.: Introductory Techniques for 3-D Computer Vision. Prentice Hall, 1998.

15

(a)

(b)

(c)

(d)

Fig. 10. Rening image size and aspect: (a) The original image. (c) The normal dis-

placement between the original and the distorted image (not shown). (c) The recovered
image. (d) The normal displacement eld between images (a) and (c).

16