Chapter 5

Facial Recognition
Why Faces

2
Why Faces?

3
Why Faces?

4
 Facial Features
• This technology includes the use of camera to
acquire the key features of a face and analyze it for
identification and verification.
• Two primary approaches for face recognition are:
– Transform Approach
– Attribute-Based Approach

5
 Facial Features
• Transform Approach
– the universe of face image domain is represented using a
set of orthonormal basis vectors.
– Currently, the most popular basis vectors are eigenfaces.

• Attribute-Based Approach
– facial attributes like nose and eyes are extracted from the
face image and the invariance of geometric properties
among the face landmark features is used for recognition.

6
Face Recognition

7
Face Recognition

8
Face Recognition

9
Acquisition

10
Acquisition

11
Acquisition

12
Acquisition

13
Acquisition

14
Acquisition

15
Enhancement

16
Enhancement

17
 Enhancement
Face Detection Methods

Either based on:

• Sliding windows
• Regions of interest.

18
Enhancement

19
Enhancement

20
Enhancement

21
Enhancement

22
 Face Detection and Location
• Face detection and location is an important stage in face recognition.
• There are two main approaches used to implement this function namely,
statistics-based and knowledge-based.
• Statistical based method
– Subspace method – find the subspace of face images which show common
features of faces, which is a good representation of the face
– Neural Network method – two classification ML technique, that is face/non-face.
There is need to train with face and non-face images. The main challenge is that
there are many kinds of non-face images which are not collected, and its slow.
• Knowledge based method
– Distribution ruler of gray-value-based
– Contour ruler
– Colour information – detects faces with use of color information of face, since
color of faces are different from background color in an image
– Movement information
– Symmetry information
23
 Face Detection: Viola-Jones
Detector
• First real-time face detector.
• Based on sliding windows.
• 4 Key Ideas
– Haar-like features.
– Integral image.
– Boosting for feature selection.
– Attentional Cascade to reject non-faces.

24
 Face Detection: Viola-Jones
Detector
Haar-Like Features (1/4)
• Binary rectangle filters
– used to extract features from the sliding window.

25
 Face Detection: Viola-Jones
Detector
Integral Image (2/4)
• Solution to apply Haar-like features fast.
• Precomputed data structure with the same
dimensions of the target image.

26
 Face Detection: Viola-Jones
Detector
Integral Image (2/4)
• Remember Haar feature value:

• Integral images allow the

computation of the sum of pixel
values in any target area in constant
time, regardless of the size of the
area.

27
 Face Detection: Viola-Jones
Detector
Boosting for Feature Selection (3/4)
• Goal:
– Select combinations of Haar-like features that are
useful for face detection.
• Solution:
– boosting, a combination of weak classifiers that
when learned in sequence and applied together, lead
to better final classification.

28
 Face Detection: Viola-Jones
Detector
Boosting for Feature Selection (3/4)

29
 Face Detection: Viola-Jones
Detector
Boosting for Feature Selection (3/4)
• Possible outcome.
• This combination is enough to
lead to perfect True Positive Rate,
but poor False Positive Rate.
• All faces are detected as positive,
but many non-faces are detected
as positive too

30
 Face Detection: Viola-Jones
Detector
Attentional Cascade (4/4)
• Make a cascade of different classifiers that
are good at rejecting faces.
• Start with simpler and faster classifiers.

31
 Face Detection Attacks

• Non-live faces and some special patterns may be

used to trigger the face detector on purpose.
• If it happens too often, it will flood the system.

32
 Face Detection Attacks

• Make-up can be used to hinder detection.

33
 Face Detection Attacks

• Make-up can be used to hinder detection.

34
 Face Enhancement: Alignment
Goal
• Make template and sample faces be in similar
poses, to make further description and
matching easier.

35
 Face Enhancement: Alignment
1. Detection of Face Landmarks E.g., position of eyes.
– There are better solutions in the literature, like using deep neural
networks.

2. Landmark Alignment
– E.g., make the positions of the eyes horizontally aligned, by rotating
the face image.

36
 Face Enhancement: Alignment
3. Make a tight crop of the face, to remove background.
– Keep eyes, nose, and mouth.

37
 Face Enhancement: Alignment
More Severe Pose Variations
• Naïve approach will not work

• Alternative approaches, like 3D information will help to do

frontalization.

38
 Face Enhancement: Illumination
Correction
• Simplest Solution
– Color histogram equalization.
• Alternatives
– Photometric normalization, illumination modeling,
etc.

39
 Feature Extraction
• Focus
– 2D-appearance-based methods.
• Types
– Handcrafted features from Computer Vision.
– Data-driven learned features from Machine
Learning.

40
 Feature Extraction: Handcrafted Features
• An expert designs what and how facial regions
should be used.

41
Feature Extraction: Handcrafted Features

• Local Binary Patterns (LBP) – to describe face texture

42
Feature Extraction: Handcrafted Features

43
Feature Extraction: Handcrafted Features

44
Feature Extraction: Handcrafted Features

45
Feature Extraction: Handcrafted Features

46
Feature Extraction: Handcrafted Features

47
Feature Extraction: Handcrafted Features

48
Feature Extraction: Handcrafted Features

49
Feature Extraction: Handcrafted Features

50
Feature Extraction

51
Feature Extraction: Handcrafted Features

52
Feature Extraction: Handcrafted Features

53
Feature Extraction: Handcrafted Features

54
Feature Extraction: Handcrafted Features

• Local Binary Patterns

55
Feature Extraction: Handcrafted Features

56
Feature Extraction: Handcrafted Features

57
Feature Extraction: Handcrafted Features

58
 Feature Extraction: Data-driven Learned
Features

• From pixels to
classification decision.
• Hierarchy of feature
extractors.
• Each layer extracts
features from previous
layer.

59
 Feature Extraction: Data-driven Learned
Features

Data Driven Face

Recognition

• Convolutional Layers -
E.g., layers 1 and 2.
• Feature extractors are
convolutional operations
which are performed on
the output of the
previous layer.
60
 Feature Extraction: Data-driven Learned
Features

Data Driven Face

Recognition

• Fully Connected Layer

E.g., layer 3.
• It performs the
classification, presenting
one score output for each
class (identity, in the case
of Biometrics).
61
Feature Extraction: Data-driven Learned
Features

62
Feature Extraction: Data-driven Learned
Features

63
Feature Extraction: Data-driven Learned
Features

64
Feature Extraction: Data-driven Learned
Features

65
Feature Extraction: Data-driven Learned
Features

66
Feature Extraction: Data-driven Learned
Features

67
Feature Extraction: Data-driven Learned
Features
Interesting Variations

• Remove fully
connected layer
• Use last
convolutional
layers as a feature
descriptor.

68
Feature Extraction: Data-driven Learned
Features

Interesting Variations

• Train the network in a

way that feature
vectors of the same
class have small
distance, while
• feature vectors from
different classes have
large distance.
69
Feature Extraction: Data-driven Learned
Features

70
Feature Extraction: Data-driven Learned
Features

71
Feature Extraction: Data-driven Learned
Features

72
 Weaknesses of Facial Features
• The images taken for templates can be hugely affected by the
surrounding factors, such as lighting whilst capturing an image,
the pose and orientation of face, hair, any facial expression,
illumination
• The human face changes with time.
• Invention and development of more complex algorithms can
successfully masquerade human faces, such as an algorithm
developed by a researcher at the University of Ottawa.
– It is an exploit algorithm which can regenerate a face from a high quality
image of a person from the templates being used.
73
 Facial Thermography
• Facial thermography uses an infrared camera which
successfully captures the heat patterns emission
which is generated by the vascular system of the
face.
• These heat patterns (also known as aura) are the
heat which passes through the facial tissues of an
individual and are unique to every human being.

74
 Facial Thermography
• It is possible to convert heat
patterns to digital data and store
in the database as a template and
use as comparison trait for
authentication and verification.
• Benefit of facial thermography
over facial geometry is that the
auras do not change over time.
75