# Robust Analytics for Video-Based Gait Biometrics

Ebenezer R.H.P. Isaac  Ebenezer R. H. P. Isaac

(February 2018)

myheadings

## Glossary

1.5in

ROBUST ANALYTICS FOR VIDEO-BASED GAIT BIOMETRICS
A THESIS
Submitted by
EBENEZER RAJIV HINSTON PRABHU I
in partial fulfillment of the requirements for the degree of
DOCTOR OF PHILOSOPHY

FACULTY OF INFORMATION AND
COMMUNICATION ENGINEERING
ANNA UNIVERSITY
CHENNAI 600 025
FEBRUARY 2018

## ANNA UNIVERSITY
CHENNAI 600 025
CERTIFICATE

The research work embodied in the present thesis entitled “ROBUST ANALYTICS
FOR VIDEO-BASED GAIT BIOMETRICS” has been carried out in the Department of
Computer Science and Engineering, Anna University, C.E.G. Campus, Chennai. The
work reported herein is original and does not form part of any other thesis or
dissertation on the basis of which a degree or award was conferred on an
earlier occasion or to any other scholar.

I understand the University’s policy on plagiarism and declare that the thesis
and publications are my own work, except where specifically acknowledged and
has not been copied from other sources or been previously submitted for award
or assessment.

K. S. EASWARAKUMAR

RESEARCH SCHOLAR

SUPERVISOR

Professor

Department of Computer

Science and Engineering

Anna University

Chennai 600025.

## ABSTRACT

Gait analysis is the study of the systematic methods that assess and quantify
animal locomotion. The research on gait analysis has considerably evolved
through time. It was an ancient art, and it still finds its application today
in modern science and medicine. Gait finds a unique importance among the many
state-of-the-art biometric systems since it does not require the subject’s
cooperation to the extent required by other modalities. Hence by nature, it is
an unobtrusive biometric. Gait is associated with three types of signals,
viz., kinetic, kinematic and EMG. Kinesiological EMG is mostly used for
clinical purposes and kinetic measurement instruments are confined to a
limited space. The applicability of kinetic and EMG data for the purpose of
biometrics is thus constrained. Kinematic observation of gait is much more
efficient and successful in literature. Also, the cost of kinematic
measurement instruments such as a video camera and mobile accelerometer are
much more cost effective than the apparatus required for EMG and kinetic
observation.

There are two aspects to biometrics: hard and soft. Soft biometrics include
determination of height, weight, gender, or ethnicity with gender recognition
being the most common form associated with gait research. The method proposed
in this thesis, Pose-Based Voting employs a scheme which delineates the gait
instance as a sequence of poses which is used to predict the gender of the
respective subject through a voting scheme.

Hard biometrics associate people with their innate traits that identifies
them. This data can be used either for identification or for authentication.
Gait identification, widely known as gait recognition, is the process of
mapping a given gait instance to a trained identity. On the other hand, gait
authentication shows how closely the given gait instance matches to identity
claimed. Recent advancements show that the more favourable biometric accuracy
is attained through the use of gait templates. A gait template is a collation
of silhouettes extracted from a gait video sequence. Many templates are
proposed till date aimed to decrease the error of gait recognition. However,
they still suffer from the effect of covariate factors such as variation in
clothing and carrying conditions. Masking the templates to include only the
covariate-resilient regions improve the overall performance of the recognition
system. The proposed framework, the Genetic Template Segmentation (GTS),
automates the process of fitting the masking boundaries to select the regions
that correspond to the optimum performance.

The gait templates are also used for authentication that follows a similar
procedure for feature extraction. The test instance and the stored gallery
instance is separated by a distance in a hyperplane. In traditional gait
authentication, if this Euclidean distance falls within a permissible
threshold, then the test sequence is said to be authorized as claimed
identity. However, Euclidean threshold-based methods trade off a set amount of
False Accept Rate (FAR) for an acceptable False Reject Rate (FRR). This thesis
proposes an intuitive technique called Multiperson Signature Mapping (MSM) for
gait authentication, which converts a gait recognition system to a powerful
authentication system. The salient feature of this method allows to decrease
the FAR in proportion to the system population while still attaining an FRR
equal to the error rate of the base recognition system.

Though the MSM framework enhances the performance of any existing gait
authentication method, when the system population decreases, its greatest
strength becomes its weakness. To circumvent this problem another framework,
the Bayesian Thresholding, is designed. This method measures the distance
between the test and gallery instances in the form of posterior probabilities.
This drastically decreases the overall error and can boost the power of any
authentication system in general.

## ACKNOWLEDGEMENT

I wish to record my deep sense of gratitude and profound thanks to my research
supervisor Dr. K. S. Easwarakumar for his well-organized approach and
inspiring guidance which helped me bring this thesis to fruition. I also thank
Dr. D. Manjula, Professor and Head, Department of Computer Science and
Engineering, Anna University, CEG Campus for the amenities provided to carry
out my research. Further, I express my gratidude to my Doctoral Committee
members for their valuable suggestions during the course of my research work.

I like to convey my sincere thanks to my mentors Dr. Susan Elias, Associate
Professor, School of Electronics Engineering, VIT University, Chennai Campus,
and Dr. Srinivasan Rajagopalan, Department of Physiology and Biomedical
Engineering, Mayo Clinic, Rochester, Minnesota, USA, for providing a great
support by spending their priceless time for the research discussions with me.

I like to extend my thanks to the faculty and non-teaching staff members of
the Department of Computer Science and Engineering, Anna University, CEG
Campus, for their constant support during the whole of my time in the
institution.

I am extremely indebted to my parents for all their help and for their
boundless encouragement and sacrifice throughout the course of my education.

EBENEZER R. H. P. ISAAC

chapter15 mm10 mm section25 mm10 mm subsection35 mm12 mm subsubsection49 mm15
mm paragraph40 mm20 mm subparagraph50 mm 20 mm table10 mm10 mm figure10 mm10
mm

CHAPTER NO. TITLE PAGE NO.

######  TABLE OF CONTENTS

  1. ABSTRACT
  2. ACKNOWLEDGEMENT
  3. LIST OF SYMBOLS AND ABBREVIATIONS
    1. 1 INTRODUCTION
      1. 1 Taxonomy of Gait Analysis
      2. 2 Historical Advents
        1. 2.1 Inception of Gait Analysis
        2. 2.2 Clinical Relevance
        3. 2.3 Interest in Biometrics
      3. 3 Nomenclature
        1. 3.1 Phases of Gait
        2. 3.2 Gait Parameters
        3. 3.3 Reference Planes
        4. 3.4 Performance Metrics
    2. 4 Research Directions
      1. 4.1 Active Areas
      2. 4.2 Least Explored
    3. 5 Contributions
    4. 6 Organization of the Thesis
  4. 2 LITERATURE SURVEY
    1. 7 Approaches for Gait Biometrics
      1. 7.1 Inertial Sensors
      2. 7.2 Footstep Analysis
      3. 7.3 Acoustic Data
      4. 7.4 WiFi Signals
      5. 7.5 Computer Vision
        1. 8 Soft Biometrics
          1. 8.1 Age Estimation
          2. 8.2 Gender Classification
            1. 9 Hard Biometrics
              1. 9.1 Model-free Techniques and Early Approaches
              2. 9.2 Model-free Template-based Methods
                1. 9.3 Model-free Non-template Methods
                  1. 9.4 Model-based Techniques
                    1. 9.5 Covariate Factors
                      1. 9.6 View Invariance
                        1. 10 Datasets
                          1. 11 Summary
                            1. 3 GENDER CLASSIFICATION THROUGH POSE-BASED VOTING
                              1. 12 Introduction
                              2. 13 State of the Art
                              3. 14 Method
                                1. 14.1 Elliptic Fourier Descriptors
                                2. 14.2 Row-Column Summation
                                3. 14.3 Pose-Based Voting
                              4. 15 Experimentation & Evaluation
                                1. 15.1 Dataset Configuration
                                2. 15.2 Number of Harmonics for EFD
                                3. 15.3 Performance in Partial Gait Cycles
                                4. 15.4 Overall Performance
                              5. 16 Summary
                                1. 4 GAIT RECOGNITION THROUGH GENETIC TEMPLATE SEGMENTATION
                                  1. 17 Introduction
                                  2. 18 State of the Art
                                  3. 19 Method
                                    1. 19.1 Gait Template Extraction
                                    2. 19.2 Learning Model
                                    3. 19.3 Genetic Template Segmentation
                                    4. 19.4 View Estimation
                                  4. 20 Experimentation & Evaluation
                                    1. 20.1 Dataset Configuration
                                    2. 20.2 Sagittal View Performance
                                    3. 20.3 View-Invariant Performance
                                  5. 21 Summary
                                    1. 5 GAIT AUTHENTICATION USING MULTIPERSON SIGNATURE MAPPING
                                      1. 22 Introduction
                                      2. 23 State of the Art
                                      3. 24 Method
                                        1. 24.1 Spatiotemporal Feature Extraction
                                        2. 24.2 Multiperson Signature Mapping
                                        3. 24.3 Two-pass Variation of MSM for GTS
                                      4. 25 Results & Discussion
                                        1. 25.1 Dataset Configuration
                                        2. 25.2 Threshold Method vs. MSM
                                        3. 25.3 Types of FAR
                                        4. 25.4 Covariate Factors and GTS-2P
                                        5. 25.5 Spoofing Attacks and Limitations
                                      5. 26 Summary
                                        1. 6 GAIT AUTHENTICATION USING BAYESIAN THRESHOLDING
                                          1. 27 Introduction
                                          2. 28 Mapping of Instances to Regions
                                          3. 29 Method
                                            1. 29.1 Gait Feature Extraction
                                            2. 29.2 Bayesian Thresholding
                                            3. 29.3 Two-pass Variation of BT for GTS
                                          4. 30 Results & Evaluation
                                            1. 30.1 Dataset Configuration
                                            2. 30.2 Performance Comparison with NN Threshold
                                            3. 30.3 Performance Comparison with MSM
                                            4. 30.4 Discussion
                                          5. 31 Summary
                                            1. 7 CONCLUSIONS AND FUTURE WORK
                                              1. LIST OF PUBLICATIONS

TABLE NO. TITLE PAGE NO.

###### List of Tables

  1. 1 Axes of movement

FIGURE NO. TITLE PAGE NO.

## LIST OF SYMBOLS AND ABBREVIATIONS

Table 1: Axes of movement

[late after line= |  |
---|---|---
]Data/Abbreviation.csv Key=, Description= | - |

##  CHAPTER 1 INTRODUCTION

###  1 Taxonomy of Gait Analysis

The simplest definition of gait states that it is the manner and style of
walking (?). This can refer to any animal that can walk whether bipedal or
quadrupedal. It can be more sophisticatedly defined as the coordinated cyclic
combination of movements that result in locomotion (?). This definition can be
equally applicable to any form of activity that is repetitive and coordinated
so as to cause motion of the living being originating it. Gait can vary from
walking, running, climbing and descending the stairs, swimming, hopping and so
on; all of which follows the same principle by this definition. In the context
of this thesis, the word ‘gait’ generally refers to the ‘walking gait’ of a
human.

Out of all routine activities, walking (running or jogging) is by far the
activity that makes use of most of our resources within a short span of time.
The activity includes the work of the cardiorespiratory system (?), the
musculoskeletal system (?), the nervous system (?), the renal system (?) and
metabolism (?). This level of association is why the majority of the abnormal
activities that can occur in the body reflect in the gait of the individual
experiencing it. Hence, gait analysis plays a significant role as a clinical
diagnostic. When applied in clinical practice, gait analysis proves to be more
efficient when compared to a medical diagnosis that would otherwise be
expensive and consume much more time.

The broad spectrum of research on gait analysis can be classified based on
their use or their modes of operation as illustrated in Figure
LABEL:fig:taxonomy as a fishbone (or Ishikawa) diagram. The application of
gait analysis can be broadly classified into four areas, namely, pathology
assessment, gait biometrics, biomechanics and behavioural analysis.

Clinical gait analysis focuses gait-related pathologies in depth. The analyses
include measures taken to assess the level of progress associated with a
particular pathology and detecting the onset of the disease that affects gait.
Pathologies that are frequently connected to clinical gait analysis are
Parkinson’s disease, cerebral palsy, stroke and arthritis (?). A separate
study addresses the fall of the elderly (?). It helps in predicting the fall
and to devise an effective treatment for the medical condition. Further
studies revolve around post-operative study on gait – one that focuses on how
surgical treatments affect gait (?).

The attempts on recognizing people with their gait started back at the late
1990s (?). The research showed that the nuances in the natural gait of an
individual can be considered an innate biometric that can be used to identify
them. Apart from the possible application in authentication, gait biometrics
has become a topic of serious value in forensic science. This value became of
interest to the US Defense Department at the birth of the second millennium
(?).

The mass of data that can be extracted from gait can also be of statistical
relevance to provide inferences of a person’s behaviour. Statistical gait
analysis is inherently an interdisciplinary area that can aid the other areas
of application. These include the design of a simple mobile app that can
estimate units of calories burnt to complicated emotional inference systems
(?). Statistical inferences in gait can also be utilized in demographics study
which can be used to associate people into groups and study their group
behaviour that can be influenced from gait.

There are two ways in which gait features can be classified in biomechanics –
kinetic and kinematic (?). The tools that are used for the observation limit
the type of features that can be extracted. Kinetic gait analysis assesses the
forces exerted and absorbed, moment generated and accelerations involved, but
without the details that cover positional significance or orientation of
objects concerned. A good example would be a simple foot switch, it provides
the location and magnitudes concerning force but does not account the spatial
positioning of the articulation points. Kinematic gait analysis is the
converse of kinetic gait analysis. It describes the gait parameters that
involve motion, position and orientation without considering the forces which
cause the motion. The best example of a kinematic observation tool would be
the simple RGB camera.

A relatively new mode of gait analysis uses kinesiological EMG
(electromyography). An EMG is employed for recording the electrical activity
produced by the musculoskeletal system. An in-depth analysis of gait can be
done using this method. With an EMG, doctors can precisely diagnose which
muscle tendons are damaged or under-performing from an individual’s gait. The
assessment of gait through a kinesiological electromyography (KEMG) was first
lead by Inman to study both normal individuals and amputees. According to ?,
the advent of the KEMG has revolutionized clinical gait analysis from the
understanding of the knee-ankle functional link to the structure of treatment
of cerebral palsy. EMG are not adapted for biometric feature extraction due to
their invasive nature.

###  2 Historical Advents

####  2.1 Inception of Gait Analysis

The first study on gait dates back before 300 bc by Aristotle in Greece (?)
which depicted the elements of movements of animals. A more theorized version
was published by the Renaissance Italian physiologist, G. A. Borelli in 1680
(?, English translation by P. Maquet). His work illustrated the notion of
balance and how it was conserved in locomotion by the movement of the feet and
the area of support which it provides. It was at that time that the concept of
contraction in muscles was put forward for the cause of locomotion. The phases
of a gait cycle was first depicted by the Weber brothers in 1836 (?). The
major advancements of gait analysis over time is provided in Table
LABEL:tab:history in chronological order.

Year |  Event |  Researchers
---|---|---
$\sim$300 bc |  First work on De Motu Animalium – On the Movement of Animals (?) |  Aristotle
1682 |  A more theorized work on De Motu Animalium including the contractile movement of muscles (?) |  G. A. Borelli
1836 |  Clear illustration of the gait cycle and the first published book on gait (?) |  W. Weber and E. Weber
1873 |  Analysis of gait through single plate chronophotography (??) |  E. J. Marey
1878 |  Gait analysis through chronophotography using a 24-camera array (??) |  E. Muybridge
1881 |  Observed footing pattern through ink spray shoes (?) |  K. H. Vierordt
1895 |  The effect of loads in the biomechanics of gait. The first 3-D gait analysis (?) |  Braune and Fischer
1938 |  The first operational force platform for gait analysis (?) |  H. Elftman
1947 |  Theory of motor control and motor learning for posture and movement (??) |  N. A. Bernstein
1950 |  Extensive mechanical analysis of gait from hip, knee and ankle joints (?) |  B. Bresler and J. P. Frankel
1953 |  Mechanisms involved in natural energy conservation during human gait (?) |  Saunders et al.
1981 |  Human Walking: the definitive guide for the normal gait of humans (?) |  Inman et al.

The first kinematic analysis of gait was done by two scientists, Marey and
Muybridge in the 1870s through chronophotography. French scientist E. J. ?
visualized the phases of gait by taking multiple exposures of the moving
subject on the same photographic plate (translated into English in ?). His
initial study involved the motion of birds and insects. He later created the
first visible spatiotemporal imaging of gait by capturing a human subject
while walking with the prominent skeletal structure illuminated with stripes.
English photographer, Eadweard Muybridge captured chronophotographs by using
an array of 24 cameras that were made to shoot at a specified sequence of time
within a fixed period interval. He captured many different animals in the same
way (work made available at ?). Once such chronophotograph precisely
interpreted the gait of horses proving the fact that at a certain phase of a
galloping horse has all of its hooves off the ground. His chronophotographs of
humans were taken while doing a wide range of activities (?).

Profound progress in the scientific study of gait was observed at the end of
the 19th century by German anatomist Wilhelm Braune and physiologist Otto
Fischer in Der Gang des Menschen (?). Inspired by the E. J. Marey’s
chronophotographs, they applied scientific reasoning to explain the
biomechanics of gait under the effect of load and without load. The study
involved measuring the trajectories, velocities and acceleration of
articulation points along the three-dimensional space. With the knowledge of
the masses of each segment associated with the respective articulation points,
they were able to provide further inferences to the forces that are involved
in each phase of the gait cycle. Nikolai Bernstein, a neurophysiologist of
Moscow, extended the work of Braune and Fischer in the late 1930s. His notable
achievement was his work on the construction of movements in 1947 (?) by
applying his theory of motor control and motor learning to discuss the role of
the central nervous system (CNS) in limb coordination and postural control.
His work was later translated into English in 1967 (?).

####  2.2 Clinical Relevance

K. H. ? conducted a study to analyse human gait concerning the spatial pattern
of footsteps. His work involves attaching small-scale ink spray nozzles to the
shoes of the subjects and make them to walk on a moderately long surface in
which the imprints of the footsteps due to the spraying of ink can be readily
analysed. The first force plate reported was used by Jules Amar in 1924 as a
tool to help diagnose rehabilitation of patients who were injured during the
First World War (?). The apparatus includes the use of pneumatic three-
component force plates. Its application was of clinical relevance but was not
scientifically validated for its accuracy. The first successful force platform
was introduced by ?. It consisted of a set of springs attached to force plates
spread across the stepping region. The force is measured optically using a
high frame-rate camera from the vertical up and down movement of the force
plates measuring both point of force as well as its magnitude. This apparatus
succeeded his previous experimental setup proposed in 1934 (?) which only gave
the point of application of the force but not the magnitude. All these early
force plate designs were purely mechanical in their operation.

Much of the advancements in modern gait analysis were reported from
California. This includes the detailed mechanical analysis of gait from the
hip, knee and ankle joints by ?. Most were specifically from the Biomechanics
Laboratory founded by V. T. Inman and H. D. Eberhart during the 1940s to 1960s
(?). Once such significant research includes the mechanisms in which the body
conserves energy during gait (?). The first computerized video analysis of
gait was done in the 1970’s by David H. Sutherland who was mentored by Inman
(?). He reinvented the art of EMG to be used for analysing gait. D. H.
Sutherland was known to be the founder father of clinical gait analysis (?).
The widely accepted definitive guide for normal gait was published by ?.

The history of researchers who shaped gait analysis before World War II is
detailed in ?, after its period in ? and the modern era in ?. Richard ?
provided a detailed account on the history of gait analysis focusing on the
period before the application of computerized observation.

####  2.3 Interest in Biometrics

? concluded that gait could pose a behavioural property which is unique to any
human individual provided that all possible gait parameters are taken into
account. Despite this statement, gait biometric research has proved that a
subset of the vast features of gait is sufficient to produce a system that
yields a performance of practical relevance.

Gait can be used as a non-obtrusive form of biometric (?). This unique trait
sets it apart from the other modalities. The other established forms of
biometrics include face, fingerprint, handwritten signature and iris; all of
which requires a closer observation of the subject to process the images at a
sufficiently high resolution. However, gait can be observed from a person even
from a lower resolution surveillance camera with minimal level of cooperation
from the subject who is to be identified.

The Defense Advanced Research Projects Agency (DARPA) launched the Human
Identification at a Distance (HumanID) programme in the year 2000 (ended in
2004). This research programme focused on human identification through the
face, gait and the application of new technologies. The intention of the
programme was to use the state of the art unobtrusive distance biometric
techniques on real world data. Gait-based recognition played a significant
role in here. Major institutions which took part in this programme were
Massachusetts Institute of Technology, Carnegie Mellon University, University
of South Florida (USF), University of Maryland, Georgia Institute of
Technology, and the University of Southampton (?). The datasets compiled by
most of these institutions are publicly available. Section LABEL:sec:datasets
discusses more on those datasets.

###  3 Nomenclature

The knowledge of the technical terms associated with the study of gait
analysis is required for a deeper understanding in gait research. The stages
of the human (normal) gait cycle are illustrated in Figure
LABEL:fig:gaitcycle.

####  3.1 Phases of Gait

Gait phases and positions vary between two terminologies: the traditional
terms and the standard of Rancho Los Amigos set by J. ? to handle both normal
and pathological gait. Although both of which are used interchangeably, the
knowledge of both system of terms is required to comprehend the literature. A
standard human gait cycle consists of eight key positions, with the initial
contact added twice to make the cycle complete. Note that the sequence of
positions is accounted only for one foot. Hence, a complete gait cycle or a
stride is the activity that concerns a sequence of events between two
consecutive heel strikes (or heelstrikes) of the same foot. Whereas the
activity that occurs from one foot’s heel strike to the other is called a
step. The time it takes for completing a stride is called cycle time. In a
normal gait, the gait cycle of one foot is displaced from that of the other
foot by a fixed phase shift.

The periods between consecutive gait positions are called the phases of the
gait cycle. The load response phase depicts the response made by the body to
absorb the impact of heelstrike by flexing the knee, contracting the
quadriceps. In the mid-stance, the knee extends, and the associated foot is
kept flat on the ground, i.e., both toe and heel are maintained in ground
contact. During the terminal stance, the legs extend further while keeping its
contact with the ground to advance the body forward. The legs then start to
flex again to get ready to swing at the pre-swing phase. Then, at the initial
swing, the foot accelerates to provide a forward momentum and controls its
speed at midswing. The foot decelerates to reach its position, the next
initial contact.

The period in which the foot is on the ground is called the stance phase, and
it composes 60-62% of the gait cycle (?). The remaining period, when the foot
is off the ground, is called the swing phase. The proportions, however, differ
depending on the individual and the speed of the walk. The stance phase
encompasses the first four phases of the gait cycle from initial contact to
toe off. The swing phases cover the phases between the toe off and the
successive initial contact. The stance and swing phases alternate to produce a
locomotion. While one foot is in swing phase, the other would be on stance
phase for balance.

####  3.2 Gait Parameters

Most of the following terminology is taken concerning the ones prescribed by
?.

Cycle time

is the time taken to complete a single gait cycle. It is also known as a gait
period.

Double support

is the interval in which both feet are in contact with the ground when the
stance phase of both legs overlap.

Step length

is the length covered between initial contact of one foot to that of the other
feet.

Stride-length

is the distance displaced by the foot during its stride (one complete gait
cycle); consists of two step lengths.

Cadence

is the number of steps taken in a given time usually measured in steps per
minute

Walking base

is the perpendicular distance defined by the space between the track of both
feet.

Walking speed

is usually measured in metres per second, but can also be measured in statures
(height) per second in clinical practice.

The double support period would define the difference between walking and
running. As long this period is positive, the person is walking, and her/his
speed is inversely proportional to the magnitude of this interval; otherwise,
the person is considered to be running (or jogging). The double support period
is an important measure during a walking race (or race walking) where a
negative double support time would indicate that the candidate is running
instead of walking.

The stride length of both feet is the same regardless of the effect of the
pathology. However, the step lengths commonly differ for a pathological gait
due to differing stance and swing phase proportions between the left and right
leg. For instance, a pathologically affected leg would tend to spend lesser
stance time leading to a lesser swing in the other leg. Hence, a shorter step
length on one side would infer antalgic conditions on the other side.

####  3.3 Reference Planes

Gait is usually observed in one of three planes of reference: sagittal,
transverse, or frontal. A clear depiction of the reference planes is as
provided in Figure LABEL:fig:planes. The sagittal plane divides the body into
right and left proportions. Most gait research in literature is based on the
sagittal angle as it is perceived to be the simplest angle to obtain the
salient features. When observed at a sagittal angle, the walking subject tends
to move from left to right or vice versa, i.e., the person is said to walk
frontoparallel to the viewpoint. Along the frontal or coronal plane, where the
body is divided into front and back halves, a walking subject moving forward
in motion is perceived to move towards the observer. The transverse or
horizontal plane divides the body into upper and lower halves. There is not
much literature that analyses gait in this angle due to the difficulty of
extracting information from it.

Three pairs of rotatory motion are required for the body to make any means of
movement: internal and external rotations, abduction and adduction, extension
and flexion. They are illustrated in Figure LABEL:fig:movement. In general,
flexion and extension occur in the sagittal plane, adduction and abduction
take place in the frontal plane, and other rotatory motions occur in the
transverse plane. Note that in Figure LABEL:fig:movement, the internal and
external rotation shown is performed by the knee and the thighs, not by the
ankle. If done at the ankle, the similar result is said to be caused by the
forefoot’s adduction/abduction. Flexion at the ankle is known as dorsiflexion
whereas its extension is known as plantarflexion.

####  3.4 Performance Metrics

The following terms are used when evaluating the performance of biometric
systems.

FRR

(False Reject Rate) is the rate at which an authentic subject is rejected by
the system under observation.

FAR

(False Accept Rate) is the rate at which an impostor/intruder is accepted as
an authentic subject by the system under observation.

AER

(Average Error Rate) is the average of FRR and FAR,

i.e, $\text{AER}=(\text{FRR}+\text{FAR})/2$.

(Equal Error Rate) is a measure specific for threshold-based authentication
systems. At a certain threshold, both FRR and FAR becomes equal; this error
rate is termed as the EER.

is a measure specific to distance-based identification systems. Accuracy at
rank $k$ infers the probability of the true identity being present in the top
$k$ closest identities returned by the system.

(Correct Classification Rate) is simply the accuracy of the classification
system used for identification (recognition).

(Receiver Operating Characteristic) is the curve generated by plotting the
verification rate (sensitivity; $1-\text{FRR}$) against the FAR in the case of
authentication.

(Cumulative Match Score) is the recognition rate at rank $r$.

(Cumulative Match Characteristic) is the curve generated by plotting the rank
against the CCR for identification.

###  4 Research Directions

Gait analysis is an interesting study and is proven to be applicable for
diverse domains of applications. There are many open areas of research from
which one can choose in the field. They can be segregated into two groups –
active areas and those that are least explored.

####  4.1 Active Areas

  * •

Reliable recognition independent of clothing style. Several algorithms have
been outlined to combat this problem (e.g., ??) and a handful of datasets to
help with this regard (CASIA-B, OU-ISIR-B, TUM-GAID, and so on). However,
state of the art gait biometrics algorithms do suffer a significant
depreciation when the clothing style changes.

  * •

A comparative study of gait pathologies. Various papers are published
classifying diseases concerning their nuances in gait (e.g., ???). The
clinical attention to this segment is growing as there are still so many gait-
related pathologies that are still not compared as of yet.

  * •

Gait recognition at different walking speed. With datasets like CASIA-C and
OU-ISIR-A, gait recognition at different speeds is becoming more of interest
(???). There are, however, further improvements that can be made.

  * •

Correlation between walking and running gait. When an individual switches from
walking to running, the double support time turns negative while the arm swing
increases. To achieve balance in this state, the body makes many modifications
to the posture.

  * •

View-independent gait recognition. Numerous studies that propose recognition
models that could cope with multiple views. Steady progress is still being
made in this area.

  * •

A comparative study between model-based and model-free recognition methods.
Though studies contrast on the merits and demerits of both approaches, there
is still space for an in-depth comparative analysis.

  * •

Unbiased comparison of state of the art gait recognition. The methods in
literature show conflicting results of correct classification rates when
performances of established algorithms are compared, even with the same
dataset. The existing algorithms are to be compared with standard datasets
without bias.

####  4.2 Least Explored

  * •

Correlation between kinetic and kinematic features. Studies show how kinetic
and kinematic features can be assessed together (?). Some utilize both for
gait assessment (???). However, relationship between kinetic and kinematic
features of gait is still blurred.

  * •

Characteristic differences in gait based on ethnicity. People from different
regions do have a characteristic gait. CASIA datasets are composed of Chinese
individuals, while the USF gait dataset are largely American. More datasets
are to be created across various ethnicity to study the associated differences
between them in terms of their gait.

  * •

Ranking of factors that inhibit gait. The confounding factors of gait are as
listed in Section LABEL:sec:covariate-factors. Though known, they are not
compared in a way that would determine to which extent they inhibit gait. For
instance, to what instance could training affect biometric gait? This has been
an open question to which no reports are currently available.

  * •

Different machine learning methods for gait recognition. The novelty of the
recognition algorithms lies on the technique in which features are extracted.
Commonly used classifiers are Artificial Neural Networks (ANN), $k$-Nearest
Neighbours ($k$NN), and Support Vector Machine (SVM). A detailed analysis on
how different classifiers effect the prediction rate through state of the art
gait feature extraction algorithms is yet to be reported.

  * •

Stratification of gait patterns. People can be grouped by means of their gait.
This grouping can reveal certain insights to the behaviour or traits that
members of the same group share. In theory, we could find some means to
cluster and associate them to different strata.

  * •

Gait in demographics. Deeper statistical inferences can be drawn from gait
within social groups. That is, the similarities of gait can be assessed within
a demographic group, and differences can be studied across the groups in terms
of gait. This is the converse of stratification wherein people are grouped
based on their gait similarities.

  * •

More statistical gait analysis. There are limited amount of studies reported
in this area. Gait has been statistically analysed extensively for assessing
pathology and studying gait biometrics. However, there is untapped potential
lying within the broader statistical analysis of gait.

###  5 Contributions

Four gait-based biometric assessment methods are proposed in this study: a
gender classification algorithm, a gait recognition method, and two
authentication paradigms.

Existing methods for gait-based gender classification extract spatiotemporal
relationship between the frames of a gait sequence to make a single prediction
on the subject’s gender. The method proposed in this thesis, however, makes a
prediction for every pose assumed by the subject during gait. The prediction
of the gait instance itself is taken through a majority vote. Two sets of
features are explored for this purpose, viz., EFD and RCS.

Gait recognition (or identification) is the process of identifying a person
from her/his gait alone. Not all features extracted from gait may correlate
with the identity of the subject, especially in the case of covariate factors
that affect a person’s walking style. Though gait recognition is well-
developed in literature, the algorithm proposed in this thesis optimizes the
efficiency of template-based recognition systems. The resulting template would
include only features that are least affected by the covariate factors.

Gait authentication (or verification) is a process in which the system
ascertains whether the subject is whom she/he claims to be based only on the
gait observed. The literature available for this topic is limited as most
adopt the features that are used in identification for authentication as well
while defining closeness to the claimed identity through a Euclidean
threshold. Two completely new paradigms proposed in this thesis greatly reduce
the average error rates of gait authentication systems.

This study will also attempt to answer the following research questions.

  1. (a)

Can gender classification be improved to cope with occlusions?

  2. (b)

Can chain encoding help with gait-based gender classification?

  3. (c)

How to optimize feature selection for video-based gait recognition?

  4. (d)

How to enhance the performance of gait authentication systems?

  5. (e)

Is the sagittal angle really the best angle for gait biometrics?

###  6 Organization of the Thesis

The rest of the thesis is organized as follows. Chapter 2 details the relevant
literature in the field of gait biometrics. The research contributions are
presented in chapters 3 through 6. These chapters elucidate the state of the
art and highlights their limitations followed by methods proposed to overcome
them. Algorithms that are included are experimentally verified through
established benchmark procedures and quantitatively compared with the existing
methods.

Chapter 2:

Provides a brief interpretation of the literature closely associated with the
research work along with the description of the available datasets for the
evaluation of gait biometrics.

Chapter 3:

This chapter focuses on the most popular soft biometric aspect of gait
biometrics – gender classification. Though the existing method produces close
to perfect identification, the proposed method, pose-based voting, allows for
better recognition even in partially occluded gait sequences.

Chapter 4:

Gait recognition makes up for a major share in gait literature. This chapter
presents the genetic template segmentation technique for gait recognition.

Chapter 5:

This chapter presents an innovative framework called  multiperson signature
matching to outperform the existing threshold-based gait authentication
framework. This scalable method improves its performance in proportion to the
system population.

Chapter 6:

When the system population is considerably low the effectiveness of the method
proposed in chapter 5 is suppressed. To overcome its limitation, this chapter
presents a novel framework using the multivariate Gaussian Bayes posterior
probability.

Chapter 7:

This chapter concludes the findings of the thesis and presents the work
planned for the future.

##  CHAPTER 2 LITERATURE SURVEY

Gait analysis is composed of three components, viz., kinematics, kinetics, and
electromyographic data (?). Each of these components can be separately
analysed and studied. There is a large variety of gait analysis tools that are
commercially available as shown in Table LABEL:tab:tools. A detailed account
of the different ways to measure gait analysis is provided by ?.

S.No. |  Tool | Mode |  Description
---|---|---|---
[late after line= |  |  |
]Data/GaitTools.csv Tool=, Mode=, Description= . |  |  |

###  7 Approaches for Gait Biometrics

The biometric features of gait can be extracted through a variety of ways.
According to ?, the approaches for gait biometrics can be classified into
three types, viz., wearable sensor-based, floor sensor-based and machine
vision-based methods. Research over the past decade has vastly developed the
field of gait biometrics since then. The methods can be grouped by the tools
employed for feature extraction as follows:

  1. (a)

Inertial sensors

  2. (b)

Footstep analysis

  3. (c)

Acoustic data

  4. (d)

WiFi signals

  5. (e)

2D and 3D camera (computer vision)

Each of these tools has its pros and cons that are assessed and addressed in
ongoing research.

####  7.1 Inertial Sensors

An inertial sensor is a device used to measure the motion of the attached
subject based on inertia. The ones used for gait analysis are usually capable
of measuring motion across three dimensions (triaxial). They can be either
worn or carried by the person who is to be identified. When wearable sensors
are used, especially when the system consists of multiple devices, there might
be a problem of inconsistent sensor attachment (?). Sensor placement during
testing phase should be the same as in the training phase. Inconsistent
attachment can introduce an additional error factor to the biometric system.
There is also a greater problem sensor orientation (?) which can completely
change the pattern of readings observed in the three-dimensional space.

An accelerometer is used to measure the acceleration exerted along a given
direction. A triaxial accelerometer would give readings for accelerations
along all three spatial axes. They are by far the most common form of
kinematic gait measuring equipment as they are the least expensive of the
alternatives and the easiest to calibrate and process. The frequency response
of an accelerometer shows the maximum deviation of its sensitivity (?). The
triaxial accelerometer is the most common form of inertial sensor applied for
gait recognition.

The largest inertial sensor database till date was presented by ?. It consists
of sensor data from 744 subjects. Each subject wore a gait capture belt around
the waist. The belt was fitted with three triaxial accelerometer-gyroscope
couple and a smartphone. The triaxial accelerometer in the smartphone was also
considered for the data collection. ? and ? both report an EER of 14.3% using
this dataset.

The main challenge to mobile phone-based gait authentication approaches is
that the subject should bear the phone in a specific location – usually the
trouser pocket for the most successful operation. So this constraint refrains
the user from using any other attire that does not include a pocket. Other
locations for inertial sensor placement include hip, back of the waist, arm,
and ankle. A detailed account of gait verification through wearable sensors
was given by ? and ?.

####  7.2 Footstep Analysis

A person’s footstep can be analysed either by a wearable sensor, like a
footswitch, or by a floor mounted sensor like a force platform. A footswitch
measures the pressure applied by the foot on the ground. It can extract
temporal data from swing-phase and stance-phase, and cadence and velocity.
However, it is unable to provide spatial data such as step/stride length. The
early form of footswitches used for gait analysis was designed through a
conductive material placed on the inner soles of a pair of shoes (?). The
current technology allows more efficient sensors to be embedded in the soles
of the shoe such as advanced force-sensing resistors (FSR). A simple way to
implement this design is as shown in Figure LABEL:fig:footswitch. It works on
the principle that when force is applied to a conductive polymer surface, the
resistance across varies inversely to the force applied – a method patented by
F. N. ?.

S. J. ? designed an apparatus that combines both the footswitch and inertial
sensors and later developed it (?). It includes FSR sensors along with
accelerometers and gyroscopes to a shoe thus forming a ‘GaitShoe’. This
apparatus was used to assess heel strike and toe off as well as to approximate
foot orientation and position. Although the footswitch is useful in clinical
gait analysis, the practical relevance of this device is still questionable
from a biometric perspective.

To enable gait authentication in closed spaces and further avoiding the
privacy concerns of visual surveillance systems, floor mounted systems became
an option to extract gait biometric features (?). These instruments allow the
ground reaction force (GRF) to be measured directly from the floor on which
the individual walks. The terms force plate and force platforms are used
interchangeably since a platform can be assembled through multiple force
plates, but in this context, they are described as two different instruments.
A force plate is a metal plate coated with a non-slip substance beneath which
transducers are attached that converts pressure to electrical signals which
can be amplified and processed (?). A force platform is the same in definition
but can be extended to a larger surface which can facilitate efficient
tracking of the foot placement during gait such as a walkway, e.g., GAITRite
(?).

A detailed review of footstep recognition is given by ?. Recently, ? conducted
detailed experiments on the biometric performance of the GRF using various
machine learning algorithms. They show that the EER can be reduced to below 5%
using the GRF data alone. The literature in footstep analysis is not as vast
as in computer vision-based and inertial sensor-based techniques. The main
reason is due to the lack of a publicly available dataset. Force platforms are
expensive instruments and are difficult to install, maintain and analyse. It
would be one of the most expensive gait biometric assessment tools explained
in this section.

####  7.3 Acoustic Data

The biometric features that can be extracted through footsteps can also be
derived from the sound generated by them. The temporal analysis of the
acoustic response of footsteps can be a highly unobtrusive gait biometric. The
use of acoustic footstep analysis for gait identification is relatively new in
literature. The sounds that can be heard from the same individual can vary
across different footwear and surfaces. The challenge of acoustic gait
recognition is the extraction of features that are invariant to these
conditions. ? attempted to overcome this issue through mel-cepstrum analysis.
This system was succeeded by ? using psycho-acoustics parameters, viz.,
loudness, fluctuation strength, sharpness, and roughness.

The apparatus of ? consists of three microphones to hear the sound and three
sensitive accelerometers as seismic sensors in the vicinity in which the
subjects are to be identified. The sound that is generated from the footsteps
is composed of a mixture of waves of both audible and inaudible frequencies.
The components from the higher frequency bands are first converted to the
audible range before the application of predictive modelling. ? used the
acoustic Doppler sensors as an inexpensive alternative to the customized
apparatus. ? approached the acoustic gait recognition problem in a similar way
to speech recognition through Dynamic Time Warping (DTW). ? applied acoustic
gait analysis for person identification, gender classification and shoe-type
identification where SVM was the classifier used for the prediction. ? used
only the audio data from the TUM-GAID dataset (?) with Hidden Markov Models
(HMM) for prediction.

####  7.4 WiFi Signals

Groundbreaking research at Massachusetts Institute of Technology resulted in a
novel technique to track gait using only WiFi signals. Fadel Abib and Dina
Katabi first proposed a system called Wi-Vi – WiFi Vision – to detect and
locate (to a degree) objects in motion through walls (?). The system was also
able to recognize primitive gestures. All it requires is a WiFi transmitter
and two or more receivers; it requires no wearables or attachable items for
its functioning. It can approximate the location of moving subjects behind
concrete walls, hollow walls and wooden doors by computing the Doppler shift
and spatial angle of reception. It uses a process called nulling to eliminate
static objects from detection.

Katabi’s research group generalized this concept for 3D tracking of humans
(?). Their new system, WiTrack, is a wireless device that can track a person
within a 3D space which even works through occlusion and not limited to line-
of-sight. Body parts can be coarsely tracked more efficiently than Wi-Vi but
not so in detail, yet its accuracy was claimed to exceed that of RF
localization systems at its time. WiTrack was more of an intermediate between
RF-based localization systems and human-computer interaction systems like
Kinect. It uses a single transmitter and three receivers to triangulate the
motion of the subject. WiTrack is tuned only to pick out objects in motion and
neutralize signals associated with static objects through subsequent reception
cancellation. This process is termed to be background subtraction. The
tracking is not disrupted by small moving objects. It is supposedly the first
3D tracking device with an error within the order of centimetres; median of
deviation along $(x,y,z)$ is $(10,13,21)$. While the first design (?) was only
intended to track one human at a time, it was developed further (?) for multi-
person localization. Recent developments of WiFi-based gait recognition
includes WiWho by ? and WiFi-ID by ?.

####  7.5 Computer Vision

Computer vision is by far the oldest and the most researched approach for gait
biometrics. While MoCap systems are used extensively in clinical aspect,
methods that require just regular 2D video can be easily employed in legacy
surveillance systems. The rest of the methods discussed in this thesis employ
only computer vision.

##### MoCap Systems

Motion capture (MoCap) provides the most accurate technique to measure gait
analysis. This approach consists of attaching reflective markers to strategic
locations on the subject’s body. Special cameras emit infrared (IR) strobe
lights which reflect off the markers and back into the video camera. The
camera can then pick up point-light motion of the reflectors that are visible
from its viewpoint. An array of cameras (mostly eight) is positioned around
the subject so that by computing on the inputs of the cameras combined, a
three-dimensional depiction of the subject can be visualized.

A simplified schematic of a typical MoCap system is as shown in Figure
LABEL:fig:mocap. They are of great value in clinical practice as they provide
precise analysis required for treating gait related pathologies (???).

##### RGB Camera

A typical MoCap system can be very expensive and complex to set up. Two-
dimensional RGB cameras, on the other hand, are cheap to obtain and do not
require a complicated setup process as associated with MoCap systems. Since
they lack the expressiveness of a fully functional MoCap system, their
application for clinical use is limited. Nevertheless, due to their
unobtrusive and pervasive nature, they find their application extensively in
gait identification.

? proposed a method to extract the articulation points from a human gait from
a single RGB camera. These points together formed a point distribution model
(PDM)(?) which would hence be helpful for a model-based processing of human
motion during gait. 100 contour points were approximated using the silhouette
border, and 13 articulation points were labelled on the image manually. This
labelling would facilitate the construction of an Active Shape Model (ASM)
(?). The ASM would combine the PDM shape model along with the grey information
of associated points to be able to generalize to new images in the dataset.
The CASIA-B was used for experimentation. By training the ASM using only 14
subjects along four directions, the system was able to approximate the
landmark points to unseen images in the dataset.

##### Microsoft Kinect

The main components of the Kinect is an IR projector, an RGB camera, and an IR
camera. Details of its function can be found in ?. The concept behind depth
perception involves the analysis of structured light, a patented technique by
PrimseSense (?). The infrared projector projects a structured image called a
speckle pattern. Objects in range reflect back the speckle pattern. When this
pattern is viewed away from the angle of projection, parallax occurs. The
parallax would aid the IR camera in depicting the distance of the objects in
view from the device. By combining the image feed from the RGB camera, depth
information inferred from the IR camera and machine learning the Kinect can
produce 3D model of the object it sees. It is much less expensive than a MoCap
system and is more flexible than a simple RGB camera. Several techniques are
proposed to extract gait features from a Kinect camera (???). It is much more
effective in gait-based recognition (??), and it can even be utilized for
clinical gait analysis (?).

###  8 Soft Biometrics

Soft biometrics are characteristic that can apply to a certain group of
individuals. Some of the prominent human soft biometrics include ethnicity,
gender, age, height, weight, eye colour, scars and marks, and voice accent
(?). These properties are considered ‘soft’ because they do not distinguish an
individual from all others. For instance, there can be more than one person
who possesses a given height and weight, and it is possible to mimic the
accent of another person. Out of the above, the first four can be identified
through gait alone. Prominent research on soft biometrics focus on age
estimation and gender classification (?).

####  8.1 Age Estimation

For an average human, the muscle fibres and muscle coordination deteriorate in
the later stages of his/her life. This process affects the gait of the aged
human. The probable gait-related symptoms that can be exhibited at the elderly
stage are the loss of stability, momentary imbalance, and tendency to fall
(?). The degree to which any of these symptoms occur can differ from one
person to another but are indicators of old age nonetheless. A person’s gait
can also be an indicator of the level of maturity. One can easily tell the
gait of an adolescent from that of an adult even when height is not a factor.

? studied the difference in the age-related biometric component of gait across
both genders using a subset of the USF dataset consisting of 79 subjects with
ages ranging from 19 to 59 years. GEI (?) was adopted for feature
extraction111The GEI is explained in detail in Section LABEL:sec:gei-based and
multi-label guided subspace was applied to reduce the dimensions while the
$k$-Nearest Neighbour ($k$NN) algorithm was used for the classification. The
age information was encoded as a sequence of 8 binary labels, one of which was
also the gender. One $k$NN was trained for each of the labels in the sequence.
The label sequence was decoded to obtain the estimation of the age of the
subject. The mean absolute error was reported to be approximately 5.6 years.

? conducted a much more extensive experimentation for age estimation wherein
the gait instances from 1728 subjects with the ages between 2 to 94 years were
recorded. The Gaussian Process Regression was considered to be a highly
efficient face-based age estimation algorithm (?) and it was adopted by ? for
gait-based age estimation. They have concluded that the performance of
frequency domain features and the GEI features are equal with a mean absolute
error of 8.2 years.

####  8.2 Gender Classification

Gender is found to be the most popular soft biometric that can be derived from
human gait. The first major work on gait-based gender recognition was done by
?. The silhouette of each frame in the gait cycle was segmented up into seven
regions, and an ellipse was fit to each of them. The geometric parameters of
the ellipses collectively become the feature set of the gait instance. The
similarity measures between male and female instances were trained using an
SVM classifier to form the recognition system. A similar method of segmenting
was suggested by ?. Only this time, the GEI of the gait instance was segmented
instead of the individual silhouettes. The GEI template was split into 6
regions whose boundaries were obtained from the analysis of the training
instances. A simple pixel-wise similarity measure was adopted to form the
feature space which was reduced through independent component analysis (ICA)
and classified through SVM. ? also split the GEI to multiple components based
on body parts. Additionally, they assigned weights to each region which was
then fed to SVM for classification. The weights signify the effectiveness of
gender discriminability to the respective component.

? adopted an entirely different method for spatiotemporal feature extraction
to compiling gait templates such as the GEI. The process involves extracting
shape features from multiscale grids. Their method reports the best
performance till date. As the previous methods were constrained in terms of
the angle of view (mostly at the sagittal angle), ? have designed a method
that is said to work in arbitrary view angles. They propose the cluster-based
averaged gait image (C-AGI) as an alternative to the GEI to cope with multiple
walking directions. Sparse representation metric learning and classification
(?) was used for gender prediction.

Apart from the studies conducted with standard video cameras, there were also
research done with three-dimensional data in connection with gender
identification from gait. ?? used Microsoft Kinect (?) to get depth
information along with the RGB visual information. As the data obtained was
computationally large, both required the use of extensive feature reduction
before classifying with SVM. ? claim to improve the recognition accuracy with
the help of motion capture information. The work in this thesis focuses on the
data obtained from traditional cameras as it is possible to obtain optimal
classification rates without the use of depth information. Furthermore,
algorithms designed for conventional two-dimensional cameras have a much
greater scope as they can be readily augmented with legacy surveillance
framework with minimal modification.

###  9 Hard Biometrics

Gait is characteristic to one’s individuality it is considered to be an
accepted form of unobtrusive biometrics (?). Hence, in theory, it does not
require the cooperation or the awareness of the individual being observed.
Some researchers still argue that gait itself as a soft biometric
characteristic (??), most researchers today have proven that a gait signature
can be established as a hard biometric. A simplified template for a gait
recognition system is as shown in Figure LABEL:fig:gaitrecog. A gait sequence
is a temporal record of a person’s gait. These are usually translated into a
features database after some means of preprocessing and feature extraction.
When the test gait sequence is given for identification, the system refers the
gait feature database and returns the closest match (if any) according to some
criteria.

The approaches for gait recognition can be broadly classified into two types:
model-based and model-free. Due to its simplicity and efficiency in use,
model-free methods are more prevalent. Model-free techniques can be further
categorized as template-based and non-template methods.

Many benchmark datasets are available to compare the performance of one
algorithm with another. Recent literature commonly use the USF Gait Challenge
dataset and CASIA-B. The aforementioned and the remaining available datasets
for gait biometrics will be detailed in Section LABEL:sec:datasets.

The recognition process involves the use of an established machine learning
algorithm such as the HMM (used in ?????). Some apply different techniques
such as canonical analysis (??), spatiotemporal correlation (?), and DTW
(???). A few others devise newer techniques for closeness representation. All
others employ the nearest neighbour classifier (recent examples: ??).

####  9.1 Model-free Techniques and Early Approaches

The vast majority of model-free techniques tend to have a strong reliance on
spatiotemporal analysis of silhouettes of the individual during gait. A
spatiotemporal analysis takes into account the variation in the spatial domain
with respect to that in the time domain. So when this is applied to gait
recognition, the analyses involves the observation of the spatial locations of
body parts and their movement in different stages in time.

Almost all model-free methods have background subtraction and silhouette
extraction as the first step. Background subtraction is a simple method in
which the change in pixel values between one frame and the successive frame is
observed to bring out only the objects that are seen in motion. From these
objects, the moving human silhouette can be extracted. The result of
background subtraction is usually binarized in which the moving object seems
to be white and the background is black, or vice versa in some cases. The
novelty mainly lies in how the features are extracted.

##### Earlier model-free methods

The earliest known spatiotemporal gait recognition techniques started in the
late 80s. (?) proposed to recognize gait at a sagittal angle with the subject
walking frontoparallel. It modelled the human gait in the form of a set of
spatiotemporal snakes (?) from the slices of the moving parts of the human
contour along the time domain. The recognition accuracy they have obtained
with 26 image sequences across five human subjects reaches up to 83%. ?
introduced the concept of optical flow in gait recognition. In principle, the
points in the image sequence that vary with time tend to oscillate
periodically during the subject’s gait. By observing the optical flow of these
points, the $m$ time varying scalars can be produced. The phases of
oscillations from these scalars are used to represent the gait instance. They
reached a higher recognition rate of up to 95% by testing their technique with
seven instances for each of six human subjects.

An earlier version of template matching method was proposed by ?. They used
the Eigenspace transformation (EST), as adopted by ?, to convert the gait
taken as a sequence of images to a template called the ‘eigengait’. On top of
this, canonical space transformation was applied with generalized Fisher
linear discriminant function to separate the classes boundaries required for
prediction. However, they also seemed to use the small dataset used by Little
and Boyd for their application to show a questionable accuracy of 100%.

Though the above methods show attractive recognition rates, all the methods
proposed at those times before suffered one major drawback: their accuracies
were biased to their samples which were too small when considering the
application as a biometric. The advent of the DARPA’s HumanID programme in the
break of the second millennium led to a major development in gait biometric
research. Bigger datasets with more variations challenged researchers to
design better algorithms.

####  9.2 Model-free Template-based Methods

A gait template can be obtained as the result of transforming the sequence of
silhouette images taken from a gait video to a single image that holds the
composition of the motion-related features of the sequence. Some of the
notable template-based gait recognition methods in literature are listed in
Table LABEL:tab:template-based.

? assessed the symmetry of the extracted silhouette using a generalized
symmetry operator. The contours obtained from the silhouette sequence were
used to produce a symmetry map. Euclidean distance between Fourier descriptors
was used as a similarity measure for gait recognition. They reported a CCR of
97.3% for $k=1$ and 96.4% $k=3$ using the nearest neighbour classifier.

? proposed a unique method to recognize gait by analysing the contours of the
silhouettes. The shape of the contour of a given silhouette sequence was
converted to a template with the use of Procrustes shape analysis. Different
exemplars were created for each viewpoint. They tried three types of nearest
neighbour algorithms, viz., NN, $k$NN, and ENN (NN with class exemplar), in
which ENN provided the best results for gait recognition.

Experimental results of ? suggests that DTW and HMM can be combined to produce
a better gait recognition result. DTW was used to align the motion of the arms
and legs to normalize the phase of gait while HMM was used to define the leg
dynamics.

S.No. | Year | Study | Technique | Plane | Dataset
---|---|---|---|---|---
[late after line= |  |  |  |  |
|  |  |  |  |

The efforts of the University of South Florida (?) has brought forth a new
revolution to gait recognition by compiling the largest dataset of its time,
the USF Gait Challenge. It was categorized to 12 challenge probe sets for
experimentation and a gallery set for training. A simple baseline algorithm
was developed to facilitate users of the dataset to compare the performance of
gait algorithms effectively. The algorithm involves the use of the Tanimoto
similarity measure to gauge the similarity between two silhouettes which is
given by $Sim(p,q)=|p\cap q|/|p\cup q|$. Here, $p$ and $q$ are two binarized
images where each image is represented as an ordered set of pixel values. The
correlation between the silhouette similarities provided the measure of
closeness used for the recognition step. Further details of the USF Gait
Challenge dataset is provided in Section LABEL:sec:datasets.

The most notable form of silhouette-based gait recognition techniques use the
production of a Gait Energy Image (GEI) template (?) from the gait cycle.
Technically, the GEI shows how energy is dissipated spatially through the
stages of the gait cycle. It is so prevalent in literature such that
silhouette based methods that are published after its time (2006) can be
classified either as GEI-based or non-GEI-based. The GEI is created by
superimposing the pixels of silhouette sequence of a given gait by summing the
values and averaging them resulting in the output as a grey-level image of
proportional pixel intensities. Hence the final stage boils down to image
comparison of the test GEI with the GEIs in the gait database. An example of a
GEI obtained from a subject’s gait from the CASIA-B dataset along the sagittal
plane is as shown in Figure LABEL:fig:GEIseq. The features are learned based
on Multiple Discriminant Analysis (MDA) (?) after passing through Principle
Component Analysis (PCA) (?) for dimensionality reduction. The Euclidean
distances towards the class centres with respect to the features provide the
closeness measure for recognition. Experimental results of ? show recognition
rates averaging around 55.64% over all probes of the USF Gait Challenge v2.1
database, but normal recognition rates reach from 90% to 100%.

Although it was a a significant breakthrough in gait biometrics, the GEI had
one major setback, namely, covariate factors. When the subject, for instance,
wears an overcoat, the CCR can drop to less than 50%. Many studies then
published their version of an energy image adopted from the core principles of
the GEI to overcome this drawback. They differ mainly in the collation
operation – the operation used to combine the silhouette sequence to a single
image. Some of the notable ones are given as follows.

  * •

Gait Flow Image (GFI) incorporates optical flow (?)

  * •

Enhanced GEI considers only dynamic regions (?)

  * •

Gait Fluctuation Image (GFlucI) highlights temporal irregularities in gait (?)

  * •

Chrono-Gait Image (CGI) maintains temporal information through colour (?)

  * •

Motion History Image (MHI) for activity recognition (?) was adapted for gait
recognition (?)

  * •

Gait Entropy Image (GEnI) incorporates Shannon’s entropy function for each
pixel (?)

  * •

Active Energy Image (AEI) takes the average of each frame-to-frame difference
(?)

  * •

Depth Gradient Histogram Energy Image (DGHEI) combines depth information from
Kinect (?)

  * •

Gait Information Image (GII) (?) incorporates Hanman-Anirban entropy function
(?)

Instead of coming up with new types of templates, it would be wiser to take
only a subset of the gait template that is resilient to covariate factors. ?
were among the first to propose this model. They formulated a single mask
through the ranking of pixel features using the Random Forests classifier.
Their panoramic gait recognition (PGR) algorithm used pose estimation for view
prediction. ? designed a method named view-invariant multiscale gait
recognition (VI-MGR) which applied Shannon’s entropy function to only the
lower limb region of the GEI. The sub-region selection was later modified by ?
by automating this segmentation procedure with a process known as group lasso
of motion (GLM). Their approach to the problem has shown significant
improvement in the covariate recognition accuracy.

Recently, the success of deep learning in the fields of image and activity
recognition has motivated its application to gait recognition. The basic
feature representations of these methods are also gait templates. ? employed a
CNN (Convolutional Neural Network) to define the degree of dissimilarity
between two GEI templates, i.e., a probe GEI and a gallery GEI. Deep learning
requires a lot of training samples, so they used the largest gait dataset
publicly available which is the OU-ISIR-LP dataset. This method, named GEINet,
produced an accuracy of 95% for same angle and 80-95% for cross angle
performance. ? proposed the DeepGait design to outperform GEINet using the VGG
(Visual Geometry Group, ?) deep convolution model and Joint Bayesian model for
view invariance. Using the same dataset, the DeepGait achieved gait
identification rates of up to 98% and its cross-view identification accuracy
range from 88 to 98%. A more thorough experimentation using CNN was conducted
by ? on both OU-ISIR-LP and CASIA-B datasets. They used three different CNN
configurations for gait identification. Their experimentation revealed that
the ensemble of the networks with GEI and additional temporal features gave
the best accuracy. Though their cross-view normal recognition performance
became the state-of-the-art, their accuracy for covariate factors are
relatively low. The average CCRs with SetB ranged from 80 to 90%, and SetC
ranged from 60 to 75 across view angles.

As attractive as deep learning sounds, its integration with legacy camera
networks is not so simple due to its computational complexity. Moreover, the
experiments of ? and ? revealed that it is possible to attain close to ideal
gait recognition accuracy without the need for a deep learning framework.

####  9.3 Model-free Non-template Methods

Though found to be efficient in practice, not all silhouette-based methods in
literature involve the production of a template image. Notable non-template
methods are shown in Table LABEL:tab:non-template. ? have claimed to have
attained recognition rates above 75% by running experiments on 114 subjects of
the SOTON video gait dataset. Their method monitors the temporal changes in
the areas of the clipped gait window segmented by masked sectors. Using these
time-varying area metrics, they formulate a feature vector for recognition.

In the work of ?, accounted the average distance from the centre of the
silhouette. Each silhouette was represented as a feature vector which was
composed of a sequence of angular transforms made on segmented angular slices
$\Delta\theta$. In the USF gait challenge dataset, a period in the gallery
(reference) is not equal to that of the probe sequence (test). Hence, a
technique called linear time normalization was utilized to make the feature
matrix of each probe and gallery sequence comparable by compensating for the
difference in the sequence length. The same method was used in their next
paper (?) but with a different feature extraction technique. In this work, the
body as depicted in the silhouette was segmented into body components. The
components were ranked with according to their proportional relevance during
the comparison operation.

? promoted the fusion of face and gait biometrics. The exemplars, in this
case, are obtained from specified stances analogous to the formally depicted
gait cycle. These stance frames were modelled using a population EigenStance-
HMM; a method that was extended from their previous technique, population HMM
(?). A given gait silhouette sequence can be segregated into these stance
models by $k$-means clustering with Tanimoto distance as the distance measure.
A cyclic Bakis variant of HMM was modelled for the gait recognition. This gait
recognition algorithm was used in combination of face recognition using
Elastic Bunch Graph Matching (EBGM) based on Gabor features to attain a much
higher performance over the Gait Challenge baseline algorithm when compared
using the toughest sets of the USF Gait Challenge dataset.

A gait recognition that analyses both shape and motion was proposed by ?. The
gait period here was depicted as ten phases. Spatiotemporal shape features are
obtained from these phases in the form of Fourier descriptors. The silhouette
contour at each step was segmented by fitting ellipses. The similarity was
then calculated by utilizing Bhattacharyya distance between the ellipse
parameters taken as features. DTW was applied to compare leading knee rotation
with relevance to arm swing pattern over a gait cycle. DTW was used to
counteract the effects of walking speed, clothing, shadows, and hair styles.

S.No. | Year | Study | Technique | Plane | Dataset
---|---|---|---|---|---
[late after line= |  |  |  |  |
|  |  |  |  |

####  9.4 Model-based Techniques

While basic spatiotemporal methods give cues as to how the body position vary
with accordance to time as a whole, it would be much more accurate to do a
spatiotemporal analysis on all articulation points separately. The
implementation of that sort falls under the category of joint trajectory or
model-based methods. That is, the trajectory of each joint is tracked live and
analysed as individual components; efforts are made to model the human
structure accurately. Table LABEL:tab:model-based list some of the model-based
gait recognition literature.

Bulb markers can be attached to certain points on the body considered
necessary for gait analysis such as ankles, knees, hands, elbows, shoulders
and torso. When observed from a camera with a low exposure, only the bulb
illumination can be perceived. This method facilitates an easier and a more
accurate analysis of gait through computer vision. ? implemented this by
strapping 18 human subjects with 16 markers at appropriate locations and
projected their gait at a sagittal angle. The sensors were able to reconstruct
a mobile skeletal structure recovered from the joints. From this data, they
were able to assess the articulation points over time. The variance in time
was normalized by applying DTW, and the recognition was based on the nearest
neighbour algorithm to produce a modest recognition rate of 73%.

A study by ? was conducted to extract gait-related parameters from all three
planes – frontal, transverse and sagittal – from surveillance cameras. 11
human subjects participated in the experiment and involved the use of 11 bulb
markers fitted to the necessary points to each subject. They were able to
observe various parameters from step length, cycle time and cadence to joint
angles and spatial positioning. After a simple analysis of variance (ANOVA) on
the gait parameters extracted, it seemed like the foot angle exhibited the
most variance and then the time average hip joint angles followed by

S.No. | Year | Study | Type | Technique | Plane | Dataset
---|---|---|---|---|---|---
[late after line= |  |  |  |  |  |
|  |  |  |  |  |

Model-based features can either be extracted through 3D cameras with the
subject fitted with IR or bulb markers, articulation points inferred from
silhouettes, or depth information from Microsoft Kinect.

the step length. Hence these are considered to be the best parameters to be
used for recognition. However, their research concluded that the gait analysis
cannot be used for identification at that time (the year 2002).

? later produced a study on the recognition of gait in different speeds of
walking, but this time, they resorted to a more comfortable reflective suit
for the articulation point signal extraction. The experimentation is described
in ?. A 12-camera VICON MoCap system was used for the 3D motion analysis. The
result based on 15 human subjects concluded that a positive linear correlation
could be observed between cadence and speed, and a negative exponential
correlation could be observed between stride time and speed.

The methods described so far require the use of complicated hardware for
better accuracy. They either require reflectors, bulb markers, or magneto
sensors to be fitted on to the points of interest of the human subject to
gather the point-light information during his/her gait. Due to their nature,
these methods are not practically feasible for a biometric application. It is
to note that the concept of being an ‘unobtrusive’ means of biometrics is lost
here.

Not all model-based techniques, however, are impractical in application. ?
show that it is possible to extract a five-link biped human model from a two-
dimensional video feed to produce a model-based gait recognition system. The
Metropolis-Hastings algorithm (?) was used for the feature extraction. An
innovative yet simple method for locating the articulation points of the lower
limb joints was implemented by ?. By making smart estimates on where to
initialize the points, the point-light data for the hip, both knees and ankles
were extracted. This method was able to extract these points over multiple
views as provided by the CASIA-B video database. By recording the temporal
changes of these points, a profile recorded could be recognized with the help
of the $k$NN algorithm.

A standard video feed would provide a two-dimensional data for processing.
When added with depth-based information, more accurate conclusions can be
drawn to aid recognition. Microsoft Kinect provides this functionality. The
Kinect was used by ? and ? by facilitating three-dimensional data flow for a
more natural and efficient biometric gait recognition. Methods using the
Kinect were found to be more successful than using silhouettes alone for
model-based analysis.

####  9.5 Covariate Factors

Covariate factors are intra-class variations that inhibit the effectiveness of
biometric gait recognition by confounding the features that can be observed
(?). The following briefly explains each factor with relevant research on how
they deviate the parameters of gait.

Walking surface. Studies show that when the surface is unstable such as a wet
surface, the walking speed, toe-in angle and step length are significantly
reduced to retain control over balance (?). However, when walking on other
irregular surfaces like grass, foam or studded with small obstacles, walking
speed can be maintained with a variable cadence and a longer stride length
(?). Even two regular surfaces such as vinyl and carpet have a significant
difference (?). In slippery surfaces, reductions can also be observed in
stance duration, load supporting foot, normalized stride length (?).

Footwear. When the footwear is considerably different, so is the gait of the
individual. This aspect especially concerns high heel users. To maintain their
stability, people wearing high heels require more control over their centre of
mass (?) and tend to have longer double support times (?). Recent studies show
that habitually shod walkers and habitual barefoot walkers exhibited a
significant gait difference when switching their footwear in terms of their
stride length and cadence (?).

Injury. When any portion of the lumbar region or lower limb is injured, the
individual naturally adopt an antalgic gait. The individual walks in a way so
as to avoid the pain caused by the injury. This style of walking restricts the
range of motion of the associated limb leading to a deviation in the usual
gait proportional to the magnitude of the injury. In-depth research on how an
injury can affect gait was provided by ?.

Muscle development. The development of muscles gives a different range of
control over the parts of the body that affect gait. The sheer mass of the
developed muscles alters the centre of mass at the associated mobile limb as
well as the body itself. The shift depends on the difference in mass. The
change in the centre of mass can modify the inclination of pressure required
for proper stability (?). An extensive study on the correlation between muscle
mass with gait performance (?) proves that muscle mass directly relates to
gait speed, especially in the case of geriatric patients.

Fatigue. When the individual is subjected to fatigue, the stability of the
concerned gait decreases while a noticeable increase in variability of gait is
exhibited (?). The time taken for the recovery towards normal gait depends on
the extent of exertion applied to the individual so as to get to a fatigued
state and the individual’s stamina. This aspect was observed from Ashley
Putnam’s thesis (?) in which a study with army cadet treadmill protocol was
conducted to analyse the effect of exhaustion on gait mechanics and possible
injury. Cadets ran till exhaustion and had their gaits observed. The resulting
gait had both inter- and intra-variable vertical stiffness in the lower limbs.

Training. When the individual is subjected to some form of physical training,
it is possible that her/his gait is also subjected to change. This change can
be evident as a result of military training, prolonged load condition,
prolonged use of particular footwear and athletic training.

Extrinsic control. Humans have an ability to control their gait to an extent
so as to differ from their usual gait. A person can mince walk, and depending
on how self-aware the individual may be she/he can walk with a swagger or
strut, a brisk walk or tip-toe. Another matter to note is the level of
awareness the individual has of his/her surroundings. Hence, individuals tend
to alter their gait to the extent that is determined by their self-control.
This concept explains how members of the army can synchronize gait during a
march.

Intrinsic control. Some elements can control a person’s gait in such a way
that the individual is sometimes unaware of the change that takes place. The
best example of this case is the emotional response or mood of the individual:
state of happiness, sorrow, anger or any other emotion strong enough to make
an impact on one’s gait. The variability can range from subtle to significant
and can vary from one person to another. Related studies are provided in ??
and ?.

Age. Although the factor may not contribute to change in gait over a short
period, it certainly does influence gait to a large extent. Ageing, in
general, causes musculoskeletal and neuromuscular losses. To compensate for
these losses, the individual makes certain adjustments which can be observed
in the individual’s gait (?).

Clothing. While the change of clothing does not necessarily modify the gait
for slight differences in weight, it might, however, show changes in the
associated silhouettes. This change would affect a major portion of gait
recognition algorithms that depend on the spatial configuration of
silhouettes. However, a greater change in the weight of the clothing, such as
a winter suit, has a higher probability of affecting the gait itself.

Load. The effect of load can significantly influence gait. In a loaded
condition such as wearing a backpack, the individual is subjected to a higher
weight in addition to his/her body weight. The foot exerts higher pressure
during plantarflexion to regulate locomotion generating a greater ground
reaction force than the unloaded condition (?). Apart from the pressure
applied, the body must cope with the change in balance for a stable gait (?).
The load can also be asymmetrical, such as a wearing a handbag, cross-bag or
shoulder bag, or carrying a suitcase. In this case, a greater difference in
the pelvic rotation is observed (?). The body shifts the pelvic movement so as
to counteract the imbalance caused by the load.

####  9.6 View Invariance

In addition to clothing and carrying conditions, the view angle is found to be
the most important covariate factor that affects gait recognition performance
(????). The range of features that can be extracted can wary widely between
the angles of observation. There are essentially two types of view-invariant
gait recognition models: view transformation model (VTM) and view-preserving
model (VPM).

VTMs (???) transform the probe sequence’s angle to match with that of the
gallery sequence. The VTM methods may differ in the measures used to gauge the
transformation accuracy (?). However, a significant level of error is
inevitable in VTM-based gait recognition (??).

VPMs consider multiple views as part of the gallery itself. This process
incorporates the view information within the feature set for the extraction of
relevant view-invariant gait features. Various methods can be employed to
facilitate this. Examples include varying width vectors (?), Grassmann
manifold (?), geometric view estimation (?), and spatiotemporal feet
positioning (?). A variant of VPM involves extraction of view-independent
features through multi-view training and then use a single gallery view for
testing (??).

###  10 Datasets

With the outbreak of a substantial amount of algorithms to analyse gait there
also comes a need to compare them. A standard gait dataset would be able to
serve means so as to benchmark such algorithms especially in the case of
biometric application. The categories of datasets cover a wide range based on
the needs of the gait analyst from lightweight datasets to large-scale
databases. An overview of the gait databases described here is given in Table
LABEL:tab:dataset.

Database | Year* | Env | View | Sub. | Seq. | Other covariates
---|---|---|---|---|---|---
[late after line= |  |  |  |  |  |
* The year indicates the year of recent release, not the year of its initial release. SOTON released its first version in 1996 and its first temporal set in 2003. OU-ISIR Treadmill datasets were iteratively developed from 2007-12. Env Legends: F - static flooring, T - treadmill, I - indoor, O - outdoor. n/a: not available; values are either unconfirmed or not published

When DARPA launched the HumanID programme in the year 2000, many institutions
joined and released their first version of the database in the year 2001.
Institutions that released their dataset for public usage include MIT, CMU,
SOTON, Georgia Tech, UMD, USF, and CASIA. Nearly all of the institutions whose
databases are described in this section are associated with this programme.
The programme ended in 2004 due to privacy issues (?), but the databases
compiled as a result is still publicly available from the institutions that
developed them. The choice of the database depends on the use-case. This
thesis involves the design of biometric algorithms that are resilient to
multiple views. The datasets with the most views are CASIA-B and SOTON-
Temporal. However, CASIA-B has 124 subjects while SOTON-Temporal has only 25.
Furthermore, most literature published in view-invariant recognition systems
also use the CASIA-B for their evaluation. Using the same dataset with similar
training and testing conditions would facilitate a fairer comparison. Due to
better balance between covariate factors, the number of subjects and the
number of instances per subject, the CASIA-B dataset is selected as a
benchmark dataset for the experimental validation of the proposed algorithms.

The CASIA datasets are compiled by the Institute of Automation, Chinese
Academy of Sciences. CASIA has developed four public gait databases till date.
The development started in 2001 at the National Laboratory of Pattern
Recognition (NLPR). In addition to the video, the silhouettes for each of
their datasets are freely available for download. With 124 individuals ($s$)
each performing 10 gait instances ($i$) observed in 11 simultaneous view
angles ($v$), the total number of sequences for the CASIA-B dataset would be

$124s\times 10i\times 11v=13640\text{ sequences}$

The 10 instances can be split up as SetA containing 6 normal instances, SetB
with 2 instances carrying a bag, and SetC consisting of 2 instances wearing a
coat. Along with an additional video of bare background, the total number of
videos becomes 15004 (approximately 17.4 GB). The background is mostly plain
light green with two stage markings covering both the wall and the floor (?).
This is the most commonly used multiview database in literature. Figure
LABEL:fig:casia-b shows a depiction of the dataset for a given time instant
from each of the 11 angles.

###  11 Summary

The biometric features of gait can be observed in a variety of ways wherein
inertial sensor-based and computer vision-based are most researched in
literature. Biometrics, in general, can be categorized as either hard or soft.
Gender is found to be the most popular soft biometric in gait analysis; age
estimation is the next most common soft biometric. Hard biometrics consider
the problem of recognition (identification) and authentication (verification).

Biometric feature extraction can be classified as either model-free or model-
based. Model-free approaches analyse the gait structure as a whole while
model-based methods analyse joint-trajectories (usually in three-dimensions).
Model-free methods can be further classified as either template-based or non-
template. Template-based methods compile a single image called a gait template
while non-template blurs between model-free and model-based methods in 2D
space. Template-based methods are more successful than non-template methods in
recent literature and have more practical scope than model-based methods.

Covariate factors increase the difficulty of gait recognition. Gait datasets
mostly focus on difference in clothing conditions, carrying conditions,
footwear and view angles. CASIA-B is currently the most commonly used multi-
view gait dataset.

##  CHAPTER 3 GENDER CLASSIFICATION THROUGH POSE-BASED VOTING

###  12 Introduction

Gender classification has several applications. In surveillance systems, the
prior identification of gender can substantially reduce the search space
required to find a specific person among many in a video feed (?). The
classification could help analyse the relationship between gender and product
preference in customer-market studies (?). The knowledge of one’s gender can
provide natural human-computer interaction in robotics (?). Many methods were
suggested to investigate the spatiotemporal patterns of the gait sequence to
infer the gender of the associated subject (?). ? claim that an effective
gender identification system can exceed the accuracy of human-based
observation.

In this chapter, we delve into a gait-based gender recognition algorithm that
employs a technique that delineates the gait instance as a sequence of poses
or frames. The system predicts the gender of each frame that constitutes the
gait instance and decides whether the subject is male or female based on
majority voting. This method is called pose-based voting (PBV) as illustrated
in Figure LABEL:fig:pbv. The intuition behind this approach relies on the fact
that humans tend to assume certain poses at each part of the gait cycle that
would reflect their innate gender. The classifier tries to make a prediction
based on the aggregate of the poses it records. Thus, the temporal element of
gait is not processed by the classifier but just the spatial features. By
removing the reliance on temporal features, the system would effectively deal
with partially occluded gait cycles and temporary occlusion.

Two feature extraction techniques are investigated in this method: elliptic
Fourier descriptors (EFD) (?), and consolidate vector of row-column summation
(RCS). The EFD is supported by its rotation invariance property while the RCS
is more simple yet effective to implement. Both of these methods are
extensively studied with the widely used CASIA-B dataset, and the results are
compared to the state-of-the-art approaches through recommended test
conditions.

###  13 State of the Art

The gait instance is given as input to the system in the same manner as it is
done for gait recognition and authentication. That is, the video feed is
converted to frames through which silhouettes are extracted. The silhouettes
are pure binary colour coded with black and white. The usual norm is then to
extract only a single gait cycle from the entire sequence and make the
prediction based on the patterns observed from that period.

? developed the current state of the art in gender classification based on
gait. The process involves dividing the normalized silhouettes to $2\times 2$
and $4\times 4$ cells and the fitting of ellipses (as in ?). The extracted
features are then modelled using MCRF along with a combination of complicated
operations including the Karhunen-Loeve transformation (?) and Xie and Beni’s
(XB) index (?). Their implementation was tested on the CASIA-B dataset to
achieve an accuracy of 98.39%.

? implemented a method to detect gender with no more than a single video frame
through a CNN. However, CNNs are known to be computationally expensive in
operation. As the application in focus concerns gait, more frames are
available per instance. Analysing each frame of a gait sequence with a simpler
algorithm can yield an equal or better result than taking only a single frame
with a complex algorithm.

Most existing implementations adopt the SVM classifier for the gender
recognition problem especially when a gait template like the GEI is used (see
???). Some relatively new methods include the mixed continuous random field
(MCRF) by ?, sparse representation-based classification by ?, and CNN by ?.

###  14 Method

The schematic flow of the proposed system is illustrated in Figure
LABEL:fig:pbv-arch. The videos containing the gait instances are delineated
into frames through which silhouettes are obtained and binarized. The
forthcoming sections discuss two different feature extraction techniques, EFD
and RCS. Instead of taking a spatiotemporal template of the entire gait
sequence, each silhouette is considered to be a separate instance to the LDA-
Bayes classifier. The classifier makes predictions on each frame of the test
video. The statistical mode (majority vote) of the aggregate predictions
become the final prediction of the system.

####  14.1 Elliptic Fourier Descriptors

Chain encoding is a technique that is used to provide a compressed
representation of image contours (?). ? presented that elliptic properties of
Fourier coefficients can be exploited to characterize a closed contour. The
resulting features are called elliptic Fourier descriptors. The major
advantage of this method is that the contours that are reconstructed are said
to be independent of the starting point of the contour. Furthermore, the EFD
has a rotation and scale invariance property that describes the image with the
same set of coefficients irrespective of the orientation and size of the
object in two-dimensional space.

Figure LABEL:fig:efd depicts the reconstruction of the EFD in different levels
of harmonics extracted from a human silhouette. The real silhouette is
superimposed with the reconstructed $N$-harmonic representations of the chain
code for reference.

The chain code V of a contour consisting of K points is

$V=a_{1}a_{2}a_{3}...a_{K}$

where $a_{i}$ can take the values from 0 through 7 representing the 8 possible
pixel directions taken anticlockwise (although the contour is traversed
clockwise). The required time to travel across the link $a_{i}$ is

$\Delta t_{i}=1+\Bigg{(}\frac{\sqrt{2}-1}{2}\Bigg{)}(1-(-1)^{a_{i}})$

The required time to plot up to $p$ chain links is given by

$t_{p}=\sum_{i=1}^{p}\Delta t_{i}$

and $T=t_{K}$ is considered to be the chain’s basic period.

The differences in the projections of $x$ and $y$ are

$\displaystyle\Delta x_{i}$
$\displaystyle=\text{sgn}(6-a_{i})\text{sgn}(2-a_{i}),$ $\displaystyle\Delta
y_{i}$ $\displaystyle=\text{sgn}(4-a_{i})\text{sgn}(a_{i})$

where sgn is a sign function given by

$\text{sgn}(w)=\begin{cases}1&\text{if}\ w>0,\\\ 0&\text{if}\ w=0,\\\
-1&\text{if}\ w<0.\end{cases}$

The Fourier series for the $x$ and $y$ projections can be expanded as follows.

$\displaystyle x(t)$ $\displaystyle=A_{0}+\sum_{n=1}^{N}a_{n}\cos\frac{2n\pi
t}{T}+b_{n}\sin\frac{2n\pi t}{T}$ (1) $\displaystyle y(t)$
$\displaystyle=C_{0}+\sum_{n=1}^{N}c_{n}\cos\frac{2n\pi
t}{T}+d_{n}\sin\frac{2n\pi t}{T}$ (2)

where

$\displaystyle a_{n}$
$\displaystyle=\frac{T}{2n^{2}\pi^{2}}\sum_{p=1}^{K}\frac{\Delta x_{p}}{\Delta
t_{p}}\Bigg{[}\cos\frac{2n\pi t_{p}}{T}-\cos\frac{2n\pi t_{p-1}}{T}\Bigg{]}$
$\displaystyle b_{n}$
$\displaystyle=\frac{T}{2n^{2}\pi^{2}}\sum_{p=1}^{K}\frac{\Delta x_{p}}{\Delta
t_{p}}\Bigg{[}\sin\frac{2n\pi t_{p}}{T}-\sin\frac{2n\pi t_{p-1}}{T}\Bigg{]}$
$\displaystyle c_{n}$
$\displaystyle=\frac{T}{2n^{2}\pi^{2}}\sum_{p=1}^{K}\frac{\Delta y_{p}}{\Delta
t_{p}}\Bigg{[}\cos\frac{2n\pi t_{p}}{T}-\cos\frac{2n\pi t_{p-1}}{T}\Bigg{]}$
$\displaystyle d_{n}$
$\displaystyle=\frac{T}{2n^{2}\pi^{2}}\sum_{p=1}^{K}\frac{\Delta y_{p}}{\Delta
t_{p}}\Bigg{[}\sin\frac{2n\pi t_{p}}{T}-\sin\frac{2n\pi t_{p-1}}{T}\Bigg{]}$

Equations LABEL:pbv-eq:1 and LABEL:pbv-eq:2 are expanded up to $N$
harmonics222The derivation of the elliptic Fourier descriptors for closed
contours can be found in detail in the work by ?. In the generalized
equations, $N=\infty$. The value of $N$ determines the complexity of fit. The
DC components $A_{0}$ and $C_{0}$ are given by

$\displaystyle A_{0}$ $\displaystyle=\frac{1}{T}\sum_{p=1}^{K}\frac{\Delta
x_{p}}{2\Delta
t_{p}}\bigg{(}t_{p}^{2}-t_{p-1}^{2}\bigg{)}+\xi_{p}\big{(}t_{p}-t_{p-1}\big{)},$
$\displaystyle C_{0}$ $\displaystyle=\frac{1}{T}\sum_{p=1}^{K}\frac{\Delta
y_{p}}{2\Delta
t_{p}}\bigg{(}t_{p}^{2}-t_{p-1}^{2}\bigg{)}+\delta_{p}\big{(}t_{p}-t_{p-1}\big{)}$

where

$\displaystyle\xi_{p}$ $\displaystyle=\sum_{j=1}^{p-1}\Delta
x_{j}-\frac{\Delta x_{p}}{\Delta t_{p}}\sum_{j=1}^{p-1}\Delta
t_{j},\qquad\xi_{1}=0,$ $\displaystyle\delta_{p}$
$\displaystyle=\sum_{j=1}^{p-1}\Delta y_{j}-\frac{\Delta y_{p}}{\Delta
t_{p}}\sum_{j=1}^{p-1}\Delta t_{j},\qquad\delta_{1}=0$

For each harmonic level, four more coefficients are generated, $a_{n}$,
$b_{n}$, $c_{n}$, and $d_{n}$. These parameters are used as features for the
training. Thus, the EFD provides a total of $4N+2$ features per sample for a
fixed value of $N$.

####  14.2 Row-Column Summation

One way to extract both horizontal and vertical features of a silhouette would
be to take the count of white pixels in each of $x$ and $y$ axes. An
illustration of this is given in Figure LABEL:fig:rcs. The figure depicts the
number of white pixels in each row $r$ and column $c$. The properties of the
binarized silhouette are exploited to facilitate the vectorized version of
this operation.

Let $r_{i}$ denote the count of white pixels at row $i$, and $c_{j}$ be number
of white pixels in column $j$. The row and column vectors are given by

$\vec{r}=\big{(}r_{1},r_{2},r_{3},...,r_{n}\big{)}\qquad\text{and}\qquad\vec{c}=\big{(}c_{1},c_{2},c_{3},...,c_{m}\big{)}$

where $n$ and $m$ are taken as 240 in this implementation. The number of white
pixels of a row of a binarized image can be taken as the sum of all values in
that row. Thus, $\vec{r}$ contains the row-wise sum of the silhouette.
Similarly, $\vec{c}$ stores the column-wise sum. Together, these vectors
describe the pose of the silhouette in the image. The concatenation of both
these vectors, $\vec{r}~{}^{\frown}\vec{c}$, can be utilized as the feature
set of the image and are called the row-column summation or RCS.

Note that each silhouette is taken to be a square rather than a rectangle with
the height greater than the width. This size makes sure that the entire space
of movement freedom is covered by the subject. According to the anatomy of an
average human, the arm-span is approximately equal to the height (?). This
notion is also adopted by ? when generating a public dataset of GEI from the
CASIA-B dataset.

####  14.3 Pose-Based Voting

The initial step of nearly any gait-based biometric system is silhouette
extraction which is achieved through background subtraction. In cases where
the environment is highly dynamic, algorithms like the one suggested by ? can
be employed. They use a modified version of the colour difference histogram
through fuzzy c-means clustering to eliminate the excessive noise in the
background. As the dataset in this study (CASIA-B) is composed only of a
static background, a slightly primitive foreground detection routine is
applied. Further description of the dataset is provided in Section
LABEL:sec:datasets. In this step, the difference between the background frame
$b$ and the current frame at instance $t$ would give the silhouette of the
moving object at frame $t$ (?). A standard camera can introduce noise between
successive frames even in a static environment. Hence, a Gaussian blur
operation applied before the difference is computed to eliminate the subtle
noise.

Preconditions:

vList: list of gait video instances

gLabels: corresponding vector of gender labels

procedure PBVtrain($\mathit{vList},\mathit{gLabels}$)

$X\leftarrow\text{EmptyMatrix}$

$y\leftarrow\text{EmptyVector}$

for each index $i$ in vList do

$\mathit{silhouettes}\leftarrow\text{BackgroundSubtract}(\mathit{vList}[i])$

$\mathit{normSilhouettes}\leftarrow\text{Normalize}(\mathit{silhouettes})$

$\mathit{sampleSet}\leftarrow\text{ExtractFeatures}(\mathit{normSilhouettes})$

$\mathit{label}\leftarrow\mathit{gLabels}[i]$

for each sample in samplesSet do

Append sample to $X$

Append label to $y$

end for

end for

$L\leftarrow\text{LDAFit}(X,y)$

$X\leftarrow L.\text{transform}(X)$

$M\leftarrow\text{MGBayesFit}(X,y)$

$\mathit{model}\leftarrow[L,M]$

return model

end procedure

Each silhouette extracted is clipped and scaled to standard proportions. This
procedure is considered to be normalization (?). Each sample contains a
complete human silhouette with the dimensions $240\times 240$ pixels and
binarized. A pseudocode of the training process is given in Algorithm
LABEL:alg:train. The feature vectors are extracted from each silhouette and
are labelled to form the training set. This study compares the effectiveness
of both EFD and RCS feature extraction methods. The feature space is then
transformed through Linear Discriminant Analysis (LDA) and trained through a
multivariate Gaussian Bayes’ rule (MGBayesFit). The model thus produced is
composed of a set of parameters that are used to make the gender prediction
during the testing phase. These include the transformation vectors from LDA
and the probabilities from Bayes’ rule.

The LDA transformation maximizes inter-class distance while minimizing the
intra-class distance (?). LDA is more commonly used in gait recognition
algorithms along with principal component analysis (?). As the number of
features is already sufficiently small for processing, an unclassified feature
reduction like PCA would not be necessary.

The transformed data is passed through a multivariate Gaussian model of Bayes’
rule (?) for classification. This is not a naïve Bayes implementation where a
product of multiple univariate normal distributions are taken as the
likelihood. The Bayes classifier is given by

$\Pr(y\mkern 1.7mu{=}\mkern 1.7muk\mid x)=\frac{\Pr(x\mid y\mkern
1.7mu{=}\mkern 1.7muk)\cdot\Pr(y\mkern 1.7mu{=}\mkern 1.7muk)}{\Pr(x)}$

where the likelihood term, $\Pr(x\mid y\mkern 1.7mu{=}\mkern 1.7muk)$, is
calculated with $X$ distributed as a multivariate normal distribution,
$X\sim\mathcal{N}(\mu_{k},\Sigma_{k})$.

Preconditions:

instance: single gait video instance

model: trained PBV model

procedure PBVpredict($\mathit{instance},\textit{model}$)

$\mathit{silhouettes}\leftarrow\text{BackgroundSubtract}(instance)$

$\mathit{normSilhouettes}\leftarrow\text{Normalize}(\mathit{silhouettes})$

$\mathit{sampleSet}\leftarrow\text{ExtractFeatures}(\mathit{normSilhouettes})$

$[L,M]\leftarrow\mathit{model}$

$\mathit{predictions}\leftarrow\text{EmptyList}$

for each sample in samplesSet do

$x_{t}\leftarrow L.\text{transform}(sample)$

$p\leftarrow\text{MGBayesPredict}(x_{t})$

Append $p$ to predictions

end for

$\mathit{prediction}\leftarrow\text{ModeStat}(\mathit{predictions})$

return prediction

end procedure

The salient part of the algorithm lies in the testing routine (Algorithm
LABEL:alg:test). The preprocessing step is analogous to the training sequence
where each silhouette is normalized and has its feature extracted (with either
EFD or RCS). Once each silhouette is classified, the statistical mode of the
separate predictions become the final prediction of the system. This event can
be thought of as each prediction trying to elect for a specific gender which
can assume the pose it depicts and the final selection being the majority vote
of the individual silhouettes.

A specific experiment is conducted to test the handling of occlusion which
leads to the partial observation of gait cycles. In this setup, a complete
gait cycle of each training instance is given to both the existing and
proposed PBV method for learning. Then, during testing, partial gait cycles
were given ranging from 10% to 100% of the gait cycle pertaining to the test
gait instance. The existing method would attempt to compile a gait template
and then process it for classification. The PBV method, on the other hand,
judges the gender based on the votes currently accumulated.

###  15 Experimentation & Evaluation

The Python programming framework is used for all simulations in this thesis.
Classification and feature transformation routines are applied with the help
of the Scikit-Learn package (?). All graphs were plot using the Matplotlib
(?).

####  15.1 Dataset Configuration

All experiments have been conducted using the video instances from the CASIA-B
gait database (?) as it is currently the most common database to evaluate
gender recognition. It holds videos containing the gait of 93 males and 31
females in three different covariate factors: normal, carrying a bag, and
wearing a coat. The dataset and testing conditions are formulated exactly as
prescribed by ? with 31 female subjects and 31 randomly selected male subjects
following a 31-fold cross validation evaluation. All six instances of normal
walk, without bag or coat, observed at the sagittal angle were utilized for
the selected subjects. The total number of video instances hence used are
$62\times 6=372$. As this is a two-class categorization problem, the equal
proportion of both classes avoids bias in the classification results. A
31-fold cross-validation testing scheme is used for evaluation. Each fold is
disjoint and contains all videos of one male and one female subject.

####  15.2 Number of Harmonics for EFD

The code for EFD was extended from Alessandro Ferrari’s implementation (?).
The complexity of the Fourier descriptors is determined by the number of
harmonics expanded, $N$. This value is empirically chosen in accordance with
the effect it has on the system accuracy by conducting a standard testing-
training split with varying values of $N$. The training test composes of 25
folds while the remaining 6 folds form the testing set. This configuration is
similar to the test conditions used by ?. The outcome is presented in Figure
LABEL:fig:nselect. The classifier accuracy is the CCR of the LDA-Bayes
classifier, i.e., the proportion of silhouettes correctly classified to the
total number of silhouettes in the test set. The system accuracy is the CCR of
the PBV framework, i.e., the fraction of video instances correctly classified.
One can clearly visualize the impact of the majority vote on the system. At
the point where $N=8$, the system accuracy stabilizes while the classifier’s
CCR slowly increases. $N$ is set to 12 to safely avoid probable bias in the
selection of the training and testing folds.

####  15.3 Performance in Partial Gait Cycles

Gait is composed of multiple phases which occur as repeated sequence of
events. A gait cycle is a period between the occurrence of any two of the same
gait event in succession (?). Almost all of the algorithms reported till date
require the silhouettes of a complete gait cycle for an optimal CCR. The
proposed framework would facilitate adequate performance even at partially
observed gait cycles. A simple technique is employed to detect gait cycles to
test this hypothesis.

? estimated gait cycles by tracking the count of white pixels in the lower
half the silhouettes. The pixel count would produce an oscillation signal when
plotted. The silhouettes between any three troughs or crests of the signal
would yield a complete gait cycle. However, the signal extracted could
potentially contain noise which made the detection process difficult in
certain cases. ? suggested using variation in locally linear embedding (LLE)
coefficient to overcome this issue. On the contrary, to provide a simpler yet
effective approach for gait period estimation, a variant of the method used by
? in conjunction with smoothing using the Savitzky-Golay filter (?) is
applied.

In this implementation, the white pixel count is taken below the half-way
point from the probable knee region till the bottom end of the silhouette to
minimize the noise that could occur. The oscillation produced can be observed
as shown in Figure LABEL:fig:UnprocOsc. Savitzky-Golay filter is applied
(SciPy implementation by ?) to smooth the distortions in the signal. This
technique keeps track of the adjacent points to approximate the position of
the current point that would fit with the least error according to the least
squares principle. The resulting curve would look almost like a sine curve as
shown in Figure LABEL:fig:SmoothedOsc. The silhouettes between three
observable troughs are taken to compose a single complete gait cycle.

The intuition behind selecting three troughs is that at the first trough, both
feet are together with one on the floor and the other performing midswing. The
second trough denotes the half-cycle where the latter leg is stable while the
former is at midswing. The full cycle completes at the third trough when all
phases of gait are encompassed.

The GEI is used as a benchmark to represent the characteristic of algorithms
that require a complete cycle (???). The classifier that is based on the GEI
is trained in a similar fashion to those in literature with pixel intensities
as attributes. The features are reduced through LDA and classified with SVM.

The experimental setup involves testing both the PBV and the GEI-based
framework in increasing proportions of the gait cycle from partially observed
to completely observed. Learning is applied as usual with the availability of
the whole of the training video. Then, during testing, partial gait cycles
were given ranging from 10% to 100% of the gait cycle pertaining to the test
gait instance. The GEI-based method would attempt to compile a gait template
with the available silhouettes and then process it for classification while
the PBV follows the voting method as described in Section LABEL:sec:proposed.
The 31-fold cross-validation accuracy is recorded at each step. To this end,
the objective is to prove that when LDA is applied, both SVM and Bayes’ rule
perform similarly. Hence, both SVM and Bayes are also compared in the analysis
with both GEI and PBV methods. SciKit Learn package (?) was used for LDA and
SVM. Moreover, the experiment is repeated for 3 different random combinations
of male subjects (as there are only 31 female subjects) and the final scores
are averaged at each stage.

The plots in Figure LABEL:fig:partial clearly show that the proposed PBV
method outperforms the standard template-based method. The difference is more
pronounced at lower proportions of observed gait periods than at the complete
cycle. Also, it can be observed that both SVM and Bayes’ rule perform almost
equally with both GEI and PBV frameworks as both are preceded with LDA
transformation. Within PBV, the RCS features provide a much more robust CCR
compared to the EFD feature set.

####  15.4 Overall Performance

Method |  | CCR(%) *
---|---|---
?? |  | 93.28
? |  | 95.97
? |  | 96.66
? |  | 98.39
? |  | 98.00
Proposed PBV-EFD |  | 94.89
Proposed PBV-RCS |  | 100.00

* 31-fold cross validation with 31 males and 31 females.

The performance of the overall system along with that of the existing
implementations are compared in Table LABEL:tab:result. The proposed PBV
system gives an ideal result with the RCS feature scheme and a moderately good
result with the EFD feature set. The RCS relies only on the spatial features
as it incorporates both the body shape and pose of the subject that enables it
to characterize the gender of the subject under observation. Note that the PBV
accuracy that is tabulated is greater than that in Figure LABEL:fig:partial.
This phenomenon is because, during the overall test, the system considers all
possible silhouettes of the video instance; not just that of a single gait
cycle. Each video contains slightly more than a single gait cycle while some
may even contain up to two. The more silhouettes that are extracted, the more
electors there would be participating in the voting scheme leading to a higher
probability that the outcome would be correct.

The harmonics were recomputed for higher orders up to 40 and tested again to
affirm the performance of EFD, yet the same accuracy was obtained. Though the
EFD may seem a little inaccurate compared to the other state of the art, one
can hypothesize that its full potential can be realized through its rotation
and scale invariance property. Unlike the RCS and the other methods in the
literature, the EFD is said to generate the same set of descriptors for any
orientation of the image. This property would be useful in surveillance
systems where the camera orientation can vary, causing the silhouettes
extracted to seem rotated in 2-D space.

###  16 Summary

The state-of-the-art methods for gait-based gender recognition extract
spatiotemporal features from the gait sequence and project the entire gait
sequence as a single instance. On the contrary, the proposed PBV projects each
frame of the sequence as an instance. The PBV scheme aggregates the
predictions of each silhouette of a single gait sequence based on a majority
vote to give the final gender prediction.

Existing implementation of gait-based gender recognition show promising
results but do not consider the problem of temporary occlusion. The proposed
PBV approach yields an ideal gender recognition accuracy and is robust to
temporary occlusion of sequences of a gait video instance.

Two distinct feature representations were used in PBV, namely, EFD and RCS.
After a pass through LDA, Bayes’ rule can provide close to equal performance
to SVM. The results of both existing models and proposed model were compared
based on the CASIA-B gait dataset to show that PBV with RCS outperforms the
state-of-the-art methods.

##  CHAPTER 4 GAIT RECOGNITION THROUGH GENETIC TEMPLATE SEGMENTATION

###  17 Introduction

Gait recognition is the process of identifying a person through only their
gait patterns. Although the term should be technically gait identification,
the term gait recognition is much more pronounced in literature. As discussed
in Section LABEL:sec:hard-biometrics, there are both model-free and model-
based methods to do this task. Model-based methods (????) attempt to track the
dynamic changes in the articulation points during gait and hence require
intense computational effort. The model-free approach is the most preferred as
it captures the gait patterns without this requirement.

Recent methods adopt the use of gait templates as they are simple to implement
yet highly effective in practice with the GEI (?) being the most popular one
among them. The GEI quickly became the most successful method for multi-view
gait recognition. Its major drawback was its weakness to covariates like
clothing and load carrying which could adversely affect its performance. Many
similar methods followed aiming to mitigate this weakness with their
implementation of gait templates. Notable templates include the AEI (?) and
the GEnI (?). With a slight trade-off in normal walk gait recognition, these
new templates were able to produce a better recognition accuracy over the
clothing and carrying covariates in gait.

In this chapter, a VPM is devised that can be applied to any gait template for
gait identification. To refine the templates themselves, a method is proposed
to automate its segmentation process with the use of the genetic algorithm
(GA). These segments depict the optimal regions of the gait template that can
be used to obtain the best recognition result at any visually affecting
covariate factor. As a design choice, Bayes’ rule is used instead of the
widely adopted $k$NN.

###  18 State of the Art

Research suggests that using a proportion of the gait template that is least
affected by the covariate factors would improve the overall accuracy of the
gait recognition system. ? formulated a single mask through the ranking of
pixel features using the Random Forests classifier. Their panoramic gait
recognition (PGR) algorithm uses pose estimation for view prediction. ?
designed a VPM named view-invariant multiscale gait recognition (VI-MGR) which
applied Shannon’s entropy function to the lower limb region of the GEI. The
sub-region selection was later modified by ? automating this segmentation
procedure with a process known as group lasso of motion (GLM). Their approach
to the problem has shown significant improvement in the covariate recognition
accuracy.

Though the following implementations do not concern view-invariance or
covariate factors, their aspects add to the motivation of this approach. ?
have shown how incorporating the head and shoulder mean shape (HSMS) along
with the Lucas-Kanade variant of the gait flow image (GFI) (?) greatly
improves recognition accuracy. The genetic algorithm (?) was previously used
by ? to optimize the selection of model-based gait parameters and also by ?
for the selection of superimposed contour features.

###  19 Method

An overview of the method is illustrated in Figure LABEL:fig:gts-arch. The
first step is to extract the gait template (such as the GEI) from the video
that contains the gait sequence. After which the database is split into two
disjoint sets – tuning set and evaluation set. The tuning set is fed to the GA
to formulate the segments for optimal performance. Only those segments are
extracted from the evaluation set to test the final accuracy of the system.

####  19.1 Gait Template Extraction

The model-free template representation of gait composes the feature set for
this study. All gait templates are produced in a similar procedure to the one
given below. Silhouettes in here are obtained through background subtraction
and encoded in grayscale.

  1. 1.

Extract only the silhouettes of the subject during a single gait cycle.

  2. 2.

The silhouettes are center-aligned and scaled to a standard size; 240 x 240 in
this case.

  3. 3.

The standardized silhouettes for a given gait sequence are merged through a
collation process to generate the gait template.

The silhouettes are usually binarized composing only of pure white (1) and
black (0) pixels before collation. The characteristic difference between one
type of template to another lies in the collation process adopted to compile
the template image. The final image that is produced after the collation
contain pixel values varying between 0 to 1. These values are scaled to 0 to
255 for grayscale representation.

If $N$ is the number of frames in the subset, and $B_{t}$ is the normalized
silhouette at frame $t$, then the GEI (?) template of the gait instance is
collated as

$G_{\text{GEI}}(x,y)=\frac{1}{N}\sum_{t=1}^{N}B_{t}(x,y)$

Similarly, the collation operation for the AEI (?) is given as follows
assuming $B(0)=[0]$ (black frame).

$G_{\text{AEI}}(x,y)=\frac{1}{N}\sum_{t=1}^{N}\big{(}B_{t}(x,y)-B_{t-1}(x,y)\big{)}$

The GEnI (?) template is given by

$G_{\text{GEnI}}(x,y)=-z\log z-(1-z)\log(1-z)$

where $z=p_{1}(x,y)$ is the probability of the pixel $(x,y)$ being white in
the given subset of frames from $B_{1}$ through $B_{N}$. Samples of the above
templates are illustrated in Figure LABEL:fig:templates for different
covariate conditions of CASIA-B.

####  19.2 Learning Model

The gait template that is extracted is of the dimensions $240\times 240$ which
yields the total of 57600 pixels. Each of these pixels functions as a feature
for recognition. Feeding an excessively large amount of features directly to a
machine learning algorithm would not be advisable due to following reasons:

  1. (a)

Most of these features may not correlate with the objective function, which in
this case is gait recognition.

  2. (b)

Many of the required features may be linearly correlated and hence are
redundant in the feature set.

  3. (c)

Selecting only the most useful features can considerably speed up the training
time of the model.

Hence the features require reduction before being processed by a machine
learning algorithm. Feature reduction algorithms can either be supervised or
unsupervised. Unsupervised feature reduction algorithms transform the given
feature space of unlabelled instances while the supervised alternative
requires labelled instances for feature reduction.

For this application, the features are preprocessed by Principal Component
Analysis (PCA) followed by a multi-class Linear Discriminant Analysis (LDA).
The Multi-class LDA, also referred to as Multiple Discriminant Analysis (MDA)
(?), is a supervised dimensionality reduction method that would maximize
inter-class distance while minimizing intra-class distance. PCA (?) is an
unsupervised dimensionality reduction algorithm that projects the given
features to feature space that corresponds to the highest variance. The use of
PCA yields a net positive effect on the performance of the classifier
concerning both processing time and accuracy. This type of MDA with PCA is
sometimes referred to as Canonical Discriminant Analysis (CDA) (?).

$k$NN is the standard classifier used for gait recognition. This study
explores Bayes’ rule as an alternative to $k$NN. The performance of these two
classifiers along with a few other compatible classifiers on the GEI template
is depicted in Table LABEL:tab:classifier-comp and graphically illustrated in
Figure LABEL:fig:classifier-comp. The parameters of all the classifiers have
been tuned to obtain the best possible performance. ? provides the detail for
each of the algorithms applied. The Bayes’ rule performs marginally better
than $k$NN which perform equally well as the SVM with linear kernel.

Classifier | Normal | Bag | Coat | Mean | Std
---|---|---|---|---|---
$k$NN | 99.60 | 73.39 | 27.42 | 50.40 | 32.50
Linear SVM | 99.60 | 72.98 | 27.82 | 50.40 | 31.93
MG Bayes | 99.19 | 75.00 | 27.82 | 51.41 | 33.36
Random Forest | 93.95 | 20.97 | 10.89 | 15.93 | 7.13
Neural Net | 97.58 | 55.65 | 25.40 | 40.52 | 21.38
AdaBoost(RF) | 94.76 | 20.97 | 9.27 | 15.12 | 8.27
Naive Bayes | 95.16 | 37.90 | 15.32 | 26.61 | 15.97

Although it is well adapted to many problems, Random Forest (RF) (?) does not
apply well the problem of gait recognition. Due to its potential in remote
sensing (?), the RF’s parameters were optimized and further improved with
AdaBoost333The details of AdaBoosted Random Forest and its optimization is
explained in the paper ?. However, applying AdaBoosted Random Forest does not
make an improvement either. This is because Random Forest is not as much
effective when the number of instances per class is small, especially when the
system contains highly dependent features. In this case, the training set only
contains four video samples for each subject, i.e., four instances per class.
This would mean that the RF does not have enough samples to learn the pattern
well.

From the above discussion, we can arrive at a decision to use the Bayes’ rule
for the classification step of the gait recognition algorithm. Let $x$ be an
instance composed of features $[x_{1},x_{2},x_{3},...,x_{n}]$ and $y$ be the
discrete outcome, then Bayes’ rule is given by

$\Pr(y\mkern 1.7mu{=}\mkern 1.7muk\mid x)=\frac{\Pr(x\mid y\mkern
1.7mu{=}\mkern 1.7muk)\cdot\Pr(y\mkern 1.7mu{=}\mkern 1.7muk)}{\Pr(x)}$

where $\Pr(x\mid y\mkern 1.7mu{=}\mkern 1.7muk)$ is called the likelihood. The
likelihood can be either naïve or multivariate. In the naïve Bayes model, the
likelihood is the product of the probability density functions $f$ of multiple
univariate Gaussian distributions like so

$\Pr(x\mid y\mkern 1.7mu{=}\mkern
1.7muk)=\prod_{i=1}^{n}f(x_{i}\mid\mu_{k}^{(i)},\sigma_{k}^{(i)})$

The complete Bayes’ model, on the contrary, employs multivariate Gaussian
probability density as follows.

$\displaystyle\Pr(x\mid y\mkern 1.7mu{=}\mkern 1.7muk)$