# A Fully Automated System for Sizing Nasal PAP Masks Using Facial Photographs

Benjamin Johnston Student Member, IEEE and Philip de Chazal Senior Member,
IEEE Research supported by ARC grant FT110101098Ben Johnston and Philip de
Chazal are with the Sleep Research Group, Charles Perkins Centre, School of
Electrical and Information Engineering, University of Sydney, Sydney, NSW,
2006, Australia, (phone: +612 911 41528; e-mails: {ben.johnston,
<EMAIL_ADDRESS>
###### Abstract

We present a fully automated system for sizing nasal Positive Airway Pressure
(PAP) masks. The system is comprised of a mix of HOG object detectors as well
as multiple convolutional neural network stages for facial landmark detection.
The models were trained using samples from the publicly available PUT and MUCT
datasets while transfer learning was also employed to improve the performance
of the models on facial photographs of actual PAP mask users. The fully
automated system demonstrated an overall accuracy of 64.71% in correctly
selecting the appropriate mask size and 86.1% accuracy sizing within 1 mask
size.

###### Index Terms:

OSA, PAP, deep learning, machine learning, transfer learning, facial
landmarking, telemedicine

##  I Introduction

Sleep apnoea is a condition which leads to the repetitive cessation of
breathing during sleep and commonly produces symptoms such as poor quality
sleep, excessive daytime sleepiness, headaches and weight gain, as well as
being associated with an elevated risk of other diseases such as
cardiovascular disease and diabetes [1]. The most common form of sleep apnoea
Obstructive Sleep Apnoea (OSA) is caused by either the partial or complete
blockage of the upper airway and can be effectively treated by preventing the
blockage from occurring. Since its inception in 1981 by Sullivan et al. [2],
positive airway pressure, or continuous positive airway pressure (CPAP) has
been the gold standard for the treatment of OSA. PAP therapy is applied
through the use of a specialised medical device which delivers a highly
controlled wave of pressurised air to the upper airway, acting as a pneumatic
splint and preventing the blockage of the pharynx that characterises OSA. One
vital component of the CPAP device is the specialised mask that must be worn
by the patients during sleep. The mask must be able to provide a seal that
maintains the required therapy pressure to ensure airway remains open, despite
any movement of the patient’s head or body or collisions of the mask with the
bed or pillow. The mask must also be sufficiently comfortable to allow a
patient to wear it during hours of sleep, which is particularly important as
while maintenance of therapy pressure is essential for treatment, mask related
side effects have been shown to dissuade patients from actually using their
device [3]. Patients who do not use their device obviously do not receive any
benefit from therapy. Mask related side effects such as red marks, pressure
leakage into the eyes and skin abrasions have been reported in previous
studies [4, 5] with one such study reporting the occurrence of mask leak and
red marks in 48% and 40% of participants respectively [6]. Given that mask
related performance issues are a significant contributor to patient usage
rates and rates of discontinuing therapy [7] it is extremely important the
mask issued to a patient is the most appropriate model and size.

###  I-A Current Mask Sizing Methods

The exact process for selecting, sizing and issuing CPAP masks varies
depending upon regulatory jurisdictions, manufacturers requirements and
clinician’s own professional experience. In general however the process for
selecting the most appropriate mask size can be summarised into one of two
methods: use of a manufacturer supplied sizing template and use of previous
experience and expertise. Almost every manufacturer provides a physical guide
for determining the most appropriate mask size for a patient. These guides
(such as that shown in Figure 1) use the position of patients nares relative
to specific markers as a means of indicating the correct mask size. Thus, the
ESON nasal mask sizes are as follows: Small: 0 - 37mm, Medium: 37 - 41mm,
Large: 41 - 45mm and Too Large $>$ 45mm. While Too Large is not an official
mask size, it is used to indicate if a patient would not fit any of the mask
size options.

Figure 1: ESON Mask Sizing Guide

The second and most commonly used method employs the experience and expertise
of the clinicians as well as the feedback of the patient during the
consultation process, without the use of the abovementioned guide. This
method, while not always issuing the manufacturer recommended size can easily
and quickly incorporate in-field feedback during the decision making process.

As the cost of healthcare across the world continues to increase, and with an
estimated 59% of the adult population suffering from sleep apnoea [8, 9],
there is an increasing need to ensure that the time clinicians spend with
patients is most effectively used. The use of a fully automated mask sizing
system such as that described within this article could assist in alleviating
some of this time pressure for both the clinician and patient; while combined
with other telemedicine services could allow consultations to be completed
remotely, removing the need for remote and rural patients to travel the often
large distances to their nearest clinic.

The work presented within this paper extends upon our previous work where a
semi-automated solution for sizing nasal PAP masks was presented[10]; still
requiring human intervention to determine the scale of the image being
presented. The solution presented within this paper is one step closer to
assisting clinicians in that human intervention is not required to predict the
appropriate mask size.

##  II Datasets

A total of four separate datasets were used during the development of the
automated CPAP mask sizing system, including two publicly available datasets:
MUCT [11] and PUT [12]. One dataset collected during this study contains 206
facial photographs of male and female subjects, many of which were PAP
patients with an Australian 20 cent piece positioned on their forehead.
Physical nose width measurements were also taken of the participants using
vernier calipers; henceworth this dataset will be referred to as the test set.
One additional dataset of 131 facial photographs of PAP and non-PAP
participants was collected without any physical measurements or the presence
of the coin in the image; this dataset will be referred to as the post-train
set. One final dataset, described in more detail below was artificially
generated using both the MUCT and PUT datasets in addition to photographs of
both the head and tails side of an Australian 20 cent piece, downloaded from
the internet.

(a) MUCT example

(b) PUT example

(c) Test-set #

(d) Generated coin data

Figure 2: Datasets (# Image used with permission from Priyanshu Gupta)

Samples from the MUCT and PUT datasets were selected for use during this
study, as opposed to others databases such as 300W [13] and menpo [14] due to
their similarity to the data collected during this study. The MUCT dataset,
while containing images of approximately the same pose, with different
lighting is known for being composed of participants from a variety of
demographics; with both sexes, a range of different ages and ethnicities being
represented in the set. The MUCT samples selected for use where those images
captured from camera angle $a$ with all lighting options, both genders and
participants with and without glasses.

The PUT dataset, while not as ethnically diverse as the MUCT dataset, contains
samples of varied lighting and approximately the same pose as that shown in
the datasets containing PAP patients. All samples in the PUT dataset were
selected for use. Another reason for the selection of both MUCT and PUT is
that these datasets contain landmark annotations identifying the width of the
nose as required by the nasal PAP mask sizing guide. These landmarks were
manually annotated for images taken during this study and for the test set,
landmarks indicating the width of the coin were also manually annotated.

###  II-A Artificially Generated Coin Dataset

Given that only 245 samples of photographs of Australian 20 cent pieces were
available for use it was decided that more data would be required to produce
models capable of detecting the width of the coin with sufficient accuracy and
allowing a large enough test set to measure system performance. To overcome
this hurdle and remove the need to take a large number of images of a coin,
the MUCT and PUT datasets were used to artificially generate images of the
coin similar to that present within the test set. Six images of the heads side
of a 20c piece were downloaded from the internet in addition to another six
images of the standard tails side of coin containing impressions of the
platypus design.

For each image in the sample of MUCT and PUT datasets, one of the 12 coin
images were randomly sampled for inclusion in the generated sample. For the
MUCT dataset the intra-ocular distance was used to scale the image of the coin
to an appropriate size for the subject’s face; for the PUT samples the
distance between the inner corners of the eye was used. The approximate
location of the coin in the image was specified to be above the highest
landmark in the set and inline with the nose tip. Using uniform random
sampling, an offset was then applied to this position to increase the
variability of the coin position with respect to the face. The image of the
coin was then rotated by a random angle between -60 and 60 degrees to the
horizontal, a random gaussian blur was applied with $1<\sigma<4$ and finally
the brightness and contrast was randomly adjusted within $\pm 20\%$. All
random values were generated using uniform random sampling.

##  III Methods

The system architecture, as shown in Figure 3 is composed of multiple object
detectors (face, nose and coin), convolutional neural networks for landmark
identification followed by a stage which combines the results of landmark
identification stages to compute a recommended mask size. The face detector is
the entry point for the system, as all information that is required by the
system is contained within the bounding box of the face. In this
implementation the OpenCV implementation of the Viola & Jones algorithm [15]
was used with the pre-trained front-on face detector. For the nose and coin
detection stages; the Dlib implementation [16] of the Dalal & Triggs HOG based
detector [17] was used, training the detectors using a subset of the noses
present within the MUCT and PUT datasets as well as a sample of the coins
within the generated dataset.

Figure 3: System Architecture

###  III-A Nose Width CNN Stage

After the nasal region is extracted by the nose detector the sub-image and
corresponding training landmarks were resized to 42 x 42 RGB pixels, while
maintaining the perspective of the image. Those images, which after resizing
had a height of less than 42 pixels, were filled with values of zero to reach
the required size. For each training sample, the matrix of $2\times 2$
landmarks were flattened to form a single 4 unit vector. The image and
coordinate values were then scaled between -1 and 1 by dividing by their
respective maxima and subtracting the mean; when applying the test set to the
model, only the input image values required scaling.

The nasal regions of each image of the MUCT/PUT datasets, along with the nose
width coordinates were first extracted, then left/right flipped copies of the
data were constructed to double the amount of data available for training. The
selected data was randomly split into training and validation sets with a
consistently seeded random number generator; 70% of the data was allocated for
training and 30% for validation. The training and validation sets were then
used to tune the design of the nose width model, starting with a simpler,
single hidden layer neural network model and increasing the number of hidden
units, number of layers, as well as number of convolutional and pooling layers
until the error score was minimised. A relu non-linearity function was applied
to hidden and convolutional layers, max pooling was used for all pooling
layers and a root mean square error function applied as the objective
function. All weights, including weights for bias values were initialised with
random values between $\pm\sqrt{6/(units_{in}+units_{out})}$; stochastic
gradient descent was applied using minibatches of 128 samples, a fixed
learning rate of $1\times 10^{-4}$, momentum of 0.9 and patience of 50 epochs.
After the hyperparameter sweep the following architecture was selected: 1
Convolutional Layer with a $3\times 3$ filter and 32 feature maps, a max-
pooling layer a fully connected layer with 800 hidden units and the output
layer with 4 units.

Following training, the nose detector was then applied to 83 facial
photographs of PAP patients which along with the manually annotated landmarks
were normalised and reshaped to continue the training process. The samples
were also flipped left to right to create an additional 83 samples for
training. The previously saved weights were reloaded into the model and
stochastic gradient descent continued using a batch size of 1 and instead of a
70%/30% training / validation set split, 90% of the data was allocated for
training. The significantly larger training set was specified to provide the
model with as many samples as possible to learn the features of the actual
data of interest.

###  III-B Coin CNN Stage

The coin CNN stage is the means by which the automated system determines the
scale of the facial features within the image; knowing that the diameter of an
Australian 20 cent piece is $28.65$mm, by detecting the width of the coin in
the image, the scale in pixels per millimetre (px/mm) was determined.
Following the application of the HOG coin detector, the resulting sub-image is
resized and padded using the same process as used during the nose CNN stage,
to be consistent with the smallest available image i.e. 42 x 42 RGB pixels.
The resulting image is then normalised as previously described.

During training of the coin CNN stage, the artificially generated dataset was
also split with 70% of the data allocated for training. The same
hyperparameter selection process as was completed with the nose width CNN
stage was repeated. After the hyperparameter sweep, the following architecture
was selected: 1 Convolutional Layer with a $3\times 3$ filter and 24 feature
maps, a max-pooling layer, a fully connected layer with 800 hidden units and
the linear output layer with 4 units.

Once the training error had been minimised using the artificially generated
data, transfer learning was again used to refine the model for its intended
use. A total of 51 samples were selected from the test set for this purpose
and again a 90%/10% training / validation set split was used. The model was
loaded with the optimised weights and the gradient descent process was
continued, with the 51 isolated test set samples and a batch size of one,
until the error function was again minimised.

###  III-C Mask Recommendation Stage

The mask recommendation stage is where the nose width measurement in pixels
and the scale information as provided by the coin are combined to compute the
physical nose width of the patient in millimetres. This measurement was then
compared to each of the corresponding mask sizes. As described in section I-A
the issuance of PAP mask sizes is not an exact science; as such PAP mask
manufacturers provide an overlap for mask sizes, allowing patients who fall
within an overlap to be considered correctly sized if they receive one of
either of the adjacent sizes. To reflect this situation within the field we
applied a 5% tolerance to sizing decisions. For any individual whose ‘ground
truth’ size is within 5% of a sizing boundary, the predict size was considered
correct if one of either the adjacent sizes were issued.

##  IV Results

The face and nose detectors within the system correctly detected all faces and
noses within the test set, while the coin detector successfully identified all
but one of the present coins. Tables II and II summarise the sizing
performance of the automated system, while figure 4 provides examples of the
CNN stages performing well and poorly. Using the confusion matrix (Table II)
the fully automated PAP mask sizing system produced an accuracy of 64.71% and
86.1% within one size.

|  | Predicted Mask Sizes
---|---|---
Actual Mask Size |  | Small | Medium | Large | Too Large
Small | 75 | 15 | 5 | 8
Medium | 16 | 33 | 5 | 9
Large | 4 | 3 | 10 | 1
Too Large | 0 | 0 | 0 | 3

TABLE I: Confusion matrix of predicted mask sizes

| Sensitivity (%) | Pos. Predict. (%)
---|---|---
| S | M | L | TL | S | M | L | TL
Pred | 76 | 52 | 56 | 100 | 79 | 65 | 50 | 14

TABLE II: Sensitivity & positive predictivity performance of predicted mask
sizes (S: Small, M: Medium, L: Large, TL: Too Large)

(a) Good Examples

(b) Poor Examples

Figure 4: Predicted landmark results

##  V Discussion

The work completed throughout this study provides an excellent foundation for
creating a fully automated PAP mask sizing system. Despite the limited
availability of custom data, the use of similar, publicly available datasets
and transfer learning produced a system with 64.71% accuracy on unseen data
from the custom set. In order for a similar system to be deployed for use in
the field, the accuracy must be further improved. It is expected that by
increasing the size and quality of the PAP patient data set that overall
performance would increase as training could occur on the actual data as
opposed to ‘similar’ data. The dataset could also be enhanced by improving the
consistency of pose, facial expressions and lighting via clearer user
instructions.

##  VI Conclusion

This study presented a fully automated method of sizing the ESON nasal PAP
mask through the use of a multi stage convolutional neural network and
transfer learning. The system produced an accuracy of 64.71% and 86.1% within
one size.

## References

  * [1] Terri E. Weaver, Judith Reishtein, and Amy M Sawyer,  “Adherence to CPAP treatment and functional status in adult obstructive sleep apnea,”  in Sleep Apnea: Pathogenesis, Diagnosis and Treatment, p. 435. 2011\.
  * [2] C E Sullivan, F G Issa, M Berthon-Jones, and L Eves,  “Reversal of obstructive sleep apnoea by continuous positive airway pressure applied through the nares,”  Lancet, vol. 1, no. 8225, pp. 862–865, 1981.
  * [3] Clifford A. Massie and Robert W. Hart,  “Clinical outcomes related to interface type in patients with obstructive sleep apnea/hypopnea syndrome who are using continuous positive airway pressure,”  Chest, vol. 123, no. 4, pp. 1112–1118, 2003.
  * [4] Richard B Berry,  “Improving CPAP compliance – man more than machine,”  Sleep Medicine, vol. 1, no. 3, pp. 175–178, jul 2000.
  * [5] Peter Gay, Terri Weaver, Daniel Loube, and Conrad Iber,  “Evaluation of positive airway pressure treatment for sleep related breathing disorders in adults.,”  Sleep, vol. 29, no. 3, pp. 381–401, 2006.
  * [6] Anastasia Amfilochiou, Venetia Tsara, Likurgos Kolilekas, Evangelia Gizopoulou, Chrysoula Maniou, Demosthenes Bouros, and Vlasios Polychronopoulos,  “Determinants of continuous positive airway pressure compliance in a group of Greek patients with obstructive sleep apnea,”  European Journal of Internal Medicine, vol. 20, no. 6, pp. 645–650, 2009.
  * [7] H M Engleman, S E Martin, and N J Douglas,  “Compliance with CPAP therapy in patients with the sleep apnoea/hypopnoea syndrome.,”  Thorax, vol. 49, no. 3, pp. 263–6, 1994.
  * [8] Seckin O Ulualp,  “Snoring and obstructive sleep apnea.,”  The Medical clinics of North America, vol. 94, no. 5, pp. 1047–55, 2010.
  * [9] Torpy JM, C Lynm, and Golub RM,  “Sleep apnea,”  JAMA, vol. 305, no. 9, pp. 956, mar 2011.
  * [10] Benjamin Johnston, Alistair McEwan, and Philip de Chazal,  “Semi-automated nasal PAP mask sizing using facial photographs,”  in 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). jul 2017, pp. 1214–1217, IEEE.
  * [11] S Milborrow, J Morkel, and F Nicolls,  “The MUCT Landmarked Face Database,”  Pattern Recognition Association of South Africa, 2010.
  * [12] Andrzej Kasinski, Andrzej Florek, and Adam Schmidt,  “The PUT face database,”  Image Processing and Communications, , no. 13, pp. 59–64, 2008\.
  * [13] Christos Sagonas, Epameinondas Antonakos, Georgios Tzimiropoulos, Stefanos Zafeiriou, and Maja Pantic,  “300 Faces In-The-Wild Challenge: database and results,”  Image and Vision Computing, vol. 47, pp. 3–18, 2016.
  * [14] Stefanos Zafeiriou, George Trigeorgis, Grigorios Chrysos, Jiankang Deng, and Jie Shen,  “The Menpo Facial Landmark Localisation Challenge: A Step Towards the Solution,”  in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, jul 2017.
  * [15] Paul Viola and Michael J Jones,  “Robust Real-time Object Detection,”  International Journal of Computer Vision, , no. February, pp. 1–30, 2001.
  * [16] Davis E. King,  “Dlib-ml: A Machine Learning Toolkit,”  Journal of Machine Learning Research, vol. 10, pp. 1755 – 1758, 2009.
  * [17] N. Dalal and B. Triggs,  “Histograms of Oriented Gradients for Human Detection,”  in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05). 2005, vol. 1, pp. 886–893, IEEE.