Further author information: (Send correspondence to J.I.O.)
J.I.O.: E-mail<EMAIL_ADDRESS>Telephone: +54 249 438
5690

# A ResNet is All You Need? Modeling A Strong Baseline for Detecting Referable
Diabetic Retinopathy in Fundus Images

Tomás Castilla  Yatiris Group, PLADEMA Institute, UNICEN, Campus
Universitario, Tandil, Argentina  Marcela S. Martínez  Centro de Oftalmología
Martínez, Hipólito Yrigoyen 380, Pehuajó, Argentina  Mercedes Leguía  Servicio
de Oftalmología, Hospital de Alta Complejidad En Red ”El Cruce” Dr. Néstor
Carlos Kirchner, Av. Calchaquí 5401, Florencio Varela, Argentina  Ignacio
Larrabide  Yatiris Group, PLADEMA Institute, UNICEN, Campus Universitario,
Tandil, Argentina  Consejo Nacional de Investigaciones Científicas y Técnicas,
CONICET, Tandil, Argentina  José Ignacio Orlando  Yatiris Group, PLADEMA
Institute, UNICEN, Campus Universitario, Tandil, Argentina  Consejo Nacional
de Investigaciones Científicas y Técnicas, CONICET, Tandil, Argentina

###### Abstract

Deep learning is currently the state-of-the-art for automated detection of
referable diabetic retinopathy (DR) from color fundus photographs (CFP). While
the general interest is put on improving results through methodological
innovations, it is not clear how good these approaches perform compared to
standard deep classification models trained with the appropriate settings. In
this paper we propose to model a strong baseline for this task based on a
simple and standard ResNet-18 architecture. To this end, we built on top of
prior art by training the model with a standard preprocessing strategy but
using images from several public sources and an empirically calibrated data
augmentation setting. To evaluate its performance, we covered multiple
clinically relevant perspectives, including image and patient level DR
screening, discriminating responses by input quality and DR grade, assessing
model uncertainties and analyzing its results in a qualitative manner. With no
other methodological innovation than a carefully designed training, our ResNet
model achieved an AUC = 0.955 (0.953 - 0.956) on a combined test set of 61007
test images from different public datasets, which is in line or even better
than what other more complex deep learning models reported in the literature.
Similar AUC values were obtained in 480 images from two separate in-house
databases specially prepared for this study, which emphasize its
generalization ability. This confirms that standard networks can still be
strong baselines for this task if properly trained.

###### keywords:

Diabetic retinopathy, Fundus photography, Deep Learning, Image classification

##  1 INTRODUCTION

Automated detection of referable diabetic retinopathy (DR) in color fundus
photographs (CFP) has been extensively explored in the literature [1], as it
aids in reducing the cost of large scale screening campaigns and favors the
access to early treatment [2, 3, 1]. Deep Convolutional Neural Networks (CNNs)
have became the de facto standard for this task [4, 5, 6, 7, 8, 1, 9, 10, 11,
12], improving results up to a point in which they are either comparable or
superior to those obtained by trained ophthalmologists. [3, 13, 1] Authors
generally attribute the accuracy of their results to their proposed
methodological innovations, which can vary from novel architectural modules
[14] to new pre-training strategies [7, 15, 12] or multitask designs [16],
among others [1, 17]. Surprisingly, we observed that little to no attention is
payed to simple yet influential factors such as integrating multiple data
sources for training, applying good preprocessing strategies on the input
images or calibrating data augmentation techniques. Our hypothesis is that
this leads to incomplete or unfair comparisons with suboptimally trained
baselines, being unknown if training standard classification networks in a
proper manner is more beneficial than using complex architectures.

Recent studies have analyze the influence of these factors on DR
classification or grading algorithms [9, 11]. In particular, Huang et al. [11]
demonstrated that a systematic investigation of training losses, input image
resolutions, data augmentation strategies and learning rate scheduling
techniques allows a standard ResNet-50 model to achieve state-of-the-art
results for DR grading. Building on top of these results, in this paper we
present a strong baseline for detecting referable DR from CFPs based solely on
a properly trained standard ResNet architecture. This binary classification
task differs in that, instead of classifying the DR grade, it detect cases
that need to be referred to an ophthalmologist (namely moderate to severe non-
proliferative DR and proliferative DR) from those that are not at risk (with
no DR or mild DR signs). Our model is based on a simple ResNet-18
architecture, pre-trained on ImageNet and fine-tuned using images preprocessed
with standard practices. To increase its ability to generalize to new
databases, we crafted a multiethnic training set based on several public
sources, while incorporating a series of data augmentation strategies
empirically calibrated using a held-out validation set. The proposed model was
evaluated using nine public test sets with more than 61.000 images, and two
additional in-house data sets specially prepared for this study, namely
Martínez and HEC. To this end, we applied an extensive protocol accounting for
multiple clinically relevant observations, including image level and patient
level evaluations, discriminating responses according to the quality and DR
grade of the input, assessing model uncertainties and qualitatively analyzing
model’s attention through class activation maps. Notoriously, our simple
ResNet-18 baseline performed in line or even better than other recently
published, more complex deep learning architectures, even though no problem-
specific innovations were implemented in ours. These results empirically
confirm our hypothesis that properly trained standard classification networks
can be strong enough to compete with other state-of-the-art approaches, and
that existing comparisons in the literature are sometimes unfair and
suboptimal. To avoid this issue and favor reproducibility, we publicly release
our training, validation and test data partitions and our test set
predictions. Finally, we identify a series of weak points in this baseline and
recommend alternative lines of research to address them.

##  2 MATERIALS AND METHODS

###  2.1 Materials

We used public databases that provided either referable/non-referable DR
labels or DR grades or manual segmentations of DR lesions (Table 1). Binary
labels were generated from grades by mapping no DR or mild non-proliferative
DR (NPDR) to the non-referable DR class, and moderate and severe NPDR or
proliferative DR (PDR) to the referable DR class. When only lesion
segmentations were available, we assigned binary labels using Decenciere’s et
al. criterion[18]. We also created two in-house private test databases by
retrospectively collecting anonymized images from the Ophthalmological Center
Martínez (Pehuajó, Argentina)–Martínez set–and Hospital “El Cruce” (Florencio
Varela, Argentina)–HEC set–. The protocol to access the images was approved by
the hospital’s Ethics Committee, in accordance to the tenets of the
Declaration of Helsinki. Martínez comprises 484 images acquired using a non-
mydriatic Cristal Vue NFC-700 fundus camera with a field-of-view (FOV) of
$45^{\circ}$. HEC, on the other hand, corresponds to an atlas of 35 images
acquired using a Topcon TRC-NW8 non-mydriatic camera, similar to the one used
in the DR2 set. Both sets were manually labeled by two independent
ophthalmologists (MM and ML, respectively), indicating which images correspond
to referable DR cases.

###  2.2 Image preprocessing

Referable DR classification approaches use different image preprocessing
strategies, with two being the most commonly seen, namely cropping out the
empty areas outside the CFP field-of-view (FOV),[19, 14, 20] and enhancing the
contrast of DR lesions and vascular structures by subtracting an estimated
background. [21, 5, 11]

FOV cropping allows to standardize the resolution of images that were
collected from multiple sources or acquired with different devices.
Furthermore, it preserves their overall aspect ratio when resized for
training, as seen in Figures 1(a) and 1(b). This operation requires to
approximate the border of the FOV, which we did by summing the red, green and
blue intensities of the input image in a pixel-wise manner, then thresholding
the resulting matrix using Otsu’s method, and finally applying a filling holes
operation. Figure 1(b) (left) depicts the resulting binary mask. The smallest
possible bounding box was then predicted from it, and its coordinates are used
to crop the area of interest before resizing. The final image is observed in
Figure 1(b) (right).

Table 1: Training, validation and test partitions created from public
databases and our two in-house sets.

Dataset | Num. samples |  Training |  Validation |  Test
---|---|---|---|---
|  Non-referable DR |  Referable DR |  Total |  |  |
APTOS2019 111 |  2175 |  1487 |  3662 |  3662 |  0 |  0
DeepDRID [22] |  900 |  700 |  1600 |  1200 |  0 |  400
DDR [8] |  6896 |  5626 |  12522 |  6260 |  2503 |  3759
EyePACS 222 |  71548 |  17154 |  88702 |  28098 |  7026 |  53576
IDRiD [23] |  193 |  323 |  516 |  372 |  40 |  103
FCM-UNA [24] |  191 |  566 |  757 |  0 |  0 |  757
1000Fundus [25] |  56 |  88 |  144 |  0 |  0 |  144
DR2 [4] |  337 |  98 |  435 |  0 |  0 |  435
MESSIDOR 2 [18] |  1287 |  457 |  1744 |  0 |  0 |  1744
DIARETDB1 [26] |  43 |  46 |  89 |  0 |  0 |  89
Martínez (private) |  454 |  30 |  484 |  0 |  0 |  484
HEC (private) |  26 |  9 |  35 |  0 |  0 |  35
Total |  89433 |  27735 |  117168 |  39592 |  9569 |  61526
|  |  |  |  |  |

Contrast enhancement, on the other hand, has been widely applied since
Graham’s contribution to Kaggle EyePACS competition [21]. As the author
originally mentioned, it facilitates to identify small red lesions associated
with early DR grades, specially in areas of uneven illumination (Figures 1(c)
and 1(d)). Formally, the contrast enhanced image $I_{\text{ce}}$ is obtained
from the original image $I$ by doing:

$I_{\text{ce}}(i,j;\sigma)=\alpha I(i,j)-\beta G(i,j;\sigma\ast
I(i,j)+\gamma,$ (1)

where $i,j$ are the horizontal and vertical coordinates of each pixel,
$\alpha=\beta=4$ are blending parameters, $\ast$ is the convolution operator,
$G(\cdot)$ is a Gaussian kernel with $\sigma=\frac{w}{90}$ for $w$ the width
of $I$, and $\gamma=128$ is an anchor intensity. To avoid undesired artifacts
when applying the kernel in the borders of the FOV, pixels outside it are
replaced by the mean RGB intensities of the pixels within the convex hull of
the originally predicted FOV (Figure 1(c), left). Notice from Eq. 1 that the
Gaussian filter approximates a background (Figure 1(c), center) that, once
subtracted to the image, enhances in $I_{\text{ce}}$ (Figure 1(c), right) both
tiny red lesions and thin vessels (Figure 1(d)).

###  2.3 Deep learning model and training setup

We chose a ResNet [27] as neural network backbone due to its popularity for
referable DR classification [28, 14, 29, 9, 30]. Among its many variants, we
selected a ResNet-18 (Figure 2), the one with the smallest capacity. No
adaptations were made on the architecture compared to its original definition.
[27] Inputs are RGB fundus images preprocessed as described in Section 2.2,
resized to $512\times 512$ pixels and with intensities normalized to the
interval $[-1,+1]$. Outputs, on the other hand, are one-hot encoding vectors
for non-referable (0) and referable (1) DR classes. The network comprises 17
convolutional layers and a fully connected one. Each convolutional layer is
followed by batch normalization and ReLU activations, with a number of filters
that is doubled every 4 layers, starting with 64. Adaptive average pooling is
used to map the last tensor of convolutional activations into a vector of
fixed length 512 features to feed the last classification layer. A softmax
activation is applied on top of the output logits to map them into
probabilities.

Images from databases described in Section were used to construct our
training, validation and test partitions (Table 1). As EyePACS
222https://www.kaggle.com/c/diabetic-retinopathy-detection/data and IDRiD [23]
only provide training and test splits, we integrated 10% of their
corresponding training sets in our validation set, preserving the remaining
ones for training and their test sets for testing. Similarly, we followed the
pre-defined partitions from DDR [8], and the original DeepDRID [22] training
set was used for training. As DeepDRID does not provide labels for its test
set, we used its validation set for testing instead. Likewise, we discarded
the validation and test sets from APTOS2019
111https://www.kaggle.com/competitions/aptos2019-blindness-detection/data and
integrated only its training set to ours. The remaining databases (FCM-UNA
[24], 1000Fundus [25], DR2 [4], MESSIDOR 2 [18] and DIARETDB1 [26]) were all
used as test sets. Since 1000Fundus includes images with multiple conditions,
we created a test set using only normal subjects and those with different DR
grades, discarding other diseases and observations. Finally, Martínez and HEC
sets were used as independent test sets, too.

We used the pretrained on ImageNet, PyTorch 1.10 implementation of this
network. A standard cross-entropy loss with no class weighting was used as
training objective, minimized using Adam optimization with an initial learning
rate of 0.0001. This parameter was decreased to its half every time the area
under the receiver-operating characteristic curve (AUC) in the validation set
was not improved for a maximum of 20 epochs. Batch size was fixed to 128
images per iteration, training for a maximum of 150 epochs and using early
stopping with a patience of 40 epochs, based in the validation set AUC.
Additional regularization was introduced using weight decay with a rate of
0.001 and online data augmentation techniques that were randomly and
successively applied on the images before contrast enhancement. Operations
included color alterations, horizontal and vertical flippings, rotations and
scalings, with parameters empirically fixed using the validation set. The
model was trained on a cloud-computing platform using an NVIDIA V100 GPU with
16 GBs of RAM, and evaluated in a standard i7 CPU with 16 GB of RAM and an
NVIDIA 3060 GPU with 12 GB of memory.

(a) No preprocessing

(b) FOV crop

(c) Contrast enhancement

(d) Detail

Figure 1:  Preprocessing operations. (a) If no preprocessing is used, image
resizing affects its aspect ratio. (b) FOV cropping better preserves the
aspect ratio. Left: Estimated FOV. Right: Image cropped around the FOV and
resized to $512\times 512$ pixels. (c) Contrast enhancement [21]. Left: convex
hull of the estimated FOV. Center: Estimated background. Right: contrast
enhanced image $I_{\text{ce}}$. (d) Details of input image $I$ (left) and
$I_{\text{ce}}$ (right). Figure 2: ResNet-18 architecture of our proposed
baseline. Curved arrows represents standard residual connections, while dotted
ones are residual connections with zero-padding to match dimensions.

###  2.4 Evaluation metrics and data analysis

We quantitatively evaluated the proposed baseline using standard binary
classification metrics such as AUC, sensitivity (Se) and specificity (Sp). AUC
was computed directly from referable DR probabilities predicted by the model.
Se and Sp were calculated using the argmax of these outputs, hence without
choosing an optimal operating point. Statistically testing the differences in
AUC is not possible as models included in the comparison did not release their
individual predictions. Therefore, we used 95% confidence intervals (95% CI)
for all metrics, obtained using bootstrap with replacement with $n=1000$
samples for ROC computation and Wilson’s method [31] for Se and Sp. To
qualitatively analyze results, class activation maps were automatically
obtained using the XGrad-Cam method [32], as implemented in Torchcam
333https://frgfm.github.io/torch-cam/. To account for multiscale features, the
algorithm was applied to the second, third and fourth convolutional blocks of
the network, and fused into a single one. In each case, the maps were computed
for the predicted class: thus, if the model predicted a non-referable case,
then attributions were obtained for that specific class, and viceversa. We
also computed the uncertainty of the model as the entropy of the predicted
probabilities to study how much it varies under different scenarios.

##  3 RESULTS AND DISCUSSION

Table 3 presents the AUC, Se and Sp values and their 95% CI obtained for
classifying referable DR cases in each test set. Results reported by other
deep learning models evaluated with the same data sets and metrics than ours
are also included. To the best of our knowledge, existing works only report
comparable values in DDR, DR2, EyePACS, IDRiD and MESSIDOR 2, while ours is
the first to be also evaluated in 1000Fundus, DeepDRID, DIARETDB1 and FCM-UMA.
When mixing all public datasets, our ResNet reported an AUC = 0.955 (0.953 -
0.956), with Se = 0.752 (0.744 - 0.759) and Sp = 0.977 (0.976 - 0.979). In
terms of computational efficiency, each prediction took 0.07 $\pm$ 0.004
seconds per image. In our private data sets, the model showed results
comparable to those in the public ones, including in Martínez, which has
images acquired with a device unseen during training. Surprisingly, our
results in public sets are in pair or even better than those reported in the
literature, which lie within or below the 95% CIs. In DDR, our ResNet obtained
AUC values higher than those achieved with the lesion detection model by Zago
et al. [10]. A similar behavior is observed in IDRiD, were our model slightly
outperforms Hervella et al. [12] method as well. Notice that this latter model
uses a CNN pretrained in a paired data set of co-registered CFPs and
fluorescein angiographies, which are costly to obtain. Alternatively, our
simple model fine-tuned from ImageNet was able to obtain comparable results.
When training their approach using the same strategy than ours, Hervella et
al. reported numbers notoriously lower than those achieved with our method
[12] (AUC = 0.885 vs. 0.946, respectively). In MESSIDOR 2, the highest
observed AUC and Se values were reported by Gulshan et al. [3]. However, such
an approach was trained using 128175 images, while ours uses one-third of that
amount. The method by Gargeya et al. [6], on the other hand, which used 75137
images for training, reported lower AUC and Sp values than our ResNet, which
was trained with half this amount of scans. This might be attributed to our
learning scheme based on images collected from multiple sources and a well-
calibrated data augmentation strategy, which allows to diversify the
appearance of CFPs in the training set and therefore strengthen the model
generalization ability. Li et al. [20] achieved slightly better results using
an ensemble of five Inception-v4 networks trained with less than 9000 images
of size $299\times 299$ pixels each. Instead, we used a larger training set
with higher resolution images but for learning a single model, which in
practice corresponds to a less number of parameters. Furthermore, authors
reported an AUC $=0.951$ (0.947-0.954) when training a single model with
$1488\times 1488$ pixel images, which is lower than what we obtained. In the
EyePACS set, which features more than 50000 studies for evaluation, our
algorithm slightly exceeds the AUCs reported by Pires et al. [15] and Quellec
et al. [7]. Finally, our model also obtained a higher AUC than Pires et al.
[15] in DR2. Such an approach uses data augmentation and multiple trainings at
different image resolutions to make the model robust to changes in scaling.
Ours, instead, achieves this ability by simply learning from diverse data.

Any automated screening system should alert the patient if dangerous DR signs
are detected in any of their eyes. Thus, we conducted a per-patient evaluation
combining responses from multiple images of the same patient, using EyePACS
and MESSIDOR 2, which provide one image per eye of the same individual, and
DeepDRID, which includes two images per eye. A subject was considered
referable if at least one of their images was labeled as that in the ground
truth. Similarly, we took the maximum predicted probabilities of all images of
their left and right eyes to produce a single probabilistic response of the
model. Results are summarized at the bottom of Table 3. When contrasting them
with the per-image evaluation, improvements in Se are observed at the cost of
decreases in Sp. These changes are more evident in DeepDRID and EyePACS, were
Se is increased from 88.3% to 98% and from 73.2% to 77.3%, respectively, while
Sp is decreased from 86.8% to 80% and from 97.9% to 96.9%, respectively. A
similar yet less compelling behavior is also observed in MESSIDOR 2. AUC, on
the other hand, is increased in DeepDRID when using multiple images per eye,
while in EyePACS and MESSIDOR 2 the AUC is slightly slower. When comparing
these values with other state-of-the-art alternatives, we observe that Pires
et al.[15] reports higher results. However, tehir approach integrates
responses from both eyes by feeding features from the classification network
to a dedicated classifier. Ours, on the contrary, reproduces the alternative
used by Zago et al. [10] of taking the maximum predicted probability. Hence,
it is not possible to determine if these differences are due to the neural
network itself or the patient-level classifier.

Table 2: Per-image (top) and per-patient (bottom) evaluation for referable DR
classification in all test sets in terms of AUC, Se, Sp and their 95% CI.

Per-image referable DR classification
---
Test dataset |  Method |  AUC (95% CI) |  Se (95% CI) |  Sp (95% CI)
1000Fundus |  ResNet-18 (ours) |  1.000 (1.000 - 1.000) |  1.000 (0.958 - 1.000) |  0.982 (0.906 - 0.997)
DDR |  Zago et al. 2020 [10] |  0.833 (0.819 - 0.846) |  - |  -
|  ResNet-18 (ours) |  0.965 (0.960 - 0.970) |  0.749 (0.728 - 0.769) |  0.978 (0.971 - 0.984)
DeepDRID |  ResNet-18 (ours) |  0.959 (0.944 - 0.972) |  0.883 (0.828 - 0.922) |  0.868 (0.817 - 0.907)
DIARETDB1 |  ResNet-18 (ours) |  0.981 (0.956 - 0.999) |  0.957 (0.855 - 0.988) |  0.930 (0.814 - 0.976)
DR2 |  Pires et al. 2019 [15] |  0.963 (0.938 - 0.981) |  - |  -
|  ResNet-18 (ours) |  0.974 (0.962 - 0.985) |  0.847 (0.763 - 0.905) |  0.961 (0.935 - 0.977)
EyePACS |  Quellec et al. 2017 [7] |  0.944 |  - |  -
|  Pires et al. 2019 [15] |  0.946 |  - |  -
|  ResNet-18 (ours) |  0.951 (0.949 - 0.954) |  0.732 (0.723 - 0.740) |  0.979 (0.978 - 0.980)
FCM-UNA |  ResNet-18 (ours) |  0.986 (0.980 - 0.992) |  0.882 (0.852 - 0.906) |  0.990 (0.963 - 0.997)
IDRiD |  Zago et al. 2020 [10] |  0.796 (0.715 - 0.892) |  - |  -
|  Hervella et al. 2022 [12] |  0.944 |  - |  -
|  ResNet-18 (ours) |  0.949 (0.914 - 0.980) |  0.828 (0.718 - 0.901) |  0.897 (0.764 - 0.959)
MESSIDOR 2 |  Gulshan et al. 2016 [3] |  0.990 (0.986 - 0.995) |  0.961 (0.924 - 0.983) |  0.939 (0.924 - 0.953)
|  Gargeya et al. 2017 [6] |  0.940 |  0.930 |  0.870
|  Voets et al. 2019 [33] |  0.853 (0.835 - 0.871) |  0.818 |  0.687
|  Zago et al. 2020 [10] |  0.944 (0.925 - 0.966) |  0.900 (0.860 - 0.961) |  0.870 (0.863 - 0.871)
|  Li et al. 2022 [20] |  0.977 (0.974 - 0.981) |  0.923 (0.917 - 0.925) |  0.947 (0.937 - 0.954)
|  ResNet-18 (ours) |  0.973 (0.967 - 0.979) |  0.895 (0.863 - 0.920) |  0.941 (0.927 - 0.953)
HEC |  ResNet-18 (ours) |  0.961 (0.900 - 1.000) |  1.000 (0.610 - 1.000) |  0.862 (0.694 - 0.945)
Martínez |  ResNet-18 (ours) |  0.955 (0.927 - 0.980) |  0.800 (0.627 - 0.905) |  0.934 (0.907 - 0.953)
Per-patient referable DR classification
DeepDRID  (100 patients) |  ResNet-18 (ours) |  0.980 (0.957 - 0.999) |  0.980 (0.895 - 0.996) |  0.800 (0.670 - 0.888)
|  |  |  |
EyePACS (26788 patients) |  Pires et al. 2019 [15] |  0.955 (0.951 - 0.958) |  - |  -
|  Zago et al. 2020 [10] |  0.821 (0.812 - 0.829) |  - |  -
|  ResNet-18 (ours) |  0.948 (0.945 - 0.951) |  0.773 (0.762 - 0.783) |  0.969 (0.966 - 0.971)
MESSIDOR 2 (870 patients) |  Pires et al. 2019 [15] |  0.982 (0.974 - 0.989) |  - |  -
|  Zago et al. 2020 [10] |  0.944 (0.927 - 0.965) |  - |  -
|  ResNet-18 (ours) |  0.970 (0.960 - 0.979) |  0.907 (0.866 - 0.936) |  0.920 (0.896 - 0.939)
|  |  |  |

We performed an additional experiment discriminating classification outputs
according to quality labels provided for EyePACS by Fu et al. [34] (”reject”,
”usable” and ”good”) and Zhou et al. [16] (”bad” and ”good”), and those
available for DeepDRID (”bad” and ”good”). Figure 3 presents the ROC curves
obtained for each type of image, jointly with image samples for each category.
In EyePACS, the model obtained lower AUC values for bad quality images,
regardless of the source of the quality labels. On the other hand, results for
usable and good images are almost equivalent when separated using Fu et al.
[34] labels. Finally, a similar behavior is observed in DeepDRID, with AUC
values that are comparable between good and bad quality scans.

(a) EyePACS (Fu et al. [34] labels)

(b) EyePACS (Zhou et al. [16] labels)

(c) DeepDRID

Figure 3: ROC curves and AUC values obtained for referable DR detection in
images separated by their quality. Shadings correspond to 95% CI. Borders of
each image are color coded according to their quality label.

Figure 4 depicts ROC curves presenting the differences in performance to
recognize cases of non-referable DR from moderate NPDR, first, then non-
referable RD vs. severe NPDR, and finally non-referable RD from proliferative
RD (PDR). This was achieved by separating subsets from DDR, IDRiD, MESSIDOR 2,
EyePACS, DeepDRID and FCM-UMA using their disease grade, and creating
different mixed versions in which the non referable set was always fix and the
referable one changed. The highest AUC values were obtained for the more
advanced grades, while results for moderate NPDR were also high yet smaller.
Noticeable, in some cases RDP detection was less accurate than severe NPDR
detection, although differences are not compelling.

(a) DDR

(b) DeepDRID

(c) EyePACS

(d) FCM-UNA

(e) IDRiD

(f) MESSIDOR 2

Figure 4: ROC curves and AUC values obtained using referable DR probabilities
to classify non-referable DR from moderate, severe and very severe NPDR, and
RDP and advanced RDP. Shadings correspond to 95% CI.

Finally, Figure 5 illustrates some representative qualitative results obtained
in images from FCM-UNA. Each example includes the image, its preprocessed
version and its XGrad-Cam on top of the original scan. We also included the
ground truth DR grade, a mark indicating if the image was correctly classified
(✓) or not (✗), the predicted class and referable DR probability and the
uncertainty estimate. According to the ground truth labels, the ResNet-18
model correctly classified almost all non referable DR cases in FCM-UNA, even
in dark, low quality images as those shown in (a) and (c). XGrad-Cams in these
cases highlight either the macula, the optic disc, the normal nerve fiber
layers in the vascular arcades and some arteriovenous crossings (see arrows in
(a) and (d)). Regions with cotton wool spots or even with no apparent clinical
meaning are also activated in (c) (top and bottom arrows, respectively). The
case shown in (b) corresponds to one of the two false positives in FCM-UNA,
which was classified as referable with a high probability. The image shows
signs of hypertensive retinopathy, with an inferior ghost vessel and pigmented
lesions at a macular level. The XGrad-Cam shows activation on these lesions,
indicating that they were took into account for predicting the case as
referable. Alternatively, (d) depicts an image indicated as non-referable DR
by the ground truth labeling, that the model correctly classified as such.
Lesions associated with a maculopathy are observed at the vecinity of the
fovea, including a potential exudate, that the activation map seems to ignore.
When analyzing the referable DR examples, we can observe that the model has
usually high confidence when classifies them correctly, which is reflected by
the low uncertainty values. Although activation maps localize certain
microhemorrhages (see arrows in (e), (g) and (k)) and exudates (top right
arrow in (g) and (i)), they usually ignore areas with large hemorrhages (top
left arrow in (g), bottom arrows in (i) and (k)). Furthermore, big activation
blobs are seen in (e), (g), (j) and (i) over areas that do not correspond to
relevant signs but e.g. artifacts (j). The errors on the right hand side of
Figure 5, on the other hand, are associated to low quality inputs in which the
model made predictions with high uncertainty. The case in (f), for instance,
is a false negative of the model, even though the image shows pathological
signs such as microhemorrhages, soft exudates, venous dilations and
pathological arteriovenous crossings. The XGrad-Cam only highlights the
latter, which might indicate that the model struggled to identify the other
signs and therefore made a negative prediction. A similar behavior is seen in
(l), where only a small bright lesion is captured as relevant. The model
detected cases (h) and (j) as non-referable DR, while they are labeled as
severe NPDR cases in FCM-UNA. However, no abnormalities are clearly
observable, which might indicate that they are wrongly tagged in the database.
Finally, we asked two experienced ophthalmologists (ML and MM) to evaluate the
utility of the contrast enhancement operation. According to them, it allows to
better visualize vascular structures (b, j), white lesions (c, l) and small
hemorrhages (e, f, g, k), although in other cases also enhances artifacts (i),
does not increase the overall quality of the image (a, d) or even hide large
hemorrhages (bottom arrow in (i)).

Figure 5: Qualitative results obtained in FCM-UNA dataset. Each example
includes DR grade, the image cropped around the FOV area (left), its
preprocessed version (center) and the associated XGrad-Cam (right), jointly
with the predicted class, the predicted probability for referable DR and the
uncertainty of the prediction.

##  4 CONCLUSIONS

We presented a strong baseline for referable DR detection from CFPs based on a
standard ResNet-18 classification network. By combining data from multiple
sources and a calibrated data augmentation strategy, we showed that it is
possible to obtain an accurate model that performs in line or even better than
other more complex state-of-the-art approaches. This is in line with other
studies showing that optimally trained baselines can be better than most of
the recently published alternatives. [35] Furthermore, it highlights the
importance of collecting more diverse data before extensively modifying the
methodology. In particular, our model showed high AUC values even when applied
on our in-house databases, highlighting its generalization ability to inputs
acquired with previously unseen devices, most likely due to uniforming images
using a common preprocessing approach. By reproducing multiple clinically
relevant scenarios, our evaluation simultaneously showed that the model can be
used as a strong baseline but some weak points should be improved, aiding us
to envision other promising lines of research. In particular, we observed that
our ResNet struggles more to recognize moderate NPDR cases than those
suffering from more advanced disease grades. While this might be a consequence
of potential ambiguities in the ground truth labels, future efforts should be
made in improving the discrimination ability in these cases. One potential way
to do so is by training a multitask version that simultaneously predict the
referable DR probability and the presence/absence of DR related lesions such
as microaneurysms, hemorrhages, exudates or neovascularizations. The
complementary nature of these tasks might aid to further improve in previously
ambiguous classification scenarios. Moreover, we qualitatively observe that
the contrast equalization technique, despite useful to highlight small
lesions, might enhance acquisition artifacts and hide large hemorrhages.
Learning preprocessing parameters or simply creating a multichannel input by
combining this image with the original one might eventually solve this
limitation and let the network chose relevant features by its own.
Furthermore, a certain relationship seems to arise between the image quality
or the correctness of the response with prediction uncertainties. This is
aligned with other previous observations in DR grading [36], suggesting that
this link should be further exploited to improve the accuracy of the models
also in this binary task. XGrad-Cams, on the other hand, showed a limited
ability to detect some clinically relevant features such as exudates in the
macular area. This can be associated with how the referable DR class was
defined. Recent studies have suggested to integrate the risk of macular edema
in this binary target [28], as it also corresponds to a referable condition.
This might certainly aid and enrich the outputs of the model. Finally,
predicting the presence of lesions as mentioned above might help to improve
these maps, further strengthening their applicability for lesion detection. To
account for future comparisons, our results are publicly released at
https://github.com/TomasCast/sipaim-2022-resnet.

###### Acknowledgements.

This study was partially funded by PICTs 2019-00070 and startup 2021-00023
granted by Agencia I+D+i (Argentina) and by PIP 2021-2023 11220200102472CO
from CONICET (Argentina). We also thank NVIDIA for granting cloud-based GPU
computing hours as part of their Applied Research Accelerator Program.

## References

  * [1] Grzybowski, A. et al., “Artificial intelligence for diabetic retinopathy screening: a review,” Eye 34(3), 451–460 (2020).
  * [2] Scotland, G. S. et al., “Costs and consequences of automated algorithms versus manual grading for the detection of referable diabetic retinopathy,” BJO 94(6), 712–719 (2010).
  * [3] Gulshan, V. et al., “Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs,” JAMA 316(22), 2402–2410 (2016).
  * [4] Pires, R. et al., “Beyond lesion-based diabetic retinopathy: a direct approach for referral,” IEEE J. Biomed. Health Inform. 21(1), 193–200 (2015).
  * [5] Van Grinsven, M. J. et al., “Fast convolutional neural network training using selective data sampling: Application to hemorrhage detection in color fundus images,” IEEE TMI 35(5), 1273–1284 (2016).
  * [6] Gargeya, R. and Leng, T., “Automated identification of diabetic retinopathy using deep learning,” Ophthalmol. 124(7), 962–969 (2017).
  * [7] Quellec, G. et al., “Deep image mining for diabetic retinopathy screening,” Med. Image Anal. 39, 178–193 (2017).
  * [8] Li, T. et al., “Diagnostic assessment of deep learning algorithms for diabetic retinopathy screening,” Inf. Sci. 501, 511–522 (2019).
  * [9] Yip, M. Y. et al., “Technical and imaging factors influencing performance of deep learning systems for diabetic retinopathy,” NPJ Digit. Med. 3(1), 1–12 (2020).
  * [10] Zago, G. T. et al., “Diabetic retinopathy detection using red lesion localization and convolutional neural networks,” Comput. Biol. Med. 116, 103537 (2020).
  * [11] Huang, Y. et al., “Identifying the key components in resnet-50 for diabetic retinopathy grading from fundus images: a systematic investigation,” arXiv preprint arXiv:2110.14160 (2021).
  * [12] Hervella, Á. S. et al., “Multimodal image encoding pre-training for diabetic retinopathy grading,” Comput. Biol. Med. 143, 105302 (2022).
  * [13] Gulshan, V. et al., “Performance of a deep-learning algorithm vs manual grading for detecting diabetic retinopathy in india,” JAMA Ophthalmol. 137(9), 987–993 (2019).
  * [14] Liu, Y.-P. et al., “Referable diabetic retinopathy identification from eye fundus images with weighted path for convolutional neural network,” Artif. Intell. Med. 99, 101694 (2019).
  * [15] Pires, R. et al., “A data-driven approach to referable diabetic retinopathy detection,” Artif. Intell. Med. 96, 93–106 (2019).
  * [16] Zhou, K. et al., “Fundus image quality-guided diabetic retinopathy grading,” in [Computational Pathology and Ophthalmic Medical Image Analysis ], 245–252, Springer (2018).
  * [17] Islam, M. M. et al., “Deep learning algorithms for detection of diabetic retinopathy in retinal fundus photographs: A systematic review and meta-analysis,” Comput. Methods Programs Biomed. 191, 105320 (2020).
  * [18] Decencière, E., , et al., “Feedback on a publicly distributed image database: the Messidor database,” IAS 33(3), 231–234 (2014).
  * [19] Sahlsten, J. et al., “Deep learning fundus image analysis for diabetic retinopathy and macular edema grading,” Sci. Rep. 9(1), 1–11 (2019).
  * [20] Li, F., Wang, Y., Xu, T., Dong, L., Yan, L., Jiang, M., Zhang, X., Jiang, H., Wu, Z., and Zou, H., “Deep learning-based automated detection for diabetic retinopathy and diabetic macular oedema in retinal fundus photographs,” Eye 36(7), 1433–1441 (2022).
  * [21] Graham, B., “Kaggle diabetic retinopathy detection competition report,” University of Warwick , 24–26 (2015).
  * [22] Liu, R. et al., “DeepDRiD: diabetic retinopathy—grading and image quality estimation challenge,” Patterns , 100512 (2022).
  * [23] Porwal, P. et al., “Indian diabetic retinopathy image dataset (IDRiD): a database for diabetic retinopathy screening research,” Data 3(3), 25 (2018).
  * [24] Benítez, V. E. C. et al., “Dataset from fundus images for the study of diabetic retinopathy,” Data Br. 36, 107068 (2021).
  * [25] Cen, L.-P., , et al., “Automatic detection of 39 fundus diseases and conditions in retinal photographs using deep neural networks,” Nat. Commun. 12(1), 1–13 (2021).
  * [26] Kauppi, T. et al., “The DIARETDB1 diabetic retinopathy database and evaluation protocol,” in [BMVC ], 1, 1–10 (2007).
  * [27] He, K. et al., “Deep residual learning for image recognition,” in [CVPR ], 770–778 (2016).
  * [28] Bellemo, V., Lim, Z. W., Lim, G., Nguyen, Q. D., Xie, Y., Yip, M. Y., Hamzah, H., Ho, J., Lee, X. Q., Hsu, W., et al., “Artificial intelligence using deep learning to screen for referable and vision-threatening diabetic retinopathy in africa: a clinical validation study,” The Lancet Digital Health 1(1), e35–e44 (2019).
  * [29] Hacisoftaoglu, R. E., Karakaya, M., and Sallam, A. B., “Deep learning frameworks for diabetic retinopathy detection with smartphone-based retinal imaging systems,” Pattern Recognit. Lett. 135, 409–417 (2020).
  * [30] Dai, L. et al., “A deep learning system for detecting diabetic retinopathy across the disease spectrum,” Nat. Commun. 12(1), 1–11 (2021).
  * [31] Wilson, E. B., “Probable inference, the law of succession, and statistical inference,” Journal of the American Statistical Association 22(158), 209–212 (1927).
  * [32] Fu, R. et al., “Axiom-based grad-cam: Towards accurate visualization and explanation of CNNs,” arXiv preprint arXiv:2008.02312 (2020).
  * [33] Voets, M., Møllersen, K., and Bongo, L. A., “Reproduction study using public data of: Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs,” PloS one 14(6), e0217541 (2019).
  * [34] Fu, H. et al., “Evaluation of retinal image quality assessment networks in different color-spaces,” in [MICCAI ], 48–56, Springer (2019).
  * [35] Goyal, A. et al., “Revisiting point cloud shape classification with a simple and effective baseline,” in [ICML ], 3809–3820, PMLR (2021).
  * [36] Araújo, T. et al., “DR— GRADUATE: uncertainty-aware deep learning-based diabetic retinopathy grading in eye fundus images,” Med. Image Anal. 63, 101715 (2020).


upper/lower bound on the cardinality of $Y$ which is an $I_{C}$-free-
code-$J_{D}$-design. The following is a formal consequence of (5.6), stating
the relation with classic theory [6, Section 3].

######  Theorem 5.5.

The Mac-Williams transformation $Q^{\vee}|_{{{I}^{{\kern-0.74997pt}\wedge}}}$
maps (5.7) to (5.8):

$\sum_{i\in{{{I}^{{\kern-0.74997pt}\wedge}}}}a_{i}(Y)\cdot
i\mapsto\sum_{j\in{{{J}^{{\kern-0.74997pt}\wedge}}}}b_{j}(Y){\operatorname{ev}}_{j}.$

In concrete,

$b_{j}(Y)=\sum_{i\in{{{I}^{{\kern-0.74997pt}\wedge}}}}a_{i}(Y)Q_{j}(i).$

######  Remark 5.6.

The meaning of the values (5.7) is clear, and called the inner distribution
[6, Section 3.1] (up to a factor of $\\#Y$), but there, the meaning of the
values $b_{j}(Y)=\sum_{i\in{{{I}^{{\kern-0.74997pt}\wedge}}}}a_{i}(Y)Q_{j}(i)$
is not clear, just the semi-positivity is proved. Now the value
$b_{j}(Y)=||\operatorname{avg}_{Y}^{j}||_{{\operatorname{HS}}}^{2}$ has a
clear interpretation. On the space $C({{{X}^{{\kern-1.04996pt}\wedge}}})_{j}$,
the operator norm of $\operatorname{avg}_{Y}^{j}$ is
$||\operatorname{avg}_{Y}^{j}||_{{\operatorname{HS}}}$, since

$\sup_{f\in
C({{{X}^{{\kern-0.74997pt}\wedge}}})_{j}\setminus\\{0\\}}\frac{|\operatorname{avg}_{Y}^{j}(f)|}{||f||}=\sup_{f\in
C({{{X}^{{\kern-0.74997pt}\wedge}}})_{j}\setminus\\{0\\}}\frac{(f,\overline{\operatorname{avg}}_{Y}^{j})}{||f||}\leq\frac{||f||\cdot||\operatorname{avg}_{Y}^{j}||}{||f||}=||\operatorname{avg}_{Y}^{j}||,$

and the equality holds for $f=\operatorname{avg}_{Y}^{j}$. This may be
interpreted as the worst-case error in approximating the integration for
$C(X)_{j}$ by $\operatorname{avg}_{Y}$ (where the true integration value is
zero for $j\neq j_{0}$, because of the orthogonality to the constant functions
$C(X)_{j_{0}}$), see a comprehensive book by Dick-Pillichshammer [7,
Definition 2.10] on quasi-Monte Carlo integration. Another remark: the
injectivity (5.1) implies that $Y$ and $Y^{\prime}$ have the same inner
distribution

$\\#((Y\times
Y)\cap{{{R}^{{\kern-1.04996pt}\wedge}}}^{-1}(i))=\\#((Y^{\prime}\times
Y^{\prime})\cap{{{R}^{{\kern-1.04996pt}\wedge}}}^{-1}(i))\mbox{ for all
$i\in{{{I}^{{\kern-1.04996pt}\wedge}}}$}$

if and only if

$||\operatorname{avg}_{Y}^{j}||_{{\operatorname{HS}}}=||\operatorname{avg}_{Y^{\prime}}^{j}||_{{\operatorname{HS}}}\mbox{
for all $j\in{{{J}^{{\kern-1.04996pt}\wedge}}}$}.$

###  5.3. $(t,m,s)$-nets and $(t,s)$-sequences

The $(t,m,s)$-nets are point sets for quasi-Monte Carlo integration, well-
studied, see [18]. For $(t,m,s)$-nets, the LP bound by ordered Hamming schemes
introduced by Martin-Stinson [16] yields strong lower bounds on the
cardinality of the point set $P$, see [16] and Bierbrauer [3] for example. We
do not have such an application, but profinite association schemes may be used
to formalize the notion of $(t,s)$-sequence [18], as mentioned at the last of
this section.

The purpose of this section is to reprove some results of Marin-Stinson, and
see a relation to pro-ordered Hamming schemes. The main result Corollary 5.19
of this section is proved in Martin-Stinson[16, Theorem 3.4] by using the
notion of dual schemes and weight enumerator polynomials. Here we shall give a
self-contained proof avoiding the use of these tools. Our proof matches to the
formalization in Section 5.2, relying on the average functional
$\operatorname{avg}_{Y}$ introduced in Definition 5.1.

Recall the kernel scheme in Definition 4.19, where the set of alphabets is
$Z_{v}:={\mathbb{Z}}/v$, and $X_{n}=Z_{v}^{n}$,
$I_{n}=\\{1,2,\ldots,n\\}\cup\\{\infty\\}$,

${{\operatorname{bot}}}:X_{n}\to I_{n}$

in Definition 4.24 give the kernel scheme as a translation scheme. We shall
identify a point in $X_{n}$ with a point in the real interval $[0,1]$, by

$(x_{1},\ldots,x_{n})\mapsto 0.x_{1}x_{2}\cdots x_{n}\in[0,1),$ (5.9)

where the right expression means the $v$-adic decimal expansion. Then, we have
a map

$X_{n}^{s}\to[0,1]^{s}.$

We shall define $(t,m,s)$-nets following Niederreiter [17, Definition 4.1]. An
elementary interval of type $(d_{1},\ldots,d_{s})$ is a subset of $[0,1]^{s}$
of the form

$\prod_{i=1}^{s}[a_{i}v^{-d_{i}},(a_{i}+1)v^{-d_{i}}),$

where $a_{i}$ and $d_{i}$ are non-negative integers such that
$a_{i}<v^{d_{i}}$. Its volume is

$\prod_{i}^{s}v^{-d_{i}}=v^{-\sum_{i=1}^{s}d_{i}}.$

######  Definition 5.7.

Let $0\leq t\leq m$ be integers. A $(t,m,s)$-net in base $v$ is a multi-subset
$P\subset[0,1]^{s}$ consisting of $v^{m}$ points, such that every elementary
interval of type $(d_{1},\ldots,d_{s})$ with $d_{1}+\cdots+d_{s}=m-t$ (hence
volume $v^{t-m}$) contains exactly $v^{t}$ points of $P$.

Our purpose is to state these conditions in terms of designs $Y\subset
X_{n}^{s}$ for a finite multi-subset $Y$.

######  Definition 5.8.

A finite multi-subset $Y$ in $X$ is said to be uniform on $X$ if
$\\#(Y\cap\\{x\\})$ is constant for $x\in X$, that is, $\\#g^{-1}(x)$ is
constant for $x\in X$ if $Y$ is defined by a map $g:Z\rightarrow X$.

######  Definition 5.9.

For an $s$-tuple of non-negative integers ${\mathbf{d}}=(d_{1},\ldots,d_{s})$
with each component less than or equal to $n$, define

$X_{\mathbf{d}}:=\prod_{i=1}^{s}Z_{v}^{d_{i}},$

and the projection

${\operatorname{pr}}_{\mathbf{d}}:X_{n}^{s}\to X_{\mathbf{d}}$

by taking the left most $d_{i}$ components of the $i$-th $X_{n}$ in
$X_{n}^{s}$, for $i=1,\ldots,s$.

######  Definition 5.10.

A non-empty multi-subset $Y\subset X_{n}^{s}$ is ${\mathbf{d}}$-balanced, if
the image of $Y$ in $X_{\mathbf{d}}$ by the map $X_{n}^{s}\to X_{\mathbf{d}}$
is uniform on $X_{\mathbf{d}}$.

######  Proposition 5.11.

A multi-subset $Y\subset X_{n}^{s}$ of cardinality $v^{m}$ gives a
$(t,m,s)$-net if and only if it is ${\mathbf{d}}$-balanced for any non-
negative tuples ${\mathbf{d}}=(d_{1},\ldots,d_{s})$ with
$d_{1}+\cdots+d_{s}=m-t$.

The proof is immediate when we consider the mapping (5.9), since then there is
a natural one-to-one correspondence between $X_{\mathbf{d}}$ and the set of
elementary intervals of type $(d_{1},\ldots,d_{s})$. We may state the
conditions in terms of the dual group.

######  Lemma 5.12.

Let $X$ be a finite abelian group, and $Y$ a finite multi-subset in $X$. For
$f\in C(X)$, define

$I(f):=\operatorname{avg}_{X}(f)=\frac{1}{\\#X}\sum_{x\in X}f(x).$

(This is the true integral of $f$ over the finite set $X$). The following are
equivalent.

  1. (1)

$I(f)=\operatorname{avg}_{Y}(f)$ holds for any $f\in C(X)$.

  2. (2)

The finite multi-subset $Y$ is uniform on $X$.

  3. (3)

$\operatorname{avg}_{Y}(\xi)=0$ holds for any non-trivial
$\xi\in{{\check{X}}}$.

###### Proof.

Suppose that the first condition holds. Let $\chi_{x}$ be the indicator
function of $x\in X$. The condition
$I(\chi_{x})=\operatorname{avg}_{Y}(\chi_{x})$ implies
$\frac{1}{\\#X}=\frac{\\#(Y\cap\\{x\\})}{\\#Y}$, hence $Y$ is uniform on $X$.
The converse is obvious.

Suppose again the first condition. For any non-trivial $\xi$, $I(\xi)=0$. This
implies that $\operatorname{avg}_{Y}(\xi)=0$. For the converse, we assume the
third condition. Note that ${{\check{X}}}$ is a base of $C(X)$, so it suffices
to show that $I(\xi^{\prime})=\operatorname{avg}_{Y}(\xi^{\prime})$ holds for
any $\xi^{\prime}\in{{\check{X}}}$. This holds for non-trivial $\xi$, and for
the trivial $\xi=1$, $I(1)=1=\operatorname{avg}_{Y}(1)$. ∎

######  Lemma 5.13.

The dual ${{\check{X}}}_{\mathbf{d}}$ of $X_{\mathbf{d}}$ can be identified
with the set of characters

$\\{\xi:=(\xi_{1},\ldots,\xi_{s})\in\check{X_{n}}^{s}\mid{{\operatorname{top}}}(\xi_{i})\leq
d_{i}\mbox{ for }i=1,\ldots,s\\}$

(see Definition 4.25 for the notation ${{\operatorname{top}}}$).

######  Corollary 5.14.

For a multi-subset $Y\subset X_{n}^{s}$, the composition $Y\to X_{n}^{s}\to
X_{\mathbf{d}}$ is uniform if and only if

$\operatorname{avg}_{Y}(\xi)=0$

holds for any non-trivial $\xi\in{{\check{X}}}_{\mathbf{d}}$.

######  Corollary 5.15.

A multi-subset $Y\subset X_{n}^{s}$ of cardinality $v^{m}$ is a $(t,m,s)$-net
if and only if

$\operatorname{avg}_{Y}(\xi)=0$

holds for any $\xi\in{{\check{X}}}_{\mathbf{d}}\setminus\\{1\\}$ and any non-
negative ${\mathbf{d}}=(d_{1},\ldots,d_{s})$ satisfying
$\sum_{i=1}^{s}d_{i}=m-t$.

Niederreiter[17] and Rosenbloom-Tsfasman[23] defined the notion of NRT-weight,
see also Niederreiter and Pirsic[20]:

######  Definition 5.16.

Define the NRT-weight $\operatorname{wt}$ by the composition

$\operatorname{wt}:{{\check{X}}}_{n}^{s}\stackrel{{\scriptstyle{{\operatorname{top}}}^{s}}}{{\to}}J_{n}^{s}\stackrel{{\scriptstyle{{\operatorname{sum}}}}}{{\to}}{\mathbb{N}},$

where ${{\operatorname{sum}}}$ is a function taking the sum of the $s$
coordinates and ${{\operatorname{top}}}$ is defined in Definition 4.25.

######  Theorem 5.17.

Let $Y\subset X_{n}^{s}$ be a multi-subset of cardinality $v^{m}$. Then it is
a $(t,m,s)$-net if and only if for any $\xi\in{{\check{X}}}_{n}^{s}$ with
$\operatorname{wt}(\xi)\leq m-t$ and $\xi\neq 1$,
$\operatorname{avg}_{Y}(\xi)=0$.

This follows from Corollary 5.15. Following Martin-Stinson[16], let us define

${{\operatorname{shape}}}:J_{n}^{s}\to J_{n}^{s}/S_{s}.$

In Proposition 4.33 and Lemma 4.26, we observed that for $\bar{j}\in
J_{n}^{s}/S_{s}$, the corresponding eigenspace $C(X_{n}^{s})_{\bar{j}}$ is
spanned by the characters in the inverse image of $\bar{j}$ by

${{\check{X}}}_{n}^{s}\stackrel{{\scriptstyle{{\operatorname{top}}}^{s}}}{{\to}}J_{n}^{s}\to
J_{n}^{s}/S_{s}.$

It is easy to see that $\operatorname{wt}$ factors as

${{\check{X}}}_{n}^{s}\stackrel{{\scriptstyle{{\operatorname{top}}}^{s}}}{{\to}}J_{n}^{s}\stackrel{{\scriptstyle{{\operatorname{shape}}}}}{{\to}}J_{n}^{s}/S_{s}\stackrel{{\scriptstyle
0pt}}{{\to}}{\mathbb{N}},$

where the definition of $0pt$ is clear from the context. (We remark that $0pt$
is denoted by “height” in Martin-Stinson[16, P.337], but we changed the
notation to avoid the collision with a standard use of “height” in the context
of the Young diagrams.) Thus,

$\displaystyle\\{\xi\in{{\check{X}}}_{n}^{s}\mid\operatorname{wt}(\xi)\leq
m-t\\}$ $\displaystyle=$ $\displaystyle\\{\xi\in{{\check{X}}}_{n}^{s}\mid
0pt({{\operatorname{shape}}}({{\operatorname{top}}}^{s}(\xi)))\leq m-t\\}.$

The condition for $Y\subset X_{n}^{s}$ to be a $(t,m,s)$-net is that
$\operatorname{avg}_{Y}(\xi)$ vanishes for any nontrivial $\xi$ in the left
hand side (Theorem 5.17), which is equivalent to the vanishing in the right
hand side, namely, $\operatorname{avg}_{Y}(\xi)=0$ for each of $\xi\in
C(X_{n}^{s})_{\bar{j}}$ for $\bar{j}\in J_{n}^{s}/S_{s}\setminus\\{j_{0}\\}$,
$0pt(\bar{j})\leq m-t$. This is equivalent to
$\operatorname{avg}_{Y}^{\bar{j}}=0$ (the left hand side is the representation
of $\operatorname{avg}_{Y}$ in the subspace $C(X_{n}^{s})_{\bar{j}}$ defined
in (5.3) with property (5.4)) for these $\bar{j}$.

Now we proved [16, Theorem 3.4]:

######  Theorem 5.18.

Let $Y\subset X_{n}^{s}$ be a multi-subset of cardinality $v^{m}$. Then, $Y$
is a $(t,m,s)$-net if and only if $\operatorname{avg}_{Y}^{\bar{j}}=0$ for any
$\bar{j}\neq j_{0}$, $0pt(\bar{j})\leq m-t$.

In terms of the designs (see Section 5.2), we may state

######  Corollary 5.19.

Define

$J_{D}:=\\{\bar{j}\in J_{n}^{s}/S_{s}\mid\bar{j}\neq j_{0},\ 0pt(\bar{j})\leq
m-t\\}.$

Then, $Y\subset X_{n}^{s}$ is a $J_{D}$-design if and only if it is a
$(t,m,s)$-net in $X_{n}^{s}$.

By this, Delsarte’s LP method works, see [16]. The next is an example where we
may use a profinite association scheme to make a concept of $(t,s)$-sequence
inside the design theory, (see [18, Definition 4.2] and [19, Remark 2.2]).

Let us consider the pro-ordered Hamming scheme
$\overrightarrow{H}(s,\infty,v)=({{{X}^{{\kern-1.04996pt}\wedge}}},{{{R}^{{\kern-1.04996pt}\wedge}}})$
defined in Proposition 4.35. Note that
${{{X}^{{\kern-1.04996pt}\wedge}}}=(\varprojlim
X_{n})^{s}=\varprojlim(X_{n}^{s})=(Z_{v}^{{\mathbb{N}}_{>0}})^{s}$. For each
$n$, we denote by $\pi_{n}$ the surjection from
${{{X}^{{\kern-1.04996pt}\wedge}}}$ onto $X_{n}^{s}$. That is,

$\pi_{n}(x^{1},\dots,x^{s}):=([x^{1}]_{n},\dots,[x^{s}]_{n})$ (5.10)

for each
$(x^{1},\dots,x^{s})\in{{{X}^{{\kern-1.04996pt}\wedge}}}=(Z_{v}^{{\mathbb{N}}_{>0}})^{s}$,
where we put $[(x_{1},\dots)]_{n}:=(x_{1},\dots,x_{n})$ for each
$(x_{1},\dots)\in Z_{v}^{{\mathbb{N}}_{>0}}$.

######  Definition 5.20.

A sequence of points $p_{0},p_{1},\ldots\in{{{X}^{{\kern-1.04996pt}\wedge}}}$
is a $(t,s)$-sequence in base $v$, if for any integers $k$ and $m>t$, the
point set $Y_{k,m}\subset X_{m}^{s}\subset[0,1)^{s}$ consisting of the
$\pi_{m}(p_{j})$ with $kv^{m}\leq j<(k+1)v^{m}$ (considered as a multi-set) is
a $(t,m,s)$-net in base $v$.

For a comparison with a standard Definition 5.22, see Remark 5.23. We may
state the condition of $(t,s)$-sequences in terms of the designs in the pro-
ordered Hamming scheme $\overrightarrow{H}(s,\infty,v)$. Recall that
${{{J}^{{\kern-1.04996pt}\wedge}}}={\mathbb{N}}^{s}/S_{s}$, and we may define
$0pt$ so that the composition

${\mathbb{N}}^{s}\to{{{J}^{{\kern-1.04996pt}\wedge}}}=({\mathbb{N}}^{s}/S_{s})\stackrel{{\scriptstyle
0pt}}{{\to}}{\mathbb{N}}$

maps $(j_{1},\ldots,j_{s})\mapsto\sum_{i=1}^{s}j_{i}$.

######  Theorem 5.21.

In the pro-ordered Hamming scheme, we define

$J_{D_{\ell}}:=\\{h\in({\mathbb{N}}^{s}/S_{s})={{{J}^{{\kern-1.04996pt}\wedge}}}\mid
0pt(h)\leq\ell,h\neq j_{0}\\}.$

A sequence of points $p_{0},p_{1},\ldots\in{{{X}^{{\kern-1.04996pt}\wedge}}}$
is a $(t,s)$-sequence in base $v$, if and only if for any integers $k$ and
$m>t$, the point set $Y_{k,m}$ consisting of the $\pi_{m}(p_{j})$ with
$kv^{m}\leq j<(k+1)v^{m}$ is a $J_{D_{m-t}}$-design.

In the original definition, the sequence is taken in an infinite set
$[0,1]^{s}$, where infinite precision is necessary. Because of this
infiniteness, a fixed finite scheme would not be able to describe the notion
of $(t,s)$-sequence. Our profinite association scheme is a tool to deal with
the infiniteness.

We close our paper with a discussion on the definitions of $(t,s)$-sequences.
The following is a definition by Niederreiter-Özbudak[19, Definition 2.2] (to
be precise, its special case), which is a slight variant of the original
Niederreiter’s one: [18, Definition 4.2].

######  Definition 5.22.

Let $p_{0},p_{1},\ldots\in[0,1]^{s}$ be a sequence of points, with prescribed
$v$-adic expansions. (In other words, each $p_{j}$ is considered as an element
in ${{{X}^{{\kern-1.04996pt}\wedge}}}$.) It is a $(t,s)$-sequence in base $v$,
if for any integers $k$ and $m>t$, the point set $P_{k,m}$ consisting of the
$\pi_{m}(p_{j})$ with $kv^{m}\leq j<(k+1)v^{m}$ is a $(t,m,s)$-net in base
$v$.

######  Remark 5.23.

It is easy to see that Definitions 5.20 and 5.22 are the same. In Definition
5.22, we use $\pi_{m}$ in (5.10) for $p_{j}$, where $p_{j}$ is an element of
${{{X}^{{\kern-1.04996pt}\wedge}}}$. In fact, $\pi_{m}$ is not well-defined
for $[0,1]^{s}$, because of the non-injectivity of

$(Z_{v}^{{\mathbb{N}}_{>0}})\to[0,1],$

namely, for base $v=2$, for example, $0.0111\cdots=0.1000\cdots$. One way to
avoid this subtle problem may be to require the $v$-adic expansion of a real
number to be of the latter type, namely, to avoid expansions consisting of all
$(v-1)$’s after some digit. However, in important constructions such as in
Niederreiter-Xing[21], points with the former type of expansions may appear.
Thus, to define the notion of $(t,s)$-sequences as a sequence in $[0,1]^{s}$
has a subtle problem. Definition 5.20 would be a natural definition of a
$(t,s)$-sequence. In other words, the notion would be better defined in terms
of ${{{X}^{{\kern-1.04996pt}\wedge}}}$ rather than $[0,1]^{s}$. This is
essentially stated in [19, Remark 2.2]. Of course, a large part of researchers
would prefer to work in $[0,1]^{s}$.

## References

  * [1] E. Bannai and T. Ito.  Algebraic Combinatorics I: Association Schemes.  Benjamin / Cummings, Calfornia, 1984.
  * [2] A. Barg and M. Skriganov.  Association schemes on general measure spaces and zero-dimensional abelian groups.  Adv. Math., 281:142–247, 2015.
  * [3] J. Bierbrauer.  A direct approach to linear programming bounds for codes and tms-nets.  Designs, Codes and Cryptography, 42(2):127–143, 2007.
  * [4] A. W. Brouwer, A. M. Cohen, and A. Neumaier.  Distance-regular graphs, volume MATHE3, 18 of Ergebnisse der Mathematik und ihrer Grenzgebiete. 3. Folge / A Series of Modern Surveys in Mathematics.  Springer-Verlag, New York, Heidelberg, Berlin, 1989.
  * [5] J. R. Choksi.  Inverse limits of measure spaces.  London Mathematical Society, s3-8:321–342, 1958.
  * [6] P. Delsarte.  An algebraic approach to the association schemes of coding theory.  Philips Res. Rep. Suppl., 10:i–vi and 1–97, 1973.
  * [7] J. Dick and F. Pillichshammer.  Digital Nets and Sequences: Discrepancy Theory and Quasi-Monte Carlo Integration.  Cambridge University Press, Cambridge, 2010.
  * [8] C. French.  Functors from association schemes.  J. Combin. Theory Ser. A, 120:1141 – 1165, 2013.
  * [9] C. Godsil.  Generalized hamming schemes.  arXiv:1011.1044.
  * [10] P.R. Halmos.  Measure Theory, volume 18 of Graduate Texts in Mathematics.  Springer-Verlag, New York, Heidelberg, Berlin, 1970.
  * [11] A. Hanaki.  A category of association schemes.  J. Combin. Theory Ser. A, 117:1207 – 1217, 2010.
  * [12] H. Kajiura, M. Matsumoto, and T. Okuda.  Non-existence and construction of pre-difference sets, and equi-distributed subsets in association schemes.  Graphs and Combinatorics, 37:1531–1544, 2021.
  * [13] H. Kurihara and T. Okuda.  A MacWilliams type theorem and Delsarte’s theory for codes and designs on compact homogeneous spaces.  in preparation.
  * [14] W.C. Lang.  Orthogonal wavelets on the cantor dyadic group.  SIAM J. Math. Anal., 27:305–312, 1996.
  * [15] S. MacLane.  Categories for the Working Mathematician, volume 5 of Graduate Texts in Mathematics.  Springer-Verlag, New York, Heidelberg, Berlin, 2nd edition, 1998.
  * [16] W.J. Martin and D.R. Stinson.  Association schemes for ordered orthogonal arrays and (t, m, s)-nets.  Canadian Journal of Mathematics, 51:326–346, 1999.
  * [17] H. Niederreiter.  Low-discrepancy point sets.  Monatsh. Math., 102:155–167, 1986.
  * [18] H. Niederreiter.  Random Number Generation and Quasi-Monte Carlo Methods.  CBMS-NSF, Philadelphia, Pennsylvania, 1992.
  * [19] H. Niederreiter and F. Özbudak.  Low-discrepancy sequences using duality and global function fields.  Acta Arith., 130:79–97, 2007.
  * [20] H. Niederreiter and G. Pirsic.  Duality for digital nets and its applications.  Acta Arith., 97:173–182, 2001.
  * [21] H. Niederreiter and C. P. Xing.  Low-discrepancy sequences and global function fields with many rational places.  Finite Fields and Their Applications, 2:241–273, 1996.
  * [22] L. Ribes and P. Zalesskii.  Profinite Groups.  Springer-Verlag, Berlin Heidelberg, 2000.
  * [23] M. Yu. Rosenbloom and M. A. Tsfasman.  Codes for the m-metric.  Problems of Information Transmission, 33:55–63, 1997.
  * [24] M. Takeuchi.  Modern spherical functions, volume 135 of Translations of Mathematical Monographs.  American Mathematical Society, Providence, 1994.
  * [25] P. H. Zieschang.  An Algebraic Approach to Association Schemes, volume 1628 of Lecture Notes in Mathematics.  Springer-Verlag, Berlin Heidelberg, 1996.