11institutetext: Monash University 11email<EMAIL_ADDRESS>
11email<EMAIL_ADDRESS>
22institutetext: Cerenaut 22email<EMAIL_ADDRESS>https://cerenaut.ai

# Graceful task adaptation with a bi-hemispheric RL agent

Grant Nicholas 11 Levin Kuhlmann 11 0000-0002-5108-6348 Gideon Kowadlo 1122
0000-0001-6036-1180

###### Abstract

In humans, responsibility for performing a task gradually shifts from the
right hemisphere to the left. The Novelty-Routine Hypothesis (NRH) states that
the right and left hemispheres are used to perform novel and routine tasks
respectively, enabling us to learn a diverse range of novel tasks while
performing the task capably. Drawing on the NRH, we develop a reinforcement
learning agent with specialised hemispheres that can exploit generalist
knowledge from the right-hemisphere to avoid poor initial performance on novel
tasks. In addition, we find that this design has minimal impact on its ability
to learn novel tasks. We conclude by identifying improvements to our agent and
exploring potential expansion to the continual learning setting.

##  1 Introduction

Despite many high-profile successes of Deep Reinforcement Learning (RL), RL-
agents still struggle with sample efficiency and generalisation [2]. Human
beings do not experience such problems and are capable continuous learners,
able to acquire diverse skills over their lifetimes [12].

Research in neuroscience highlights the importance of hemispheric
specialisation in the human brain for learning [7, 8, 6, 15]. The right-
hemisphere is identified as focusing on ‘global’ phenomena, being competent in
novel scenarios and for exploration, while the left-hemisphere focuses on
‘local’ phenomena and learns to specialise [7, 8, 6, 19]. When a novel task is
learnt, a uni-directional shift occurs whereby responsibility for the task
moves from right to left hemisphere during learning [8]; the key to the
Novelty Routine Hypothesis (NRH) proposed in [8].

Consequently, we draw on the NRH and related neuroscientific theories to
develop an RL-agent with specialised hemispheres. In doing so, we follow the
approach of other significant discoveries in AI that use human neuroscience as
inspiration for AI algorithm development [18, 10]. We hypothesise that our bi-
hemispheric architecture will help RL-agents learn novel tasks, while avoiding
the poor initial performance typically observed for agents trained from
scratch. Essentially, bi-hemispheric agents should perform well in the initial
stages of learning by drawing on the generalist right-hemisphere, while
training the left-hemisphere to specialise in the task. This may have benefits
for continual learning, where agents encounter streams of novel tasks.

Little research in RL explores the application of bi-hemispheric brain
structure. Mixture of Experts (MoE) [11] and other multiple network
architectures are neuroscience inspired, but do not focus on hemispheric
specialisation [20, 25]. Furthermore, while Actor-Critic based approaches can
use dual networks to output actions and values, the aim is to stabilise
estimates of the policy gradient, rather than mimic brain structure [14]. In
supervised learning, Rajagopalan et al. [21] explored a bi-hemispheric
ensemble model with specialist left and generalist right hemispheres to
classify images. Neuroscientists also use bi-hemispheric deep neural networks
to test hypotheses about the human brain [23, 27, 19, 3]. This project is a
novel application of a bi-hemispheric architecture to RL.

##  2 Agent design

We constructed a bi-hemispheric agent inspired by the NRH; a right-hemisphere
has generalist capabilities and a left-hemisphere learns to specialise in
tasks. When encountering a novel task, a gating network assigns responsibility
to either the right or left hemisphere to manage performance. The agent should
draw on generalist skills in the right-hemisphere to achieve improved initial
performance over an agent trained from scratch. This should not interfere with
using the left-hemisphere to learn and eventually perform the novel task
independently. Hence, the agent is assessed on the objectives:

  1. 1.

_initial_ performance better than an agent trained from scratch

  2. 2.

_final_ left-hemisphere performance as good as an agent trained from scratch

###  2.1 Network architecture

The bi-hemispheric agent consists of two hemispheres and a gating network,
each comprising a Recurrent Neural Network (RNN), with Gated Recurrent Units
(GRU). Each hemisphere outputs an action and a value estimate. For simplicity,
we give right and left hemispheres identical architectures and omit inter-
hemispheric connections (i.e. Corpus Callosum), despite being neurologically
inaccurate [9, 3, 19, 6]. The network architecture is shown in Figure
S.LABEL:fig:network_arch.

###  2.2 Gating network

The gating network assigns ‘responsibility’ to each hemisphere using ‘gating
values’, defined as the proportional contribution of that hemisphere to the
agent’s action and value. The responsibility of the right-hemisphere is thus:

$P^{right}=1-P^{left}\;where\;P^{right},P^{left}\in[0,1]$ (1)

The value estimate is a linear combination of the right and left hemisphere
value outputs, weighted by the responsibility of each hemisphere:

$V^{bihem}=P^{right}V^{right}+P^{left}V^{left}$ (2)

Similarly, our bi-hemipheric agent’s policy samples actions from a Gaussian
distribution whose mean is also a linear combination of right and left
hemisphere means. The variance is $\sigma^{2}$ learnt by the bi-hemispheric
agent.

$\pi^{bihem}\sim\mathcal{N}(P^{right}\mu^{right}+P^{left}\mu^{left},\sigma^{2})$
(3)

The gating network takes as input the gating values from the last timestep and
the ‘value-estimate error’ $\varepsilon$ for each hemisphere $h$, defined as
$\varepsilon^{h}=V^{h}_{t}-r_{t}$. This gives the gating network information
on how well the hemisphere’s can predict task rewards. Combining this with
previous gating values lets the gating network learn relationships between
hemisphere performance and responsibility.

Finally, to encourage hemispheric shift we incorporate an additive penalty
into the bi-hemispheric agent’s loss function. This term penalises losses when
the right gating value is large:

$\beta(\frac{P^{right}}{P^{left}})^{\alpha}$ (4)

Where $\alpha$ and $\beta$ are tuneable parameters. For details see Section
S.LABEL:supp:loss.

###  2.3 Generalisation and specialisation

We trained right and left hemispheres separately using different training
processes to induce generalisation and specialisation respectively. First, we
trained the right-hemisphere using the $RL^{2}$ meta-learning algorithm [4,
26]. Second, we constructed the bi-hemispheric network by freezing the weights
of the right-hemisphere and then combined it with a randomly initialised left-
hemisphere and gating network. Finally, the left-hemisphere and gating
networks were trained together using the standard RL objective, which
maximises expected discounted rewards [24]. The left-hemisphere ‘specialises’
in a task simply by being trained to perform that specific task.

We used meta-learning for the right, as it offers both the ability to
generalise to a task-distribution _and_ adapt quickly to novel tasks [5]; both
important for strong initial performance. $RL^{2}$ is exceptional at the
latter – sometimes achieving zero-shot adaptation [4, 26, 13]. In addition
$RL^{2}$ is a memory-based meta-RL algorithm, which have been identified as
Bayesian-optimal learners as they can optimally trade off exploration and
exploitation in uncertain environments [30, 17]. This fits with the NRH’s view
of the right-hemisphere as guiding exploratory behaviour when learning novel
tasks [8].

##  3 Experiments

Our implementation is available at
https://github.com/gdubbs100/right_left_brain_rl The experiments were carried
out in two stages, shown in Figure 1. We first meta-trained the right-
hemisphere and a baseline agent on 3 tasks. We then created the bi-hemispheric
agent and trained and evaluated all agents on an expanded set of tasks over 5
seeds.

Figure 1: Experiment overview

###  3.1 Dataset

We used Meta-world [29], a meta-learning/multi-task benchmark comprised of 50
tasks that involve manipulation of objects using a Sawyer robot arm. We chose
Meta-world because of its diverse tasks with shared action and state spaces.

Meta-world tasks are designed to vary in a non-trivial manner, to allow the
evaluation of generalisation capabilities in RL-agents. Meta-world defines
differences between tasks as parametric or non-parametric. Parametric
variation involves variation in real-valued parameters such as object or goal
locations, e.g. the location of a ball where the task objective is for an arm
to reach said ball. In contrast, non-parametric variation involves qualitative
differences between tasks, e.g. the difference between opening a door and
lifting a ball [29]. Each of Meta-world’s 50 tasks exhibit non-parametric
variation between each other. Further, each task is comprised of 50 sub-tasks
which exhibit parametric variation between each other. We use Meta-world’s
definitions of variation to divide the experiment tasks into three tiers
(Section S.LABEL:supp:tasks) and classify the novelty of a task relative to
the meta-training tasks:

  1. 1.

Tier-1 tasks have only parametric variation i.e. sub-tasks with different goal
and object locations.

  2. 2.

Tier-2 tasks include similar tasks to Tier-1, but with minor non-parametric
variation e.g. the addition of a wall obstacle to a task from meta-training.

  3. 3.

Tier-3 tasks exhibit complete non-parametric variation from Tier-1. These
tasks help us understand the limits of our approach.

###  3.2 Baselines

We evaluated three baselines. First, the ‘left-only’ baseline is a randomly
initialised agent with the same architecture as the left-hemisphere of the bi-
hemispheric agent, but double the size of the GRU (i.e. increased neurons to
256 from 128). We double GRU size to ensure the same size network as the bi-
hemispheric agents, giving them equal representational capacity and enabling a
fair comparison between agents. We also evaluated a ‘right-only’ baseline – a
meta-trained agent with identical architecture and doubled GRU size. This
baseline let us determine whether it is better to simply meta-train an agent
rather than use the bi-hemispheric architecture. Finally, we used a randomly
acting agent as a lower-bound on performance; details in Section
S.LABEL:supp:agents.

###  3.3 Training approach

####  3.3.1 Meta-training

We meta-trained the right-hemisphere and right-only baseline using the
$RL^{2}$ algorithm and PPO in the ‘outer-loop’. We used three tasks from
Tier-1 and trained each agent for 50 million environment steps using identical
hyperparameters except for GRU size. Our meta-training approach was simplified
compared to Meta-world, see Section S.LABEL:supp:simplifications.

####  3.3.2 Bi-hemispheric training

For the main experiment, we selected nine Meta-world tasks from Tier-1, 2 and
3, disjoint from Meta-training sub-tasks, to train and evaluate bi-hemispheric
agents and the left-only baseline. These tasks exhibit varying degrees of
novelty from the tasks used in meta-training. For each task, we trained for 5
million environment steps using the PPO algorithm [22], as this timeframe
allowed agents to learn each task. During training, we extracted mean rewards
and median gating values from each batch. For bi-hemispheric agents, we also
tested the left-hemisphere network as an independent agent. We made
simplifications to training compared to Meta-world, following the Continual
World approach [28]; randomly sampling from 20 tasks (instead of 50) and
making goal and object positions observable, see Section
S.LABEL:supp:simplifications.

####  3.3.3 Hyperparameter selection

For each task, we used identical hyperparameters for the left-only baseline
and all bi-hemispheric agents. We selected hyperparameters that enabled best
performance for the left-only baseline. This is a conservative approach that
enabled us to evaluate bi-hemispheric agents against the left-only baselines
at their best. Hyperparameter selection was informed by values used in Meta-
world and previous studies into PPO settings [29, 1, 16]. Gating network
parameters were chosen for good final left-hemisphere performance. We also
used Meta-world hyperparameter values to inform settings for meta-training.
Hyperparameters are in given in Section S.LABEL:supp:hyperparams.

###  3.4 Main experiments

We compared bi-hemispheric agents against baselines using ‘relative rewards’;
the ratio between rewards of two agents. A relative reward of $\geq 1$
indicates superior performance over the agent in the denominator. To assess
bi-hemispheric performance against Objectives 1 and 2, we placed bi-
hemispheric agent reward in the numerator and the left-only baseline reward in
the denominator.

####  3.4.1 Initial Relative Reward

We created a metric called Initial Relative Reward (IRR) to evaluate Objective
1. IRR measures whether a bi-hemispheric agent improves on initial performance
of the left-only baseline. We calculated IRR using the ratio of median rewards
of the first $k=1$ million environment steps for the bi-hemispheric agent over
the left-only baseline:

$IRR=\frac{Median(R_{t\leq k}^{bihem})}{Median(R_{t\leq k}^{left\;only})}$ (5)

####  3.4.2 Final Relative Reward

We created a metric called Final Relative Reward (FRR) to evaluate Objective
2. Calculating FRR is similar to IRR, except we put rewards achieved by the
left-hemisphere (on its own) in the numerator, and for the last $k=1$ million
environment steps. $T$ is the total number of environment steps.

$FRR=\frac{Median(R_{t\geq T-k}^{left\;hemisphere})}{Median(R_{t\geq
T-k}^{left\;only})}$ (6)

##  4 Results

This section focuses on the Main experiment, for meta-training, see Section
S.LABEL:supp:metares. For reference, per-step rewards for Meta-world tasks are
between 0 and 10 where 10 occurs when the agent successfully achieves the
specified goal of the task [29]. Given our objectives, we focus on rewards.
However, Meta-world tasks are often evaluated on how often an agent achieves
task success, which we include in a range of additional results plots in
Section S.LABEL:supp:main_exps.

Figure 2 shows the mean rewards achieved for bi-hemispheric agents and
baselines. For the bi-hemispheric agent and left-only baseline, mean rewards
are smoothed using a rolling median over one million environment steps. Right-
only and Random baseline median rewards are calculated by sampling sub-tasks
with replacement. We used this approach as these baselines do not learn during
evaluation, hence sampled tasks can be treated as independent.

Figure 2: Training curves. The thick line indicates the median of all seed
runs, while the shaded ribbon covers the range of min and max values.

We identify three groups of results. The first group consists of the tasks
reach-v2, push-v2, reach-wall-v2 and push-wall-v2. Here, the bi-hemispheric
agent outperforms the left-only baseline initially and achieves comparable or
better performance over the whole period. Tasks in this group are tasks where
the right-hemisphere performs strongly.

The second group consists of pick-place-v2 and bin-picking-v2. For these
tasks, the left-only baseline generally outperforms the bi-hemispheric agent.
In this group, overall rewards achieved for all agents are not very high,
especially for bin-picking-v2, and vary across seeds. Notably, right-
hemisphere performance is also poor on these tasks. This is consistent with
Meta-world benchmark results, which show PPO agents fail to learn bin-picking,
and that meta-learning agents take longer to learn pick-place-v2 compared to
reach-v2 and push-v2 [29].

The third group contains the Tier-3 tasks. Here too, the left-only baseline
outperforms the bi-hemispheric agent. Bi-hemispheric agent performance can
vary significantly across seeds, and in some cases fail to learn the task at
all. Faucet-open-v2 is particularly affected by this, with many runs
performing similarly to the left-only baseline while others failed to learn at
all. Essentially, bi-hemispheric agents struggle with this degree of novelty.

####  4.0.1 Objective 1: Initial performance

Figure 3 plots IRR over tasks and seeds. Consistent with Figure 2 we see that
the bi-hemispheric agent achieves IRR scores larger than 1 for the reach and
push families of tasks across different seeds. For other tasks, bi-hemispheric
performance does not exceed the left-only baseline. For pick-place-v2 on some
seeds the bi-hemispheric agent outperforms the left-only baseline. However,
given the initial rewards achieved for pick-place-v2 are small, this may drive
the differences in IRR scores, rather than genuinely improved performance.

Figure 3: Initial bi-hemispheric agent performance relative to left-only
baseline: IRR distribution over seeds. IRR scores of greater than 1 indicate
that initial bi-hemispheric performance exceeds the left-only baseline. Median
IRR scores which exceed 1 are shown in blue, while those below 1 are in red.
The black line at $IRR=1$ indicates parity with left-only baseline
performance.

When the right-hemisphere is competent, bi-hemispheric agents could avoid the
poor initial outcomes associated with training an agent from scratch. This
indicates that our agent design could exploit generalist skills in the right-
hemisphere, when they exist, to avoid poor initial performance.

####  4.0.2 Objective 2: Hemispheric shift

Figure 4 plots FRR across tasks and seeds. Scores greater than 1 indicate that
the final performance of the left hemisphere exceeded the final performance of
the left-only baseline. The bi-hemispheric agent only achieved FRR of greater
than 1 on push-v2 and push-wall-v2. The bi-hemispheric agent achieved FRR
close to 1 for reach tasks and for faucet-open-v2. We observed FRR scores well
below 1 for other tasks.

Figure 4: Final left-hemisphere performance relative to left-only baseline:
FRR distribution over seeds.

Overall, when the right-hemisphere performs well, we observed consistently
higher FRR scores. This suggests that the right-hemisphere may have
contributed to learning by teaching the left-hemisphere. Consequently, FRR
scores may not need to exceed 1 for a bi-hemispheric approach to be
worthwhile. If improvements to initial performance achieved by a bi-
hemispheric agent are large enough, small declines in final left-hemisphere
performance may be acceptable.

####  4.0.3 Combining Objectives 1 and 2

Figure 5 plots median IRR and FRR scores. Only push-v2 and push-wall-v2 fell
within the upper right quadrant. Reach-v2 and reach-wall-v2 achieve IRR scored
greater than 1, but have FRR scores just below 1. However, all other tasks
were located in the lower left quadrant. What differentiated the tasks in the
bottom-left quadrant from those near to the upper-right is strong right-
hemisphere performance. Essentially, when right-hemisphere performed tasks,
bi-hemispheric agents achieved stronger IRR and FRR scores.

Figure 5: Comparison of IRR vs FRR. Black lines at $IRR=1$ and $FRR=1$ divide
the plot into quadrants. The upper right quadrant is the ideal outcome where
IRR and FRR exceed left-only baseline performance and the bi-hemispheric agent
achieves Objectives 1 and 2. The lower-left quadrant indicates failure to
achieve either objective.

##  5 Discussion

The key finding is that when the right-hemisphere contains generalist skills
relevant to a task, the agent can exploit them to improve initial performance
on that task. Furthermore, doing so does not significantly impact ability to
learn the task. This was achieved for tasks which exhibit non-parametric
variation from the meta-training tasks. However, this was not so for most
tasks. Without a competent right-hemisphere, bi-hemispheric agents generally
exhibit worse initial performance _and_ struggle to learn tasks to the same
degree as an agent trained from scratch. This may be partially explained by
poor right-hemisphere performance rather than the bi-hemispheric agent design.
We discuss potential solutions below.

####  5.0.1 Improved meta-training

Our approach to meta-training the right-hemisphere involved several
simplifications (Section S.LABEL:supp:simplifications), including training on
fewer tasks and for fewer environment steps. Increasing either of these may
result in more robust right-hemispheres with greater generalisation capacity.
Additionally, we saw doubling the GRU size in our right-only baseline resulted
in large improvements in performance. Consequently, improving the right-
hemisphere meta-training process may have significant benefits for bi-
hemispheric agent performance.

####  5.0.2 Training the left-hemisphere and gating network separately

Particularly for the Tier-3 tasks, we observed training trajectories that were
quite variable (see Figure 2). Some trajectories demonstrated reasonable
growth while others exhibited long flat periods without any improvement. We
hypothesise that this is due to interference between gating values and left-
hemisphere gradients. Essentially, left-hemisphere gradients are scaled down
by the gating values, which act as a dynamic learning rate. This may make
learning a novel task difficult for the left-hemisphere if the right-
hemisphere performs poorly.

On solution could be to separate the training of the gating network and left-
hemisphere. This could be achieved by training each network with an off-policy
algorithm. Off-policy algorithms use different policies for exploration and
exploitation [24]. Consequently, the bi-hemispheric agent’s joint policy would
be used to explore while the left-hemisphere and gating network could each be
trained separately, using importance sampling to treat the bi-hemispheric
agent’s decisions as their own.

####  5.0.3 Extension to continual learning

Applying bi-hemispheric agents in continual learning may be where their
capabilities become more beneficial. If agents avoid poor initial performance
on novel tasks, yet still learn, it would result in better overall
performance. For our agent to operate in this setting, it should store learnt
policies in the left-hemisphere, infer whether a task has an existing policy
or is novel, and retrieve existing policies for previously encountered tasks
from the left-hemisphere. We leave this to future work, but note that the
adaptive MoE proposed by Tsuda et al. [25] may help address the issue of task
retrieval. They used gating networks to select which expert to use for novel
tasks. This design could be applied as a left-hemisphere alongside a meta-
trained right-hemisphere.

##  6 Conclusion

We developed a novel RL-agent design with specialised brain hemispheres as per
the NRH. Our agent can exploit generalist skills, when present, from the
right-hemisphere to improve initial performance over an agent trained from
scratch, with minimal impact on its ability to learn novel tasks. Despite
this, when the right-hemisphere lacks these skills, our agent struggles to
perform novel tasks. We also identified potential improvements and extensions
to enable the agent to operate in continual settings.

## References

  * [1] Andrychowicz, M., Raichuk, A., Stańczyk, P., Orsini, M., Girgin, S., Marinier, R., Hussenot, L., Geist, M., Pietquin, O., Michalski, M., Gelly, S., Bachem, O.: What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study (Oct 2020)
  * [2] Botvinick, M., Ritter, S., Wang, J.X., Kurth-Nelson, Z., Blundell, C., Hassabis, D.: Reinforcement Learning, Fast and Slow. Trends in Cognitive Sciences 23(5), 408–422 (May 2019)
  * [3] Chang, Y.N., Lambon Ralph, M.A.: A unified neurocomputational bilateral model of spoken language production in healthy participants and recovery in poststroke aphasia. Proceedings of the National Academy of Sciences of the United States of America 117(51), 32779–32790 (Dec 2020)
  * [4] Duan, Y., Schulman, J., Chen, X., Bartlett, P.L., Sutskever, I., Abbeel, P.: RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning (2016)
  * [5] Finn, C., Abbeel, P., Levine, S.: Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In: Precup, D., Teh, Y.W. (eds.) Proceedings of the 34th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 70, pp. 1126–1135. PMLR (Aug 2017)
  * [6] Gazzaniga, M.S.: Cerebral specialization and interhemispheric communication: Does the corpus callosum enable the human condition? Brain 123(7), 1293–1326 (Jul 2000)
  * [7] Goldberg, E., Costa, L.D.: Hemisphere differences in the acquisition and use of descriptive systems. Brain and Language 14(1), 144–173 (Sep 1981)
  * [8] Goldberg, E., Podell, K., Lovell, M.: Lateralization of frontal lobe functions and cognitive novelty. The Journal of Neuropsychiatry and Clinical Neurosciences 6(4), 371–378 (Nov 1994)
  * [9] Goldberg, E., Roediger, D., Kucukboyaci, N.E., Carlson, C., Devinsky, O., Kuzniecky, R., Halgren, E., Thesen, T.: Hemispheric asymmetries of cortical volume in the human brain. Cortex 49(1), 200–210 (Jan 2013)
  * [10] Hassabis, D., Kumaran, D., Summerfield, C., Botvinick, M.: Neuroscience-Inspired Artificial Intelligence. Neuron 95(2), 245–258 (Jul 2017)
  * [11] Jacobs, R.A., Jordan, M.I., Nowlan, S.J., Hinton, G.E.: Adaptive Mixtures of Local Experts. Neural Computation 3(1), 79–87 (Feb 1991)
  * [12] Khetarpal, K., Riemer, M., Rish, I., Precup, D.: Towards Continual Reinforcement Learning: A Review and Perspectives. Journal of Artificial Intelligence Research 75, 1401–1476 (Dec 2022)
  * [13] Kirk, R., Zhang, A., Grefenstette, E., Rocktäschel, T.: A Survey of Zero-shot Generalisation in Deep Reinforcement Learning. Journal of Artificial Intelligence Research 76, 201–264 (Jan 2023)
  * [14] Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., Kavukcuoglu, K.: Asynchronous Methods for Deep Reinforcement Learning. In: Proceedings of The 33rd International Conference on Machine Learning. pp. 1928–1937. PMLR (Jun 2016), iSSN: 1938-7228
  * [15] Monaghan, P.: Hemispheric asymmetries in the split-fovea model of semantic processing. Brain and Language 88(3), 339–354 (Mar 2004)
  * [16] Ni, T., Eysenbach, B., Salakhutdinov, R.: Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs. In: Proceedings of the 39th International Conference on Machine Learning. pp. 16691–16723. PMLR (Jun 2022), iSSN: 2640-3498
  * [17] Ortega, P.A., Wang, J.X., Rowland, M., Genewein, T., Kurth-Nelson, Z., Pascanu, R., Heess, N., Veness, J., Pritzel, A., Sprechmann, P., Jayakumar, S.M., McGrath, T., Miller, K., Azar, M., Osband, I., Rabinowitz, N., György, A., Chiappa, S., Osindero, S., Teh, Y.W., van Hasselt, H., de Freitas, N., Botvinick, M., Legg, S.: Meta-learning of Sequential Strategies (2019)
  * [18] Parisi, G.I., Kemker, R., Part, J.L., Kanan, C., Wermter, S.: Continual lifelong learning with neural networks: A review. Neural Networks 113, 54–71 (May 2019)
  * [19] Peleg, O., Manevitz, L., Hazan, H., Eviatar, Z.: Two hemispheres—two networks: a computational model explaining hemispheric asymmetries while reading ambiguous words. Annals of Mathematics and Artificial Intelligence 59(1), 125–147 (May 2010)
  * [20] Pham, Q., Liu, C., Hoi, S.: DualNet: Continual Learning, Fast and Slow (Nov 2021)
  * [21] Rajagopalan, C., Rawlinson, D., Goldberg, E., Kowadlo, G.: Deep learning in a bilateral brain with hemispheric specialization (2022)
  * [22] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal Policy Optimization Algorithms (2017)
  * [23] Shevtsova, N., Reggia, J.A.: A Neural Network Model of Lateralization during Letter Identification. Journal of Cognitive Neuroscience 11(2), 167–181 (Mar 1999)
  * [24] Sutton, R.S., Barto, A.G.: Reinforcement learning: An introduction. MIT Press (2018)
  * [25] Tsuda, B., Tye, K.M., Siegelmann, H.T., Sejnowski, T.J.: A modeling framework for adaptive lifelong learning with transfer and savings through gating in the prefrontal cortex. Proceedings of the National Academy of Sciences 117(47), 29872–29882 (Nov 2020)
  * [26] Wang, J.X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J.Z., Munos, R., Blundell, C., Kumaran, D., Botvinick, M.: Learning to reinforcement learn (2016)
  * [27] Weems, S.A., Reggia, J.A.: Hemispheric specialization and independence for word recognition: A comparison of three computational models. Brain and Language 89(3), 554–568 (Jun 2004)
  * [28] Wolczyk, M., Zając, M., Pascanu, R., Kuciński, L., Miloś, P.: Continual World: A Robotic Benchmark For Continual Reinforcement Learning. In: Advances in Neural Information Processing Systems. vol. 34, pp. 28496–28510. Curran Associates, Inc. (2021)
  * [29] Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., Levine, S.: Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning. In: Proceedings of the Conference on Robot Learning. pp. 1094–1100. PMLR (May 2020), iSSN: 2640-3498
  * [30] Zintgraf, L., Schulze, S., Lu, C., Feng, L., Igl, M., Shiarlis, K., Gal, Y., Hofmann, K., Whiteson, S.: VariBAD: Variational Bayes-Adaptive Deep RL via Meta-Learning. Journal of Machine Learning Research 22(289), 1–39 (2021)


Building Trust: Lessons from the Technion-Rambam Machine Learning in
Healthcare Datathon Event

Jonathan A. Sobel1, Ronit Almog2, Leo Anthony Celi3, Michal Gaziel-Yablowitz4,
Danny Eytan2, Joachim A. Behar1*

1 Faculty of Biomedical Engineering, Technion, Israel Institute of Technology,
Haifa, Israel

2 Rambam Health Care Campus, Haifa, Israel

3 Institute for Medical Engineering and Science, Massachusetts Institute of
Technology, Department of Medicine, Beth Israel Deaconess Medical Center;
Department of Biostatistics, Harvard T.H. Chan School of Public Health,
Boston, MA, 02115, USA

4 TIMNA- Israel’s Ministry of Health Big Data Platform, The Ministry of Health

*<EMAIL_ADDRESS>
## Abstract

A joint conference and datathon event organized by the Technion, Rambam Health
Care Campus and the MIT Critical Data group took place on March 7-9th 2022 and
provided a unique opportunity to work with real-world clinical datasets and
open clinical questions that may impact medical practices. Clinicians from
Rambam posed challenges that they face in their practice. Participants mostly
originated from six different faculties and were divided into teams with mixed
skillsets. Each team was assigned a research project as well as a senior
clinical mentor and a data scientist mentor. At the end of the second day of
the competition, three teams were shortlisted and presented their analysis at
the final plenary session of a conference held the following day in front of a
panel of senior technologists, machine learning (ML) researchers and health
professionals. The datathon demonstrated the potential in crowd sourcing of
bright minds of students from different faculties and backgrounds to formulate
innovative solutions that can impact the clinical practice. It facilitated
definition of a roadmap for data preparation and safe data sharing by the
epidemiology and the IT units of the hospital. It helped understand how
opening these data to research can bear significant impacts on the management
of the hospital patients and beyond. Most importantly, this event served as a
vector for establishment of a dialogue to bridge between a the Technion, a
technology institute, and Rambam, its natural partnering hospital, in the
field of medical data science. By demonstrating to stakeholders from both
institutions the potential in such data sharing and team work, this datathon
will likely catalyze future collaborations. We plan to strengthen this bridge
through the newly established joint academic Technion-Rambam Center for
Artificial Intelligence in Healthcare (CAIH).

## Author summary

A datathon is a time-constrained competition involving data science applied to
a specific problem. In the past decade, datathons have been shown to be a
valuable bridge between fields and expertise. Biomedical data analysis
represents a challenging area requiring collaboration between engineers,
biologists and physicians to gain a better understanding of patient physiology
and of guide decision processes for diagnosis, prognosis and therapeutic
interventions to improve care practice. Here, we reflect on the outcomes of an
event that we organized in Israel at the end of March 2022 between the MIT
Critical Data group, Rambam Health Care Campus (Rambam) and the Technion
Israel Institute of Technology (Technion) in Haifa. Participants were asked to
complete a survey about their skills and interests, which enabled us to
identify current needs in machine learning training for medical problem
applications. This work describes opportunities and limitations in medical
data science in the Israeli context.

## Introduction

### Context of the event

This joint event organized by the Technion, Rambam and the MIT Critical Data
group provided a unique opportunity to understand the challenges faced by
leading researchers and clinicians working in the field of medical data
science. The Technion is one of the world’s leading science and technology
research institutes and Rambam is the largest hospital in the north of Israel.
The event took place in the city of Haifa, Israel on both the Technion and
Rambam campuses. It was organized as the inaugural event of a new joint
Technion-Rambam Center for Artificial Intelligence in Healthcare (CAIH), which
aims to serve as an academic center for medical AI committed to advanced
medical and clinical research, with significant and actionable benefit to
patient care. This first joint academic-hospital AI center in Israel, plans to
develop advanced AI systems for patient evaluation driven by complex and rapid
analysis of all the relevant medical information that has accumulated in big
medical databases over the years [1]. The Center’s opening event entitled
“Technion-Rambam Hack: Machine Learning in Healthcare,” was held on March
7-9th 2022 and was attended by about 250 people. The first two days consisted
of a collaborative information-based competition [2, 3, 4, 5] that focused on
solving real-world clinical problems through interdisciplinary teams and
access to real data [6]. The datathon was followed by a one-day conference
with lectures delivered by researchers from the Technion, Rambam, MIT, the
Israeli Ministry of Health (MOH), Clalit Health Services, GE Healthcare, and
Roche. Scientists, healthcare practitioners and policy makers from across the
globe shared their knowledge on the fascinating topic of medical AI.

### The complex structure of Israel’s healthcare system

Israel’s healthcare system is based on various organizations with diverse
types of ownership and complex relations between them [7]. Each Israeli
resident is entitled to healthcare coverage provided by one of four non-
profits health maintenance organizations (HMOs)- Clalit, Maccabi, Leumit and
Meuhedet, which compete with each other [7]. The HMOs deliver quality
community services close to the patient’s home and use various payment methods
to purchase services supplied by hospitals [8]. There are general hospitals
across Israel with diverse types of ownership: Clalit owns 14 hospitals, MOH
and non-profit organizations own 17 hospitals and Maccabi owns two hospitals.
At the same time, the MOH serves as the owner of 11 general hospitals, the
regulator that oversees all public hospital expenses and the legislator. Until
1994, Clalit and Leumit provided care to the majority of Israel’s residents
who were unionized in two separate labour organizations, while Maccabi and
Meuhedet acted as private health providers and had the privilege to define
their own admissions requirements. As a result, 5% of the population was not
covered by health insurance, and Clalit, the largest HMO in the country,
accumulated huge debts, which jeopardized its ability to provide care to its
members. The legislation of the 1994 National Health Insurance Law changed
Israel’s healthcare system dramatically. It granted minimum essential
healthcare coverage to all Israeli residents and the right to choose a
healthcare provider. The reforms in public healthcare services had far-
reaching financial ramifications. New agreements and fee updates increased the
competition among HMOs and decreased the HMOs’ use of hospital services [7].
As a result, hospital directors adopted a business-oriented attitude in order
to generate income beyond their usual financial support from the state. The
need to reduce operational costs and increase the value of care led hospitals
and HMOs to computerize their electronic health records (EHR) and invest in
digital infrastructure for their storage and sharing [9].

### Stakeholders hold different levels of data

Israel is a world leader in use of health data to reduce costs and complexity,
while increasing patient satisfaction and outcomes. For 25 years already,
Israel’s hospitals and HMOs use different EHR platforms, with each type of
healthcare organization holding access to different levels of health data. All
hospitals hold full EHRs, that include clinical measurements, lab test
results, images, text (reports made by nurses or physicians) and medications
prescribed during the patient’s visit. Community-based clinics hold
longitudinal health data that include medical history, lab results of routine
check-ups, routine medication, images taken in the clinics, and administrative
data and costs. Each public hospital owned by the MOH or non-profit
organization collects and stores data separately. Community-based clinics
affiliated with Clalit or Maccabi collect and share in-network data with their
affiliated hospitals. Meuhedet or Leumit share in-network community-based data
without the ability to link it with full hospital data. Massive volumes of
data about patients, conditions and treatments have accumulated over the
years. Very recently, the Israeli health authorities decided to invest $30M in
19 hospital programs that promoted anonymized data sharing between healthcare
organizations. To support some degree of data sharing between all
stakeholders, the MOH implemented the ”Ofek ” system, which is a secured
health information exchange network that enables hospital teams to view
patient medical history, routine medications, test results, and imaging and
pathology data in HMO records during their hospitalization [10]. In addition,
all hospitals are required to deliver detailed information to the MOH’s
hospitalization database. The collected information includes, socio-
demographic details, medical diagnostic data and administrative data from each
patient visit in every Israeli hospital. Moreover, the MOH owns more than 20
different population-level clinical datasets, which includes perinatal and
child developmental data up to the age of six, births and death causes
databases, the COVID-19 database and other national registries. In addition,
the MOH owns the TIMNA platform (Research Infrastructure for Big Data), a
national framework that supports data sharing across all types of health
organization, and enables access to the MOH’s national databases, through a
neutral, virtual and secured research environment [11, 12, 13, 14].

Overall, we organised this event to serve as a vector for establishment of a
dialogue to bridge between a the Technion, a technology institute, and Rambam,
its natural partnering hospital, in the field of medical data science. By
demonstrating to stakeholders from both institutions and beyond (academia,
HMO, companies, governmental organisations and non-governmental
organisations), the potential in such data sharing and team work, this
datathon will likely catalyze future collaborations in Israel.

## Methods

## How to organize a datathon?

To organize a successful event, several important points should be well
thought through before the event. The checklist provided below should help any
organizer in this process. One of the first decisions is related to the place
of the future event, should it be virtual or in person. Next, a list of
potential partners should be assembled to help fund the event and attract
participants. Leaders in the field who will present some of their work are an
asset to the event. Moreover, some of these partners can play the role of
mentors during the competition. Another critical point for a datathon involves
finding questions of clinical importance that can be addressed using
previously collected data. There are several options here as some events may
be more flexible in the sense that participants/projects leader can come with
their data and questions or a more directed approach with defined datasets and
questions. In all cases, the data should be properly anonymized and the
ethical statements from the institutional review board (IRB) should be
provided before the event in the case of medical data. When medical data are
used a secure infrastructure such as AWS or Azure can be set up to avoid any
leakage of medical data. Mentors that can follow each team during the whole
event are necessary to ensure the success of each project. The choice of the
targeted audience is one of the main points. The goal of a datathon is to
demonstrate the effectiveness of a multidisciplinary approach where each team
member can provide different expertise (medical, technical, social, legal and
business). There are two options there: 1) a more directed approach where
teams are built by the organizers from a pool of participants or 2) the self-
organization approach where participants choose their project before or at the
beginning of the event. Last, the reward for the winning teams is important to
acknowledge teams that have performed well as well as encourage them
continuing the research afterward. Participants are generally striving for
learning new skills and networking but the reward will increase their
motivation to succeed.

Datathon Check list

  * •

Venue: Physical/Virtual/Hybrid, dates, location.

  * •

Logistics: catering, strong WIFI, rooms and amphitheater.

  * •

Partners: Industrial, NGO, clinicians and academic stakeholders , who may fund
some part of the event (awards/venue/infrastructure) as well as deliver
relevant talks during the event.

  * •

Projects: Research project call for datasets with ethical consents (IRB) and
specified aims/questions.

  * •

IT support and secured computational infrastructure (for sensitive clinical
data to be shared with participants).

  * •

Mentors: Clinical and data science. Try to select senior mentors.

  * •

Participants: Who is your targeted audience? (students/medical
professionals/data scientists)

  * •

Communication/PR: Information to participants and advertisement of the event
(Flyers/website/social media)

  * •

Awards: Money for the winning teams or other gifts

### The Datathon days

The Datathon is an information-based competition [6]. It was held at the
Technion Faculty of Biomedical Engineering. Teams were organized to be cross-
disciplinary, and included data scientists and students from healthcare
disciplines (biomedical engineering, medicine, biology, physics, electrical
engineering, and computer science). The planning of the datathon and the
conference began approximately six months before the event (Figure 1). Several
tasks were planned in each phase of the event organization. After initial
brainstorming between the scientific committee, which included Technion
principal scientists, Rambam clinicians and MIT scientists, a fundraising
campaign was launched as list of potential speakers for the conference day was
drawn up and invitations were extended. Communication around the event was
initiated in November 2021 via social media platforms (Twitter, LinkedIn and
Facebook). Students interested in the datathon were asked to apply to the
event and were asked to complete a survey about their skills, their interests
and their level of education (Bsc., Msc., Ph.D., alumni) and specialty
(engineering or bio/med). In parallel, we contacted clinicians from Rambam and
asked them to propose projects consisting of a medical question and to provide
a relevant dataset to research the question. Four challenges proposed by
clinicians who had collected large datasets in recent years and who presented
challenging scientific questions which could be tackled by ML were selected.
The projects were 1) Prediction of newborn birthweight by maternal parameters
and previous newborn siblings birthweights [15], 2) ML-based predictive model
for bloodstream infections during hematopoietic stem cell transplantation
[16], 3) Prediction of recurrent hospitalization in heart failure patients
[17] and 4) Risk factor and severity prediction in hospitalized Covid-19
patients [18, 19]. Each challenge came from a different hospital department
and was presented with a database of approximately 2 Gb data. The data were
anonymized and structured by the Rambam epidemiology and IT teams [20, 21].
The ethical approval by the institutional review board to work on these
datasets was checked and updated, if necessary. The four challenges relied on
EHR data (i.e. structured data) [22, 23]. This was preferred over the usage of
unstructured data (time series or images) because of the format of the
Datathon which is two days long, and because of constraint on the budget for
cloud computing.

Clinicians who proposed the project were, by default, the medical mentors and
two competing teams composed of 5-7 participants were assigned to each
project. A senior data scientist was recruited for each team either from
Technion labs, or Rambam epidemiology department or industry partners. The
role of the mentors was to advise and monitor their team’s activities
throughout the entire event. Participants were selected based on their
interests and competency (studies and skills). Our goal was to have mixed
teams in terms of data analysis capacity and field knowledge to work on each
challenge. Next, the computational infrastructure was built in collaboration
with the Rambam IT team using Amazon Web Services (AWS) [24] and tests were
run to simulate the event. Our requirement was to work on a secure cloud that
provides data access without data leak or data export to a participant’s
computer. We started with a minimal configuration of an 8-core machine with
16Gb RAM for each team. During the event, teams were allowed access to more
computational resources, on request. Each team was asked to test the cloud two
weeks before the event and was allowed to set up their infrastructure with
programs and libraries (e.g., R, Python). In a second testing phase, the
internet access from the cloud was closed and teams were allowed to explore
the structure of the data using a small sample of patients. Finally, one day
before the event, the full databases were loaded to the cloud. Each team had a
separate virtual machine with personal, secured access for each team member.
During the two days of the event, the teams were split in several rooms at the
Technion Faculty of Biomedical Engineering. After the first day’s lunch, a
networking event in the form of a scientific speed dating was held with
mentors, team members and organizers, to promote future collaborations. Each
team was asked to present its work at the end of the second day. Thereafter,
using an external jury comprised of a principal investigator from the
Technion, clinicians, Rambam epidemiology and IT department, and industrial
partners, the three best teams were selected for the competition final, which
took place on the conference day.

### The Conference day

The guest talks were intended to introduce the field of clinical data science
to a wide audience, including those with both technical and non-technical
backgrounds, and to provide a perspective on its future impact on the field of
medicine. For this purpose, the conference day was structured into three main
plenary sessions. Session 1 entitled “Current trends in ML in healthcare”
addressed recent research in medical ML in the fields of omics, medical image
analysis, causal inference and physiological time series analysis [25, 26, 27,
28]. The presentations began with a general introduction and presented
examples of research questions that can be tackled by ML for the purpose of
diagnosis, risk prediction and treatment as well as basic science. Session 2
was entitled “Data stakeholders” and aimed to present the medical data
landscape in Israel, and to clearly present the different actors in Israel
i.e. Hospitals, HMOs and MOH and the data availability options, limitations,
types of partnerships and costs. The speakers emphasized which data are
available at each level (Hospitals, HMOs and MOH) as well as the bridges that
exist between the different actors. The speakers also discussed how their
entity collaborates with academia and industry. Session 3 entitled “Deployment
of ML in medical practice” discussed how ML algorithms translate to medical
practice. In other words, how ML models trained on retrospective data can be
translated to clinical practice, including algorithm integration, training and
perception of medical staff, importance of distribution shifts and model
generalization [29].

Fig 1: Datathon and conference preparation and execution timeline. Fig 2:
Datathon’s participants and mentors. A total of 50 students and alumni
participated in the event and were advised by 20 mentors. Pictures copyright
Nitzan Zohar (Technion)

## Results

The datathon was attended by 50 students and alumni from various Technion
faculties. Participants developed different strategies for the analysis of
medical data, to solve important challenges in cardiology, fetal monitoring,
intensive care, and stem cell transplantation. The students were accompanied
by 20 mentors from the Technion, Rambam and the industry. Among the 80
applicants, most were engineers (75%) and BSc students (50%). Consequently, in
order to organize teams with diverse backgrounds and education levels, we
favoured participants with MSc or PhD and with a bio/medical background. Thus,
of the selected 50 participants, 20% where PhD, 40% were MSc and 40% where BSc
(Figure 3A). We asked participants to report their field of study, their
favourite programming language and their medical field of interest and to rate
their competency in various technical skills (Figure 3B-E). Approximately 33%
of the participants reported that they had a bio/medical oriented background
(Figure 3B), and the majority of engineers came from the biomedical
engineering faculty of the Technion. Interestingly, the next largest subgroup
was computer scientists. The preferred programming language was Python, as
this language is well-suited for ML, free and well-studied across many
Technion faculties (Figure 3C). Many participants reported on good knowledge
of Matlab, as many courses at the Technion rely on this licensed computational
tool. Only a minority had previous knowledge of R and other languages used in
data-science research. Participants were interested in several medical fields
(Figure 3D), with a slight preference for radiology, oncology and cardiology.
Participants reported having a good background of data analysis and data-
visualization and ML in general (Figure 3E). They reported slightly lower
proficiency in biostatistics, modelling and image analysis. Finally,
participants were less confident in the analysis of EHR, database management,
time series and text analysis.

Fig 3: Survey results. A) Level of education, B) specialty, C) programming
skills, D) medical field of interest and E) data analysis and ML skills. Note
that participants could provide multiple answers for their orientation, skills
and interests.  Fig 4: Participants and mentors interaction. Collaboration
between students during the datathon event (left image) and a medical mentor
explaining the meaning of some of the variables to one of the datathon team.
Pictures copyright Nitzan Zohar (Technion).

## Discussion

As arose from participant surveys, this type of event attracts the young
generation of engineers, who are used to collaborative projects, open source
and open data initiatives. On the other hand, there were fewer bio/med student
applicants. This may be because they were less confident about their
contribution due to their lack of coding/data management skills. However, we
observed that successful teams were those comprised of clinicians (physicians,
nurses), data managers, hospital IT & infrastructure officer, and engineers
(developers, medical device, data acquisition). These teams achieved a better
understanding of the pathologies and challenges faced in the clinic.
Furthermore, this datathon event underscored the potential in exploiting the
bright minds of university students from various faculties and backgrounds to
design solutions that can impact clinical practice. It facilitated defining a
road for data preparation and safe data sharing by the epidemiology and the IT
units of the hospital. It helped understand how opening medical data to
research can significantly impact patient management and beyond. Most
importantly, the event seemed to be the vector necessary for establishing a
dialogue between the Technion and its natural partnering hospital (Rambam) in
the field of medical data science. Leveraging such a meeting between a
technological university and its nearby hospital institution can serve as a
model of trust building between partners. We will strengthen this bridge
through the newly established CAIH.

### Barriers to development and adoption of ML tools in medicine

There are multiple barriers to development and, moreover, adoption of ML tools
in medicine that span the full cycle, from inception to deployment. Some
relate to data availability as detailed here and others to technological
difficulties, culture- and local ecosystem-related resistance, and lack of an
enabling infrastructure in most Institutions, many of which we plan to address
as part of the mission of CAIH. Other barriers stem from a chasm between the
clinical domain experts and their ML counterparts; development of accurate,
robust, actionable, and fair ML tools requires close collaboration between
clinical domain experts, data scientists and engineers. For such collaboration
to be truly fruitful, these specialists must be able to interact and exchange
ideas and thoughts using a common language and basic understanding of key
concepts from both worlds. Typically, most clinicians and data scientists lack
this reciprocally needed knowledge. Specifically, most current-day medical
training programs in Israel and abroad do not include courses dedicated to
advanced data analytics, such as introduction to ML.

Moreover, in most medical schools, even if students wish to expand their
knowledge on ML and its applications to healthcare, there is no option to do
so. Of note, in this Regard, the Technion is unique in that it has established
several undergraduate programs for a dual bachelor’s degree in medicine and
either computer science or biomedical engineering, which can be completed
before clinical training begins. These programs require two additional years
of study. It is our view though, that considering the expected near-future
integration of multiple artificial-intelligence-based technologies into
everyday practice of most clinicians, a basic understanding of concepts
related to how such algorithms work, their advantages and failure modes will
be advantageous and even necessary. Such basic knowledge is required to
appropriately operate these tools, similar to the way a clinician must
understand how an electrocardiogram signal is generated to be able to
accurately diagnose common and rare diseases or recognize data acquisition
problems. Thus, we believe that medical schools should be cognizant of the
changing landscape and adjust their curricula accordingly, to include some
basic training in ML. The flip-side of this challenge lies in the need to
equip engineering students who plan to specialized in ML for healthcare with
basic medical knowledge and familiarity with relevant medical context. Many
leading institutions, including the Technion, established specific courses
dedicated to introducing engineering students to this field and its unique
jargon, challenges, and progress. Additional initiatives in this direction
include datathons, such as the one described here, and promotion of joint
undergraduate and graduate projects dedicated to ML in healthcare and spanning
engineering and medical faculties. In that sense, institutions such as the
Technion, which is a technology institute yet also has a large medical
faculty, bring an advantage.

### Accessing medical data in Israel: a critical appraisal

Despite the fact that Israel likely has some of the best medical datasets in
the world, due to the structuring of its healthcare system and the very early
digitization of personal medical data, access to medical data remains
challenging due to ethical concerns and regulation (data protection,
anonymisation, GDPR, HIPAA) and their potential business value (intellectual
property). This has important consequences on the Israeli academy seeking to
access these data. In addition, data fragmentation poses a major challenge.
There are three main data stakeholders in Israel: the HMOs, the hospitals and
the MOH. Each with their own agenda. Challenges arise when when seeking to use
data across multiple stakeholders, e.g., imaging data from a hospital, with
clinical endpoints from the MOH or longitudinal data which is only be
available at the HMO level. There is currently no ”one-stop shop” address for
collection of all available data. The final challenge is the infrastructure.
It is our belief that the state should be independent and sovereign when it
comes to medical data of citizens. In this respect, we believe that Israel
should have its own, i.e., physically located in Israel and solely owned by
the government, high-performance computing (HPC) facilities hosting medical
data for research and development. This is with the exception of temporary
hosting of medical data on classical clouds or international platforms in the
context of federated learning. New initiatives by the MOH, such as TIMNA, or
the new program to promote anonymized data sharing between healthcare
organizations, may tackle some of these challenges. Yet, new regulations will
be necessary for the MOH to allow medical data sharing for research purposes
in Israeli universities. Without this legal enforcement, the business focus of
health institutions may remain preponderant and limit the academic research
performed in Israel.

## Conclusion

This datathon was very attractive and built trust and establish a pipeline for
medical data preparation and sharing between medical and technology
institutions. It provided a critical appraisal of education needs for health
professionals and engineers seeking to pursue ML fields. Finally, this work
represents a first field test for medical data access in Israel. Obstacles are
numerous but Israel has the potential to exist as a main actor in the field of
medical ML because of its strong basis in engineering and medicine and its
history of medical data digitization.

## Ethical statement

Statistics were provided as part of the application process to the datathon
and were anonymized. Each participants signed an non-disclosure agreement
stating that any collected data for the challenges and during the event belong
to the Rambam HCC. Participants to the event were told that pictures would be
taken during the event and could be used for marketing purposes of the event.

## Acknowledgments

We acknowledge the financial support of the Technion-Rambam Center for
Artificial Intelligence in Medicine, the Technion Machine Learning and
Intelligent Systems (MLIS) Center, the Technion Human Health Initiative (THHI)
and the MISTI MIT - Israel Zuckerman STEM Fund. This research was partially
supported by The Milner Foundation, founded by Yuri Milner and his wife Julia.
We are grateful to the Placide Nicod Foundation for their financial support
(J.S.). We are grateful to the medical and technical staff from the IT and
epidemiology departments at Rambam HCC. We are thankful to the Technion
administrative staff for supporting the organization and communication of the
Datathon event.

## References

  * 1. Sapci AH, Sapci HA.  Artificial Intelligence Education and Tools for Medical and Health Informatics Students: Systematic Review.  JMIR Med Educ. 2020;6(1):e19285.  doi:10.2196/19285.
  * 2. Lyndon MP, Cassidy MP, Celi LA, Hendrik L, Kim YJ, Gomez N, et al.  Hacking hackathons: preparing the next generation for the multidisciplinary world of healthcare technology.  International Journal of Medical Informatics. 2018;112:1–5.
  * 3. Serpa Neto A, Kugener G, Bulgarelli L, Rabello Filho R, Hoz MÁAdl, Johnson AE, et al.  First Brazilian datathon in critical care.  Revista Brasileira de Terapia Intensiva. 2018;30:6–8.
  * 4. Pathanasethpong A, Soomlek C, Morley K, Morley M, Polpinit P, Dagan A, et al.  Tackling regional public health issues using mobile health technology: event report of an mHealth hackathon in Thailand.  JMIR mHealth and uHealth. 2017;5(10):e8259.
  * 5. Li P, Xie C, Pollard T, Johnson AEW, Cao D, Kang H, et al.  Promoting secondary analysis of electronic medical records in China: summary of the PLAGH-MIT critical data conference and health datathon.  JMIR Medical Informatics. 2017;5(4):e7380.
  * 6. Aboab J, Celi LA, Charlton P, Feng M, Ghassemi M, Marshall DC, et al.  A “datathon” model to support cross-disciplinary collaboration.  Science Translational Medicine. 2016;8(333):333ps8–333ps8.
  * 7. Naamati Schneider L.  Strategic management as adaptation to changes in the ecosystems of public hospitals in Israel.  Israel Journal of Health Policy Research. 2020;9(1):1–11.
  * 8. Clarfield AM, Manor O, Nun GB, Shvarts S, Azzam ZS, Afek A, et al.  Health and health care in Israel: an introduction.  The Lancet. 2017;389(10088):2503–2513.
  * 9. Jones GL, Peter Z, Rutter KA, Somauroo A.  Promoting an overdue digital transformation in healthcare.  Ariel. 2019;.
  * 10. Gerber A, Topaz MM.  Promoting meaningful use of health information technology in Israel: ministry of health vision.  In: Nursing Informatics 2014. IOS Press; 2014. p. 108–115.
  * 11. Fabiana B, Assaf S, Ramona S, Konikoff F, Timna N, et al.  Trends in Inflammatory Bowel Disease-Comparison Between the Arab and Jewish Population in Israel.  J Clin Gastroenterol Treat. 2017;3:041.
  * 12. Peleg M, Reichman A, Shachar S, Gadot T, Avgil Tsadok M, Azaria M, et al.  Collaboration between Government and Research Community to Respond to COVID-19: Israel’s Case.  Journal of Open Innovation: Technology, Market, and Complexity. 2021;7(4):208.
  * 13. Jaffe E, Sonkin R, Podolsky T, Alpert EA, Siman-Tov M.  The role of Israel’s emergency medical services during a pandemic in the pre-exposure period.  Disaster Medicine and Public Health Preparedness. 2020; p. 1–5.
  * 14. Organization WH, et al.  Country Cooperation Strategy: Israel, 2019-2025.  -. 2020;.
  * 15. McCowan LM, Harding JE, Stewart AW.  Customised birthweight centiles predict SGA pregnancies with perinatal morbidity.  BJOG: An International Journal of Obstetrics & Gynaecology. 2005;112(8):1026–1033.
  * 16. Gupta V, Braun TM, Chowdhury M, Tewari M, Choi SW.  A systematic review of machine learning techniques in hematopoietic stem cell transplantation (HSCT).  Sensors. 2020;20(21):6100.
  * 17. Ouwerkerk W, Voors AA, Zwinderman AH.  Factors influencing the predictive power of models for predicting mortality and/or heart failure hospitalization in patients with heart failure.  JACC: Heart Failure. 2014;2(5):429–436.
  * 18. Benaim AR, Sobel JA, Almog R, Lugassy S, Shabbat TB, Johnson A, et al.  Comparing COVID-19 and influenza presentation and trajectory.  Frontiers in medicine. 2021;8.
  * 19. Sobel JA, Levy J, Almog R, Benaim AR, Miller A, Eytan D, et al.  Descriptive characteristics of continuous oximetry measurement in moderate to severe COVID-19 patients.  medRxiv. 2021;doi:10.1101/2021.09.26.21264135.
  * 20. O’herrin JK, Fost N, Kudsk KA.  Health Insurance Portability Accountability Act (HIPAA) regulations: effect on medical record research.  Annals of surgery. 2004;239(6):772.
  * 21. Kruse CS, Smith B, Vanderlinden H, Nealand A.  Security techniques for the electronic health records.  Journal of medical systems. 2017;41(8):1–9.
  * 22. Johnson AE, Pollard TJ, Shen L, Lehman LwH, Feng M, Ghassemi M, et al.  MIMIC-III, a freely accessible critical care database.  Scientific data. 2016;3(1):1–9.
  * 23. Hyland SL, Faltys M, Hüser M, Lyu X, Gumbsch T, Esteban C, et al.  Early prediction of circulatory failure in the intensive care unit using machine learning.  Nature medicine. 2020;26(3):364–373.
  * 24. Fusaro VA, Patil P, Gafni E, Wall DP, Tonellato PJ.  Biomedical cloud computing with amazon web services.  PLoS computational biology. 2011;7(8):e1002147.
  * 25. Biton S, Gendelman S, Ribeiro AH, Miana G, Moreira C, Ribeiro ALP, et al.  Atrial fibrillation risk prediction from the 12-lead electrocardiogram using digital biomarkers and deep representation learning.  European Heart Journal-Digital Health. 2021;2(4):576–585.
  * 26. Levy J, Álvarez D, Rosenberg AA, Alexandrovich A, Del Campo F, Behar JA.  Digital oximetry biomarkers for assessing respiratory function: standards of measurement, physiological interpretation, and clinical use.  NPJ digital medicine. 2021;4(1):1–14.
  * 27. Chocron A, Oster J, Biton S, Mandel F, Elbaz M, Zeevi YY, et al.  Remote atrial fibrillation burden estimation using deep recurrent neural network.  IEEE Transactions on Biomedical Engineering. 2020;68(8):2447–2455.
  * 28. Jesson A, Tigas P, van Amersfoort J, Kirsch A, Shalit U, Gal Y.  Causal-BALD: Deep Bayesian Active Learning of Outcomes to Infer Treatment-Effects from Observational Data.  In: Ranzato M, Beygelzimer A, Dauphin Y, Liang PS, Vaughan JW, editors. Advances in Neural Information Processing Systems. vol. 34. Curran Associates, Inc.; 2021. p. 30465–30478.  Available from: https://proceedings.neurips.cc/paper/2021/file/ffa4eb0e32349ae57f7a0ee8c7cd7c11-Paper.pdf.
  * 29. Zhang J, Whebell S, Gallifant J, Budhdeo S, Mattie H, Lertvittayakumjorn P, et al.  An interactive dashboard to track themes, development maturity, and global equity in clinical artificial intelligence research.  The Lancet Digital Health. 2022;4(4):e212–e213.


# Few-Shot Classification of Skin Lesions from Dermoscopic Images by Meta-
Learning Representative Embeddings

Karthik Desingu
Dept. of Computer Science and Engineering
Sri Sivasubramaniya Nadar College of Engineering
Chennai, TN, India
<EMAIL_ADDRESS>
Mirunalini P.
Dept. of Computer Science and Engineering
Sri Sivasubramaniya Nadar College of Engineering
Chennai, TN, India
<EMAIL_ADDRESS>
Aravindan Chandrabose
Dept. of Information Technology
Sri Sivasubramaniya Nadar College of Engineering
Chennai, TN, India
<EMAIL_ADDRESS>

###### Abstract

Annotated images and ground truth for the diagnosis of rare and novel diseases
are scarce. This is expected to prevail, considering the small number of
affected patient population and limited clinical expertise to annotate images.
Further, the frequently occurring long-tailed class distributions in skin
lesion and other disease classification datasets cause conventional training
approaches to lead to poor generalization due to biased class priors. Few-shot
learning, and meta-learning in general, aim to overcome these issues by aiming
to perform well in low data regimes. This paper focuses on improving meta-
learning for the classification of dermoscopic images. Specifically, we
propose a baseline supervised method on the meta-training set that allows a
network to learn highly representative and generalizable feature embeddings
for images, that are readily transferable to new few-shot learning tasks. We
follow some of the previous work in literature that posit that a
representative feature embedding can be more effective than complex meta-
learning algorithms. We empirically prove the efficacy of the proposed meta-
training method on dermoscopic images for learning embeddings, and show that
even simple linear classifiers trained atop these representations suffice to
outperform some of the usual meta-learning methods.

_K_ eywords Dermoscopic Images, Few-Shot Learning, Meta Learning, Embedding
Network

##  1 Introduction

Annotated data is essential for any supervised learning algorithm, and the
availability of large amounts of annotated data helps deep learning methods
achieve quantum leaps in classification tasks. Deep learning networks requires
large amount of data to be robust enough to generalize feature learning for
classification during training. Even though large amount of data is available
in the digital world, availing annotated data is very difficult. In the
medical domain especially, obtaining annotated data for rare or newly emerging
diseases is even harder: employing medical experts to annotate the data, and
manually labeling the large amount is data is time consuming and expensive.
Furthermore, in some applications, the data across the classes is not
distributed evenly. This leads to biased learning. Hence, in this research
work, we propose a meta-learning approach that helps in rapid model adaptation
to deal with data scarcity and unbalanced class distribution of datasets.

Deep learning models have proven their value in the various aspects of the
automation of medical image analysis; skin cancer type classification from
dermatological images is one such area. Automated classification of skin
cancer type is complex, owing to the similarity between the different types of
lesions; the limited availability of annotated data for some cancer types due
to the relative rarity of some of these cancer types; and the characteristic
uneven distribution of samples across different classes due to similar
reasons. When deep learning models are applied on such data, they were often
biased towards the classes with large number of samples, and failed to
generalize well for skewed classes. Clearly, there is a need to facilitate
models to learn from small amounts of annotated data, and readily adapt to new
classes with small sample sizes.

In this paper, we present a baseline learning approach for few-shot image
classification in the meta-learning context. This approach has been
underappreciated in the literature thus far. To the best of the authors’
knowledge, this is the first application of this method for skin image
classification in the few-shot regime. This approach consistently outperforms
the protoypical network on two different datasets, even with the same backbone
network. The method has been applied using multiple backbone networks — simple
convolutional neural network, ResNet18 and ResNet50 [6]. Feature embeddings
learnt through this method are, in general, more representative and are able
to generalize well to unseen classes in few-shot tasks.

##  2 Related Work

There is limited availability of labeled data. Even annotated data is often
characterized by uneven distribution of samples across classes. Although deep
learning models march towards great heights in several challenging tasks [19,
7], it demands large amounts of data [3, 23] to achieve such performance.
Meta-learning algorithms, however, can learn from limited data and produce
comparable prediction performance. This not only solves the annotation
insufficiency issue, but also works well on unbalanced datasets, learning to
represent the minority classes well. Several researchers have developed,
experimented with, and implemented meta-learning successfully to obtain high
performance with limited data. Few-Shot Learning (FSL), a form of meta-
learning targets the problem of data scarcity by learning a model using set of
base classes, and adopts to new classes with just a few samples [5, 21, 18].
FSL uses a metric learning approach to achieve this [1, 17].

A model-agnostic meta-learning method was proposed in [5] which quickly adapt
to new tasks. Metric learning based on deep neural features was proposed in
[21] that maps a small labeled support set and an unlabeled example to its
label that helps to adapt to new novel classes. A prototypical network was
formulated in [18] that uses the Euclidean distance computed for a prototype
representation of each class. Furthermore, a transformer based bi-directional
decoding mechanism was used to learn correlations between same class samples
in order to connect visual clues with semantic descriptions was proposed in
[1]. Latent Embedding Optimization (LEO) was proposed to learn low-dimensional
model parameters and performs meta-learning in this space [17]. One-shot
learning setting made correct predictions using Siamese neural networks which
employ a unique structure to rank the inputs based on the similarities [10].
Matching networks applied an LSTM based context encoder to match query and
support set images [22] in one-shot learning paradigms. An LSTM based meta-
learner model was proposed to learn perfect a optimization algorithm, that
helps to train another neural network classifier in the few-shot regime [16].
An edge-labeling graph neural network (EGNN) was proposed to predict the edge-
labels, which in turn enables explicit clustering by iteratively updating the
edge-labels by exploitation of intra-cluster similarity and inter-cluster
dissimilarity [9].

##  3 Method

This section sets up and defines the preliminaries about the few-shot problem,
and the meta-learning problem in general. The specific training-testing
approach adapted is detailed in subsequent sections. To facilitate easy
comparison, the same notations used in [20] for formulations are used in this
paper.

###  3.1 Meta-Learning: Problem Formulation

The learning problem is presented as a meta-learning task with two phases —
the meta-training and the meta-testing phase. The meta-training phase is based
on the meta-training set $T$. $T$ is defined by,
$T={(D^{train}_{i},D^{test}_{i})}^{I}_{i=1}$, where each tuple
$(D^{train}_{i},D^{test}_{i})$ represents a training and a testing dataset for
a single episode of the task. Each dataset contains a few examples. The
training examples $D^{train}={(x_{t},y_{t})}^{T}_{t=1}$ constitute the support
set $S$, and the testing examples $D^{test}$ = ${(x_{q},y_{q})}^{Q}_{q=1}$
form the query set $Q$ of the episode. Both $D^{train}$ and $D^{test}$ are
sampled from the same distribution.

A base learner $A$, defined by $y_{*}=f_{\theta}(x_{*})$, is trained on
$D^{train}$ and used as a predictor on $D^{test}$. For image classification
problems, $x_{*}$ is characterized by very high dimensionality, consequently a
very high variance. Hence, $D^{train}$ and $D^{test}$ are first projected onto
a feature space using a feature embedding model that can be formulated as
$\phi_{*}=f_{\phi}(x_{*})$. The key intent of the meta-training phase is to
learn a good embedding model, so that the base learning performs with minimal
loss on a set of query tasks ($D^{test}$). This can be stated formally as,

$\phi=\mathop{\mathrm{argmin}}_{\phi}E_{T}[L^{meta}(D^{test};\theta,\phi)]$
(1)

where $\theta=A(D^{train};\phi)$ is the base learner, and is fixed through
training.

The embedding model $\phi$ is fixed when training the base learner in each
episode. The objective of the base learner on each task can then be modeled
as,

$\theta=A(D^{train};\phi)\\\
=\mathop{\mathrm{argmin}}_{\theta}L^{base}(D^{train};\theta,\phi)+R(\theta)$
(2)

where R is the regularization term, and L represents the loss function for
training the base learner.

A trained meta-learning model is then evaluated on a set of held-out tasks
termed as the meta-testing set, $S={(D^{train}_{j},D^{test}_{j})}^{J}_{j=1}$.
Once again, $D^{train}$ and $D^{test}$ form the support and query sets for the
meta-testing episode, respectively. The evaluation is done over a distribution
of the test tasks, represented by,

$\theta=E_{S}[L^{meta}(D^{test};\theta,\phi)]$ (3)

where $\theta=A(D^{train};\phi)$.

###  3.2 Prototypical Network Approach

To prepare a baseline for comparison with the proposed meta-learning approach,
the prototypical networks are adopted in this work. Prototypical network is a
distance metric based meta-learning technique which computes the mean vector
to represent each class of a support set. This vector is termed the prototype
for the particular class in the support set.

Conforming to the aforementioned formulation, a subset of classes is randomly
selected to compose one training episode. For each training episode, a support
set S, and a query set Q are sampled. For a N-way K-shot setting, the support
set is sampled to contain N different classes, with K examples per class.
Prototypical networks first use the embedding function $f_{\phi}$ to map the
images to a common feature space. Each class c in S is then represented in the
embedding space by a prototype vector $V_{c}$, which is computed as the mean
vector of the embedded inputs for all the samples in S, corresponding to class
c, as follows:

$V_{c}=\frac{1}{K}\sum_{(x_{j},y_{j})\in S_{c}}f_{\phi}(x_{j})$ (4)

Each sample in the query set is then mapped to the embedding space. The
Euclidean distance (equation 5 — Minkowski distance of second order — is
computed between the prototype vector $V_{c}$ of each class in S, and the
query example. The query example is classified under the class that it is
closest in distance to. The embedding network is trained by back propagating
the negative log loss of the query example’s distance from its classifier
class vector.

$d(p,q)=\sqrt{\sum_{k=1}^{K}(p_{k}-q_{k})^{2}}$ (5)

where, $p$ and $q$ denote the prototype and query images’ embedding vectors,
$K$ denotes the dimensionality of the embedding vectors, and $d$ denotes the
Euclidean distance.

During the meta-testing phase, the embedding network is fixed, with the rest
of the method for classifying a query example remaining the same — map the
support and query sample to the embedding space, compute prototype vectors for
each class in the support set, find the euclidean distance of the query sample
from each prototype vector and classify under the class with least distance.

###  3.3 Pre-train and Base Learner Approach

As presented in Section 3.1, the meta-training phase intends to learn an
embedding model $f_{\phi}$, that projects the input on to a lower dimensional,
but a representative, feature space. At the crux of the work presented in this
paper, lies the importance of the generalizability and characteristic nature
of this embedding model. It should be readily transferable to any new task,
such that a projection onto the learned feature space solves the new set of
tasks with very limited examples.

A great number of meta-learning algorithms aim to learn the embedding model by
training them on training tasks, modeled very similarly to the target tasks —
episodic learning used in prototypical networks, for instance [11]. However,
this paper follows the approach in [20], and posits that a model that is pre-
trained on a classification task, learns to produce highly representative and
discriminative embeddings that can be leveraged by the base learner
effectively, even when transferred to a new task.

Figure 1: Flow diagram depicting the proposed meta training and testing
approach using an embedding network.

To this end, the training datasets from each episode of the meta-training
phase are merged into one large dataset $D^{new}$, given as,

$D^{new}={(x_{t},y_{t})}^{K}_{k=1}\\\
=\bigcup{D^{train}_{1},...,D^{train}_{i},...,D^{train}_{I}}$ (6)

where $D^{train}_{i}$ is the training dataset from T for episode i.

The embedding model then becomes,

$\phi=\mathop{\mathrm{argmin}}_{\phi}L^{ce}(D_{new};\phi)$ (7)

where $L^{ce}$ denotes the cross-entropy loss between predictions and ground-
truth labels. In effect, this reduces to the task of training a simple
supervised image classifier.

The meta-testing phase, however, is modeled quite similarly to most meta-
learning setups. Concretely, for every task ($D^{train}_{j}$ , $D^{test}_{j}$)
that is sampled from meta-testing distribution, we train a base learner on
$D^{train}$j that makes predictions on $D^{test}_{j}$. In this paper, the base
learner is modeled as a linear model The linear model is attributed with
parameters $\theta={W,b}$ to include a weight term W and a bias term b, that
are learned through,

$\theta=\mathop{\mathrm{argmin}}_{W,b}L^{ce}_{t}(Wf_{\phi}(x_{t})+b,y_{t})+R(W,b)$
(8)

where R is the regularization term.

In addition, this paper also presents alternative base learners that leverage
the learned embedding model — decision trees, and nearest-neighbor classifiers
with the Minkowski distance metric of the second order.

Concretely, the embedding model is learned during the meta-training phase by
back propagating the cross-entropy loss of a classification task, formulated
using the complete set of training examples from each episode — namely
$D^{new}$. The learned embedding network is the fixed, and a base learner —
linear, tree, or any other feature classifier — is learned for each episode of
the meta-testing phase. Particularly, the supervised learner trains with the
few examples available in the support set $D^{train}$ of an episode in S, and
makes predictions for the corresponding query set $D^{test}$ of the episode.
Hence, a powerful embedding model would allow the base learner to learn to
classify on a new task, with just a few examples. Hence, the significant
difference lies in the approach adopted to learn the embedding model during
meta-training, while meta-testing still adopts episodic learning, but while
training a downstream base classifier for each episode. The proposed approach
is similar to the methods adopted in [17, 14] and differs from [4, 15] which
additionally fine-tune the embedding model $f_{\phi}$ during the meta-testing
phase. The proposed learning approach is depicted in Figure 1.

##  4 Experiments

This paper conducts experiments on two widely accepted benchmark datasets for
skin lesion classification: the ISIC 2018 Task 3 Dataset and [2] the Derm7pt
Dataset [8]. All experiments in this research can be reproduced through the
implementation that will be made available at
https://github.com/karthik-d/Few-Shot-Learning-Skin-Analysis.

###  4.1 Datasets

ISIC-2018 Task-3 [2]: This dataset consists of 10,015 dermoscopic images.
These are labeled by expert pathologists into one of the seven skin lesion
categories. Based on a standard train-test split of 80-20, a total of 7,515
images compose the train set, while the remaining 2,500 form the test set. For
the purpose of experiments, the authors resized the images from $600\times
450$ pixels to $224\times 224$ pixels, and chose four and three classes in the
meta-train and meta-test sets respectively, to formulate the meta-learning
classification episodes. Figure 2 shows sample images from the dataset. Table
1 represents the data split.

Figure 2: Samples images from each class of the ISIC-2018 dataset. Table 1: Split of the classes in the ISIC dataset into training and testing sets. Class Abbr. | Class Name | Number of Images | Data Split
---|---|---|---
NV | Nevus | 6741 | Train
MEL | Melanoma | 1119 | Train
BKL | Keratosis | 1101 | Train
BCC | Basal Cell Carcinoma | 517 | Train
AKIEC | Actinic | 331 | Test
VASC | Vascular | 143 | Test
DF | Dermatofibroma | 116 | Test

Derm7pt [8]: This dataset that includes over 2000 clinical and dermoscopy
color images belonging to 20 distinct diagnostic classes, further grouped into
5 super classes. It includes structured metadata to benchmark the training and
evaluation of automated diagnosis systems. The dataset provides a 7-point skin
lesion malignancy checklist, and bases the predictions on this. The original
image size is $768\times 512$ pixels. Once again, for experimentation, the
dermoscopic images are used, and these are resized to $224\times 224$. The
standard train-test splits benchmarked by the dataset providers is used in
these experiments. To facilitate evaluation of the proposed meta-learning
algorithm, the 5 super classes are considered. The 3 classes with the least
number of images per class are used for the meta-training phase, and the other
2 are used for meta-training. Table 2 represents the data split and the sample
image from Derm7pt is represented in the 3.

Figure 3: Sample images from each class of the Derm7pt dataset. Table 2: Split of the classes in the Derm7pt dataset into training and testing sets. Class Abbr. | Class Name | Number of Images | Data Split
---|---|---|---
NEV | Nevus | 575 | Train
MEL | Melanoma | 252 | Train
MISC | Miscellaneous | 97 | Test
SK | Seborrheic Keratosis | 45 | Test
BCC | Basal Cell Carcinoma | 42 | Test

Both the datasets are characterized by a heavy class imbalance. The classes
with the least number of data samples are moved to the test set and used as
the domain to sample the meta-test tasks, to model the real-world setting and
assess the meta-learning model on a low-data regime. In addition, this also
mitigates the issue of class imbalance learning during the meta-training
phase.

###  4.2 Implementation Details

Network Architecture: The network for the embedding model is trained with
three different backbones. The first is Conv64 — a simple 6-layer
Convolutional Neural Network (CNN) with each convolution layer comprising 64
filters with kernel size $3\times 3$. This is followed by a batch-
normalization layer, Rectified Linear Unit (ReLU) activation, and a $2\times
2$ max-pooling layer.

The second and third backbones are ResNet18 and ResNet50, respectively,
without the fully-connected layers at the end. During the meta-training phase,
the feature outputs from the backbones are connected to a N-neuron fully-
connected layer, where N is the number of classes in the training set Dnew.
This effectively sets up a supervised classification task, as described in
Section 3. During the meta-testing phase, the N-neuron fully-connected layer
is removed, and the trained network outputs a feature vector. This has a
cardinality of 64, 1000 and 2048 for Conv64, ResNet18 and ResNet50 backbones
respectively. In each meta-test episode, the network extracts a feature vector
representation for each image in the support and query sets, which in turn is
used to train the downstream base learner on the support set and make
predictions for the query set images.

Training Parameters: The experiments presented in this paper used the Adam
optimizer with an initial learning rate of $1e-{03}$. The decay rates for the
first and second moments of gradient are set at 0.9 and 0.95 respectively. No
weight decaying is performed. A grid search was performed to determine the
optimal learning and decays rates, since these hyper-parameters have a
considerable impact on the model performance. Each training batch consisted of
32 samples. The embedding models were trained for 90 epochs on the ISIC
dataset, and for 40 epochs on the Derm7pt dataset. The respective embedding
models were then used to perform the meta-testing experiments with various
few-shot settings, training and predicting with a downstream base learner for
each episode of the testing phase.

Data Augmentation: Data augmentation techniques are adopted when training the
embedding network, to allow it to learn to project to a robust and
discriminative feature space. During meta-training — that is when training the
embedding network, random crop, color jittering, and random horizontal flip as
in [12]. For the meta-testing stage, an N-way K-shot downstream base
classifier is trained for each episode. The same set of augmentations are
applied here, as part of the ablation study. We use the implementations in
scikit-learn [<cite 1 p2>] for implementing the base learner(s).

##  5 Results and Discussion

Table 3: Comparison of performance with the prototypical network — MetaDerm, used for skin lesion classification. All experiments were performed for 2-way classification with K-shots as decribed in the column headers. Metrics are macro-averaged over all classes. _Acc_ is the accuracy metric. _AuRoc_ is the area under receiver-operating-curve metric. Model | Metric | ISIC-18 | Derm7pt
---|---|---|---
1-shot | 3-shot | 5-shot | 1-shot | 3-shot | 5-shot
MetaDerm [13] | Acc | 59.3 | 67.9 | 73.0 | 62.5 | 63.9 | 66.7
AuRoc | 61.6 | 70.2 | 75.4 | 60.2 | 65.7 | 70.5
Ours-Conv64-LR | Acc | 58.9 | 68.3 | 73.1 | 62.3 | 64.6 | 67.0
AuRoc | 69.1 | 72.3 | 80.6 | 63.1 | 70.3 | 79.8
Ours-Conv64-SVM | Acc | 59.2 | 69.5 | 73.8 | 63.1 | 66.2 | 67.4
AuRoc | 71.4 | 72.1 | 82.1 | 70.3 | 74.2 | 78.8
Ours-ResNet18-SVM | Acc | 59.9 | 74.1 | 78.2 | 63.7 | 69.8 | 70.2
AuRoc | 72.3 | 76.5 | 79.8 | 72.2 | 74.1 | 81.1
Ours-ResNet50-SVM | Acc | 65.9 | 76.5 | 79.6 | 64.0 | 74.3 | 78.1
AuRoc | 72.6 | 77.4 | 80.1 | 72.9 | 78.6 | 83.2

Table 3 compares the model trained using the proposed meta-learning algorithm
with an implementation of prototypical networks. It is evident that the
proposed base learner approach consistently outperforms the prototypical
network, even when using the same backbone network. In terms of accuracy,
there is about 0.5-2% improvement, and much more in terms of the AuRoc metric.
It is worth noting that employing more complex backbones — ResNet18 and
ResNet50, shows further improvements. This suggests that the embedding network
is able to find a more representative feature space with a more complex
network, when trained using the proposed meta-training method. However, the
Logistic Regression (LR) and Support Vector Machine (SVM) classifiers exhibit
lower performance that the prototypical network in the 1-shot regime. This is
due to lesser number of training examples available to the downstream base
learner in these cases, which is not a concern in prototypical networks that
only average the feature vectors across all samples of a class.

Table 4: Ablation study with Conv64 network as the backbone using accuracy metric. _DT_ refers to a decision tree classifier used as the base learner. _NN_ describes a nearest neighbor classifier. _LR_ describes Logistic Regression classifier. _L2-Norm_ refers to normalization of the embedding vector to a unit hypersphere. _Aug_ refers to the application of the five augmentation techniques describes in Section 4.2 on each image of the support set. All experiments are performed for 2-way classification. DT | NN | LR | L2-Norm | Aug | ISIC-18 | Derm7pt
---|---|---|---|---|---|---
1-shot | 5-shot | 1-shot | 5-shot
✓ |  |  |  |  | 56.9 | 60.3 | 59.4 | 62.3
✓ |  |  |  | ✓ | 57.1 | 61.1 | 60.1 | 62.3
| ✓ |  |  |  | 60.4 | 67.8 | 62.4 | 63.1
| ✓ |  |  | ✓ | 60.7 | 67.7 | 62.5 | 63.4
|  | ✓ |  |  | 58.9 | 73.1 | 62.3 | 67.0
|  | ✓ | ✓ |  | 59.1 | 73.4 | 62.5 | 67.4
|  | ✓ | ✓ | ✓ | 62.4 | 75.2 | 64.4 | 68.9

The ablation study is performed on the Conv64 backbone for the embedding
network by applying combinations of one or more of the following elements of
the proposed classification pipeline: (1) A Decision Tree (DT) classifier is
used as the downstream base learner; (2) A Nearest Neighbor (NN) classifier is
used as the base learner; (3) A Logistic Regression (LR) classifier is used as
the base learner; (4) L2 normalization is applied on the embedding vector
obtained from the trained backbone during the meta-test phase to capsulate the
feature values into a unit hypersphere; (5) Five augmentation methods are
applpied to each image in the support set to increase the size of the training
data available to the base learner during their meta-testing phase. In effect,
each image of the support set is used to produce 5 more images. Techniques
used include random crop, color jittering, random horizontal and vertical
flipping and random rotations of upto 10 degrees in either direction.

In general, it can be noted from Table 4 that normalization and augmentation
have reasonable impact in improving the performance of the LR classifier. The
increase in performance with augmentation is higher for the 1-shot case. This
is potentially due to stark increase in number of samples available for
training the base learner due to augmentation. With the 5-shot case, the
effect of more augmentations atop 5 samples per class is less pronounced. DT
and NN classifiers, in general, perform well in the 1-shot regime, and their
effect numbs down with increasing number of samples. This is intuitive, and is
comparable with the characteristically lower margins between the performance
of the proposed method and prototypical networks for the 1-shot case (refer to
Table 3). Finally, augmentations have little or no effect on the performance
of NN and DT classifiers.

##  6 Conclusion and Future Work

This paper presents a baseline learning approach for few-shot image
classification in the meta-learning context. This approach has been
underappreciated in the literature thus far, and this is the first application
of this method for skin image classification in the few-shot regime. The
proposed approach consistently outperforms the protoypical network on two
different datasets, even when they adopt the same backbone network.

Furthermore, this paper posits empirically that even a simple linear model
suffices to generalize well for a few-shot learning task, as long as a good
representation of the data is given prepared — the embedding network.The
efficacy in the proposed meta-training approach may be traced back to the
merging of all the N, K-way classification episodes of the meta-training
dataset. Training on this single but harder N K-way classification task proves
to learn a more generalizable and representative feature space through the
embedding network. Further, the use of a single larger task for training also
facilitates the effective use of more complex backbones without the concern of
overfitting, and can learn more complex embedding spaces that transfer well to
meta-testing set.

Our future work will pivot around methods to improve the few-shot
classification performance, specifically by improving the quality of the
embeddings through different backbone networks; experiment with different
downstream classifiers; prospect the potential of the proposed method to other
logical objectives such as image segmentation; and test out the technique on
more complex medical datasets. A viable step in improving the quality of
embeddings produced direction could be to distill the embedding network to
refine the learnt feature space.

## References

  * [1] Afham, M., Khan, S.H., Khan, M.H., Naseer, M., Khan, F.S.: Rich semantics improve few-shot learning. CoRR abs/2104.12709 (2021), https://arxiv.org/abs/2104.12709
  * [2] Codella, N., Rotemberg, V., Tschandl, P., Celebi, M.E., Dusza, S., Gutman, D., Helba, B., Kalloo, A., Liopyris, K., Marchetti, M., et al.: Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1902.03368 (2019)
  * [3] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. pp. 248–255 (2009). https://doi.org/10.1109/CVPR.2009.5206848
  * [4] Dhillon, G.S., Chaudhari, P., Ravichandran, A., Soatto, S.: A baseline for few-shot image classification. CoRR abs/1909.02729 (2019), http://arxiv.org/abs/1909.02729
  * [5] Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. CoRR abs/1703.03400 (2017), http://arxiv.org/abs/1703.03400
  * [6] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition (2015). https://doi.org/10.48550/ARXIV.1512.03385, https://arxiv.org/abs/1512.03385
  * [7] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90
  * [8] Kawahara, J., Daneshvar, S., Argenziano, G., Hamarneh, G.: Seven-point checklist and skin lesion classification using multitask multimodal neural nets. IEEE Journal of Biomedical and Health Informatics 23(2), 538–546 (mar 2019). https://doi.org/10.1109/JBHI.2018.2824327
  * [9] Kim, J., Kim, T., Kim, S., Yoo, C.D.: Edge-labeling graph neural network for few-shot learning. CoRR abs/1905.01436 (2019), http://arxiv.org/abs/1905.01436
  * [10] Koch, G.R.: Siamese neural networks for one-shot image recognition (2015)
  * [11] Laenen, S., Bertinetto, L.: On episodes, prototypical networks, and few-shot learning. CoRR abs/2012.09831 (2020), https://arxiv.org/abs/2012.09831
  * [12] Lee, K., Maji, S., Ravichandran, A., Soatto, S.: Meta-learning with differentiable convex optimization. pp. 10649–10657 (06 2019). https://doi.org/10.1109/CVPR.2019.01091
  * [13] Mahajan, K., Sharma, M., Vig, L.: Meta-dermdiagnosis: Few-shot skin disease identification using meta-learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 730–731 (2020)
  * [14] Oreshkin, B.N., López, P.R., Lacoste, A.: TADAM: task dependent adaptive metric for improved few-shot learning. CoRR abs/1805.10123 (2018), http://arxiv.org/abs/1805.10123
  * [15] Raghu, A., Raghu, M., Bengio, S., Vinyals, O.: Rapid learning or feature reuse? towards understanding the effectiveness of MAML. CoRR abs/1909.09157 (2019), http://arxiv.org/abs/1909.09157
  * [16] Ravi, S., Larochelle, H.: Optimization as a model for few-shot learning. In: ICLR (2017)
  * [17] Rusu, A.A., Rao, D., Sygnowski, J., Vinyals, O., Pascanu, R., Osindero, S., Hadsell, R.: Meta-learning with latent embedding optimization. CoRR abs/1807.05960 (2018), http://arxiv.org/abs/1807.05960
  * [18] Snell, J., Swersky, K., Zemel, R.: Prototypical networks for few-shot learning. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 30. Curran Associates, Inc. (2017), https://proceedings.neurips.cc/paper/2017/file/cb8da6767461f2812ae4290eac7cbc42-Paper.pdf
  * [19] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2818–2826 (2016). https://doi.org/10.1109/CVPR.2016.308
  * [20] Tian, Y., Wang, Y., Krishnan, D., Tenenbaum, J.B., Isola, P.: Rethinking few-shot image classification: a good embedding is all you need? CoRR abs/2003.11539 (2020), https://arxiv.org/abs/2003.11539
  * [21] Vinyals, O., Blundell, C., Lillicrap, T., Kavukcuoglu, K., Wierstra, D.: Matching networks for one shot learning. In: Proceedings of the 30th International Conference on Neural Information Processing Systems. p. 3637–3645. NIPS’16, Curran Associates Inc., Red Hook, NY, USA (2016)
  * [22] Vinyals, O., Blundell, C., Lillicrap, T.P., Kavukcuoglu, K., Wierstra, D.: Matching networks for one shot learning. CoRR abs/1606.04080 (2016), http://arxiv.org/abs/1606.04080
  * [23] Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 40(6), 1452–1464 (2018). https://doi.org/10.1109/TPAMI.2017.2723009