2021

[1,2]Sat byul Seo

[1]Department of Mathematics Education, Kyungnam University, Changwon-si,
51767, Gyeongsangnam-do, Republic of Korea

2]Department of Mathematics, University of Texas at Arlington, Arlington,
76019, Texas, United States

# Quantifying constraints determining independent activation on NMDA receptors
mediated currents from evoked and spontaneous synaptic transmission at an
individual synapse

<EMAIL_ADDRESS>Jianzhong Su<EMAIL_ADDRESS>*  [

###### Abstract

A synapse acts on neural transmission through a chemical process called
synapses fusion between pre-synaptic and post-synaptic terminals. Presynaptic
terminals release neurotransmitters either in response to action potential or
spontaneously independent of presynaptic activity. However, it is still
unclear the mechanism of evoked and spontaneous neurotransmission that
activate on postsynaptic terminals. To address this question, we examined the
possibility that spontaneous and evoked neurotransmissions using mathematical
simulations. We aimed to address the biophysical constraints that may
determine independent activation on N-methyl-D-asparate (NMDA) receptor
mediated currents in response to evoked and spontaneous glutamate molecules
releases. In order to identify the spatial relation between spontaneous and
evoked glutamate release, we considered quantitative factors, such as size of
synapses, inhomogeneity of diffusion mobility, geometry of synaptic cleft, and
release rate of neurotransmitter. Simulation results showed that as a synaptic
size is smaller and if the cleft space is more cohesive in the peripheral area
than the centre area, then there is high possibility of having crosstalk of
two signals released from center and edge. When a synaptic size is larger, the
cleft space is more affinity in the central area than the external area, and
if the geometry of fusion has a narrower space, then those produce more
chances of independence of two modes of currents released from center and
edge. The computed results match well with existing experimental findings and
serve as a road map for further exploration to identify independence of evoked
and spontaneous releases.

###### keywords:

synaptic transmission, neurotransmission, NMDA receptor, mathematical
modeling, differential equations

##  1 Introduction

A synapse consists of the three components, a presynaptic neuron or terminal,
a postsynaptic neuron or terminal, and a synaptic cleft. Networks of neurons
and synapse play a key role of communication of electric signals of brain, and
are responsible for most of brain functionality. Chemical synapses between
neurons, are the main channels of information flow and storage in the brain.
Synaptic transmission between neurons is involved in most of what the brain
does. In mouse cortex are, synaptic neuropil is 84% in terms of
volumeSchuz1989 . When a neuron is active, an electrical impulse travels down
its nerve fiber and causes the release of chemical neurotransmitters from its
terminal on the presynaptic neuron. Presynaptic terminals contain pools of
synaptic vessels. They are small membrane bounded organelles. These vesicles
are filled with neurotransmitters mainly in the form of glutamate molecules.
When neurotransmitters, for example, glutamate molecules are released from the
presynaptic neuron, they diffuse into the cleft, and then the
neurotransmitters spread out to a narrow space between pre and post synaptic
neurons. The gap between pre and post synaptic terminals is about 20nm wide
and is called a synaptic cleft. The synaptic cleft consists of fluids,
proteins and other molecular obstacles. The some of released neurotransmitters
in the cleft bind to receptors on the postsynaptic neuronledoux2003synaptic .
These chemical neurotransmitters then produce secondary currents in the
postsynaptic neuron.

The random synaptic release events typically activate receptors within a
single postsynaptic site and give rise to miniature postsynaptic currents, and
therefore they have been extremely instrumental in analysis of unitary
properties of neurotransmission. In 1994, Murphy and colleagues found that
spontaneous miniature glutamate release modulates postsynaptic enzyme
activityKenny1994 . Sutton and colleagues showed that minis keep resting
protein synthesis in check and respond to stimuli that strengthen synapses by
blocking minis and increasing dendritic protein synthesisSutton2004 .
Spontaneous neurotransmission has been mentioned a homeostatic form of
synaptic plasticity and induction of synaptic scaling. Spontaneous
neurotransmission has an independent role in neuronal communication that is
distinct from that of evoked releaseKavalali2014  However, the process of
spontaneous neurotransmitter release is still unclear. It has been questioned
whether spontaneous release events and evoked release events originate from
the same vesicular pathway in presynaptic neuronsBauerfeind1994 . The relation
of evoked and spontaneous neurotransmitter releases and how they are
distributed spatially have not been precisely studied due to the lack of
direct experimental measurement. In 2008, David Zenisek found some evidence
for the spatial segregation of spontaneous and evoked neurotransmissions that
evoked release occurs from ribbon and spontaneous release happens from
extraribbon locations in a ribbon-type synaptic terminal, in the goldfish
retinal bipolarZenisek2008 . Kavalali and colleagues examined the evidence
that spontaneous and evoked vesicles originate from different pool of
glutamate stores and after releasing, neurotransmitters activate non-
overlapping postsynaptic NMDA receptors populationsSara2005 ; Atasoy2008 ;
Kavalali2011 . Melom and colleagues showed that even though release
probability is not correlated between evoked and spontaneous release of
fusion. Neuronal dynamics have two spatially segregated and regulated
information channels to induce evoked or spontaneous fusion signals
independentlyMelom2013 . Peled and colleagues suggested that although
individual synapses can participate in both evoked and spontaneous
neurotransmitter release, there is a highly well activated synapse with a
preference for only one mode of transmissionPeled2014 . Schneggenburger and
colleagues found that separate functions for $Ca^{2+}$ evoked release and
spontaneous transmissions are not necessarily from different origins of two
vesicular fusionSchneggenburger2015 .

From those studies, there have been assumed that spontaneous and evoked
process are segregated and regulated independently. But it is unclear how this
separation of synaptic currents in NMDA receptors is distributed across
individual synapses because of the limited resolution of optical microscopic
recording. It is still challenging to show how postsynaptic neurons
distinguish evoked and spontaneous neurotransmission and differentially
activate postsynaptic signaling. Reese and Kavalali showed that two signals
from spontaneous and evoked release are not correlated significantly although
spontaneous and evoked release driven NMDA receptor mediated $Ca^{2+}$
transients often occurs at the same synapseReese2016 .

To address this question, we examined the possibility that spontaneous and
evoked neurotransmission using the mathematical model improved from our
previous studyAtasoy2008 . We aimed to address the biophysical constraints
that may determine independent activation on N-methyl-D-asparate (NMDA)
receptor mediated currents in response to evoked and spontaneous glutamate
molecules releases.

Identifying the mechanism of activation of NMDA receptors is important since
the NMDA receptors (NMDARs), a family of L-glutamate receptors, are a main
target for cognitive enhancement and plays a significant role in neural
plasticity including long-term potentiation and long-term
depressionNewcomer2000 ; Collingridge2013 ; Kumar2015 . These explain the
importance of NMDARs for learning and memory. Recent studies also indicated an
age-associated NMDA receptor hypofunction and memory impairment are linked
each other, and it provides evidence age-associated may enhance oxidative
stressKumar2015 .

In this way, we described the boundaries of different factors, including size
and geometry of synaptic cleft, the neurotransmitter release rate s from
presynaptic terminals, different rates glutamate mobility that permit
independent synaptic events in the limited space of a synapse cleft. The
geometry of synapse, especially for a small size, has not been described
clearly with the limitation of microscopy. In order to simulate two modes of
neurotransmission, we assumed that spontaneous and evoked release with
different factors including a size, geometry of synaptic cleft, different
glutamate mobility rate in the cleft, and the release rate of
neurotransmitters from presynaptic terminals.

##  2 Methodology

###  2.1 Physical setting

The computational model is calculated by the diffusion and kinetics in a cubic
domain $\Omega$ of $1000\mbox{nm}\times 1000\mbox{nm}\times 1000\mbox{nm}$
excluding synaptic terminals. Let S matrix for boundary conditions,

$S=S(x,y,z)=\begin{cases}1,&\mbox{ within $\Omega$ }\\\ 0,&\mbox{otherwise
}\end{cases}.$

The presynaptic and postsynaptic terminals (two $600\mbox{nm}\times
600\mbox{nm}$ surface areas facing each other) inside $\Omega$ are non-
permeable by glutamate molecules and are excluded from computation. A cleft of
20 nm heights separates the presynaptic and postsynaptic terminals in the base
model. We assumed that size of synapse is $600\mbox{nm}\times
600\mbox{nm}\times 20\mbox{nm}$ as for the base model. Geometry for a synapse
is a three dimensional array, as shown Figure 1A, 1B.

Figure 1: Geometric domain for computational model in synaptic glutamate
diffusion:  A. Illustrates the computational domain in 3-D in the transparent
portion, inside a large cube. B. Top: a side-view of A shows a cleft of 20 nm
height separating two terminals; bottom: a top-view of A shows the cleft
surface (in color, dimension $600\mbox{nm}\times 600\mbox{nm}$) of a large
synapse. C. Two directions that are tracked peak open probabilities at each
location on postsynaptic when glutamate molecules release from the
center($R_{C}$, maybe evoked release?) and release from the edge($R_{E}$, may
be spontaneous release?). D. Geometric vesicle fusion pore for computational
model in synaptic glutamate diffusion.

For the base model (600nm by 600nm cross-section), we assumed the NMDA
receptor density of 40 $\mu m^{2}$. On the postsynaptic cleft, there are
evenly distributed 25 spots, arranged in 5 by 5 array using row-major
ordering. The concentration time course and $P_{open}$ are calculated at each
receptor. With our assumption, a release events near the center represents
evoked transmission, whereas a fusion event at the periphery of the
postsynaptic corresponds to spontaneous receptorAtasoy2008 . Two
representative locations of receptors are used as release sites, $R_{C}$ for
central release(released from the center site) and $R_{E}$ for release at the
edge(released from the edge site) as shown in Figure 1C.

We illustrated the situation where evoked glutamate release at the center
activates evoked receptor kinetics at center and spontaneous receptor kinetics
at edge in Figure 2A; analogously, spontaneous glutamate release at the cleft
edge activates both evoked receptor kinetics at center and spontaneous
receptor kinetics at edge in Figure 2B. For a large size synapse under normal
conditions (assumed in the base model), the two signals are independent as
central versus edge release yield to synaptic currents that differs in the
amplitude (or open probability) of 10 magnitudes. We begin with the base model
with instantaneous release of 4000 glutamate molecules.

Another assumptions were considered the limited rate release with additional
components of a vesicle and a fusion pore inside the presynaptic terminal as
shown in Figure 1D. The vesicle size is $40\mbox{nm}\times 40\mbox{nm}$, the
fusion pore size mimics release speed. We tested the release of glutamate
molecules from a 10 nm diameter fusion pore for a typical release and one
narrow 2 nm diameter fusion pore as a slower release.

###  2.2 Mathematical Modeling

####  2.2.1 Diffusion model from presynaptic neuron to synaptic cleft

The classical heat diffusion equation was used in the glutamate release
process on the presynaptic terminal. The heat diffusion model is reasonable
because of the size of glutamate molecules, and relatively large numbers of
the molecules being release as one time. We assumed that 4000 glutamate
molecules release out from a point sourceNielsen2004 ; Atasoy2008 . With an
estimated current dose-response profiles obtained from measurementsAnson1998 ;
Patneau1990 , it was widely believed that the glutamate-binding sites become
saturated after each synaptic vesicle releasedFrerking1995 , and the estimated
high glutamate concentration in the cleft after the release of a single
synaptic vesicle is about 1-5mMClements1996 ; Diamond1997 . This is consistent
with simulated values using Equation 1 with initial concentration of 4,000
molecules. The governing equation is

$\frac{\partial{C}}{\partial{t}}=D_{glut}\bigg{(}\frac{\partial^{2}{C}}{\partial^{2}{x}}+\frac{\partial^{2}{C}}{\partial^{2}{y}}+\frac{\partial^{2}{C}}{\partial^{2}{z}}\bigg{)},$
(1)

where $[G]=C(x,y,z,t)$ is a glutamate concentration as a function of time and
location in the vesicular space, the synaptic cleft, and the external space,
whereas $t\in[0,\infty)$ and $(x,y,z)$ is in a open region $\Omega$. The
coefficient $D_{glut}$ is the thermal diffusivity.

####  2.2.2 Factors in synapse

The diffusion coefficients $D_{glut}$ can represent the inhomogeneity of media
within the cleft with varying $D_{glut}$. The larger diffusion coefficient
value implies for glutamate to flush out to external space quickly. For a
typical synapse, the value of $0.4\mu m^{2}/ms$ is sufficient to represent
typical case of glutamate mobilityNielsen2004 . However, this may or may not
be the case with small synapses, where the evoked and spontaneous releases
occur in much closer space. The exact value of $D_{glut}$ is unknown, we used
values ranging from $0.1$ to $0.75\mu m^{2}/ms$ as feasible permeability
values for synapse. Further we can create multiple zones where $D_{glut}$
could have various values in several zones. In the models, diffusion
coefficients $D_{glut}$ were taken depending on the location within or outside
the synapses. In base model, we have a value of $D_{glut}=0.4\mu m^{2}/ms$
within the cleft and $D_{glut}=0.75\mu m^{2}/ms$ in the external space.

We considered other constraints to find favorable conditions where the signals
can be independent. One way is to assume the components in the cleft to be
different compositions. We make the center area to be less diffusive for
glutamate molecules than the peripheral region inside cleft. We created an
interior zone($\Omega^{+}$) and an exterior zone($\Omega^{-}$) as shown in
Figure 6A, 6B. $D_{glut}$ could vary zone to zone, and it represents the
inhomogeneity of material components in the cleft. Numerically, explicit
finite difference scheme with piecewise continuous coefficients has accuracy
of second order $O(h^{2})$ for the synaptic diffusion problemsSeo2017 ;
Seo2019 .

Another possibility is to reduce the amount of glutamate molecules released
per vesicle in the partial release (called kiss-and-go) scenarios, and/or the
glutamate is released in slower release rate due to partial opening of
vesicles in the membrane. When simulating the vesicular diffusion,
$D_{glut}=0.15\mu m^{2}/ms$ and $D_{glut}=0.0375\mu m^{2}/ms$ are taken for
10nm and 2nm fusion pore respectively.

####  2.2.3 Post-synaptic model

Under the glutamate mediation, receptors such as NMDA, AMPA or GABA activate
the channels for allowing ion currents. To determine the opening probability,
$P_{open}$, of an individual receptor, a state model is proposed based on the
maximum likelihood method using experimental data. The current model
consisting of three closed and two open states (3C2O) is used by Popescu
dataPopescu2004 ; Atasoy2008 . Our calculation will be based on this 3C2O
model. We note that the glutamate concentrations at the receptor locations are
included in the reaction rates of two coupled states, $C^{M}$and $C^{U}$.

The experimental data indicated the populations of synapses are clustered
around 3 groups of distinct characters. We primarily tested our hypotheses on
the medium group called M-mode is modeled by

$C^{U}\underset{58}{\overset{39[G]}{\rightleftharpoons}}C^{M}\underset{116}{\overset{19[G]}{\rightleftharpoons}}C_{1}\underset{173}{\overset{150}{\rightleftharpoons}}C_{2}\underset{2,412}{\overset{902}{\rightleftharpoons}}C_{3}\underset{1,283}{\overset{4,467}{\rightleftharpoons}}O_{1}\underset{526}{\overset{4,630}{\rightleftharpoons}}O_{2},$
(2)

The ode systems were built using kinetic model for open probability at each
receptor when glutamate molecules releases occur. The open probability is
obtained by $P_{open}=O_{1}(t)+O_{2}(t)$ Atasoy2008 . The populations satisfy
the following system of ordinary differential equations:

$\displaystyle\frac{dC^{U}}{dt}$ $\displaystyle=$ $\displaystyle
58C^{M}-39C(x,y,z,t)C^{U},$ $\displaystyle\frac{dC^{M}}{dt}$ $\displaystyle=$
$\displaystyle(116C_{1}-19C(x,y,z,t)C^{M})-(58C^{M}-39C(x,y,z,t)C^{U}),$
$\displaystyle\frac{dC_{1}}{dt}$ $\displaystyle=$
$\displaystyle-(116C_{1}-19C(x,y,z,t)C^{M})+(173C_{2}-150C_{1}),$
$\displaystyle\frac{dC_{2}}{dt}$ $\displaystyle=$
$\displaystyle(2412C_{3}-902C_{2})-(173C_{2}-150C_{1}),$
$\displaystyle\frac{dC_{3}}{dt}$ $\displaystyle=$
$\displaystyle-(2412C_{3}-902C_{2})+(1283O_{1}-4467C_{3}),$
$\displaystyle\frac{dO_{1}}{dt}$ $\displaystyle=$
$\displaystyle(526O_{2}-4630O_{1})-(1283O_{1}-4467C_{3}),$
$\displaystyle\frac{dO_{2}}{dt}$ $\displaystyle=$
$\displaystyle-(526O_{2}-4630O_{1}).$

####  2.2.4 Computational model

We simulated the process of glutamate release from presynaptic sites into the
synaptic cleft by solving the heat diffusion equation and the synaptic
transmitter/receptor kinetic process by solving a system of ordinary
differential equations. We used the problem-solving environment MATLAB, which
provides tools for solving linear equations in a numerical way. MATLAB is
efficient to display results visually through graphs in its post-
processingShampine1997 . In order to solve the heat equation numerically to
achieve glutamate concentration, we use on explicit difference method (forward
time, centered space) that is implemented in MATLAB codes.

$\displaystyle C^{n+1}_{i,j,k}$ $\displaystyle=$ $\displaystyle
C^{n}_{i,j,k}+\alpha[S_{i+1,j,k}C^{n}_{i+1,j,k}+S_{i-1j,k}C^{n}_{i-1j,k}+S_{i,j+1,k}C^{n}_{i,j+1,k}$
$\displaystyle+$ $\displaystyle
S_{i,j-1,k}C^{n}_{i,j-1,k}+S_{i,j,k-1}C^{n}_{i,k,j,k-1}-(S_{i,j,k}C^{n}_{i,j,k}+S_{i-1,j,k}C^{n}_{i,j,k}$
$\displaystyle+$ $\displaystyle
S_{i,j+1,k}C^{n}_{i,j,k}+S_{i,j-1,k}C^{n}_{i,j,k}+S_{i,j,k+1}C^{n}_{i,j,k}+S_{i,j,k-1}C^{n}_{i,j,k})],$

where $C^{n}_{i,j,k}=C(x_{i},y_{j},z_{k},t_{n})$ and
$\alpha=D_{glut}\frac{\Delta t}{(\Delta x)^{2}}$. For our simulation, we
satisfy a condition of $\alpha<(1/2)^{3}$ known as CFL condition to ensure the
scheme is numerically stableLewy1928 . We take the space step $\Delta
x=0.01\mu m$ and the time step $\Delta t=C\cdot(\Delta x)^{2}$, where $C$ is a
constant.

The glutamate concentration $C(x,y,z,t)$ simulating was averaged over $10\mu
s$ intervals (10 time steps) to reduce computational cost. MATLAB solver
ode23s is effective at crude errors and relatively faster than other
solversShampine1997 .

###  2.3 Measurement for independence

We denoted the relative ratio function as the ratio of the maximum open
probability of distal receptor over the maximum opening probability of the
receptor directly opposing the release site (center or edge). Respectively, we
call a receptor as $R_{C}$, which is directly opposing the evoked release
site, and also represents a receptor located the center of postsynaptic
terminal. A receptor $R_{E}$, directly opposing the spontaneous release site,
is located around the edge of the postsynaptic terminal. The measure(3) we
define is showing the points where ratio functions are taken for comparison:

$Measure:=\frac{\mbox{Max}(P_{open})\ \mbox{at distal
receptor}}{\mbox{Max}(P_{open})\ \mbox{at release site}\ (R_{C}\ \mbox{or}\
R_{E})}.$ (3)

When we simulate our model, we fix the denominator as one receptor either
$R_{C}$ and $R_{E}$. Then we measure Max($P_{open}$) at a receptor along the
two directions as shown in Figure 1C. The Max($P_{open}$) in other locations
will be valued between the values obtained in these two directions. If the
steepness of function for ratio is relatively large, it implies that the
evoked and spontaneous currents have a less chance of having crosstalk. We
used the measure (3) to estimate the independency with considering effective
conditions for independency such as a synaptic size, diffusion inhomogeneity,
and fusion pore. When glutamate vesicle releases over the center receptor in
Figure 1C(left), for the receptors of equal distance to release, the diagonal
direction shows the largest $P_{open}$ and vertical (or horizontal) direction
shows the smallest given the same distance. In this process the Piecewise
Hermite Cubic Interpolation (PHCI) was used to approximate the probable values
to preserve monotone or convex curve of the functionHyman1983 .

##  3 Results

We assumed the situation where evoked glutamate release activates evoked
receptor kinetics at center and spontaneous receptor kinetics at edge as shown
in Figure 2A, 2B. Spontaneous glutamate release at the cleft edge activates
evoked receptor kinetics at center and spontaneous receptor kinetics at edge.

Figure 2: A. Hypothesis that glutamate molecules release over the center at
the cleft (a typical location is directly opposing to the center on the
postsynaptic neuron, called $R_{C}$). B. Hypothesis that glutamate molecules
release over the center at the cleft (a typical location is directly opposing
to the center on the postsynaptic neuron, called $R_{E}$). C. The total number
of molecules inside of a cleft is depicted as a function of time, in an
instantaneous release of 4000 glutamate molecules at $R_{C}$ for glutamate
molecule release at the center. D. The total number of molecules inside of a
cleft is depicted as a function of time, in an instantaneous release of 4000
glutamate molecules at $R_{E}$ for glutamate molecule release at the edge.

First, we simulated the base model with an instantaneous release of 4000
glutamate molecules. Figure 2 shows the total number of molecules within the
cleft decreases according to the time when center release and edge release
occur, respectively. It takes about 0.08ms to clear a half of the molecules
out of the cleft and the population decays in exponential trends with decay
constants $8.808\times 10^{-5}$ and $4.526\times 10^{-4}$ respectively, as
shown Figure 2C, 2D . It implies that an edge release makes glutamate
molecules flushed out much quicker than that of center release.

###  3.1 Effect of size of synapses

We used the model to calculate and compare three sizes of synapses as large
synapse, medium synapse, and small synapse. Then we set areas of synapse as
$600\mbox{nm}\times 600\mbox{nm}$ for large synapse, $400\mbox{nm}\times
400\mbox{nm}$ for medium size, and $200\mbox{nm}\times 200\mbox{nm}$ for small
synapse respectively. We estimated the minimum distance between two sets of
NMDA receptors to have possible less crosstalk. The ratios of maximum opening
probabilities are plotted in Figure 3A and 3B.

Figure 3: Ratios of maximum NMDA receptor opening probabilities($P_{open}$) as
functions of receptor distance for three sizes of synapse as calculated by the
kinetics equation, when glutamates are released above the center location(A)
and the edge location(B), respectively.

Based on the criteria from our previous research Atasoy2008 ; Reese2016 , we
assumed 5-fold(0.5, see in Figure 3) is the threshold ratio as a good
indicator for independent currents. However, it is hard to assure independency
of evoked and spontaneous neurotransmission for medium ($0.16\mu m^{2}$) and
small synapses ($0.04\mu m^{2}$) in our current base model. For medium synapse
($400\mbox{nm}\times 400\mbox{nm}$), if two forms of release locate towards
opposite corners of the synaptic cleft, we might achieve sufficiently low
level of crosstalk and possibly obtain the independence of spontaneous and
evoked neurotransmissions Atasoy2008 . For small synapse $200\mbox{nm}\times
200\mbox{nm}$, there are apparently have more crosstalk between evoked and
spontaneous releases.

###  3.2 Effect of diffusion inhomogeneity in synaptic cleft

We considered an assumption that the background medium for diffusion of
glutamate molecules may be different depending on the location inside the
cleft. We created an interior zone($\Omega^{+}$) and an exterior
zone($\Omega^{-}$) as shown in Figure 4A, 4B. $D_{glut}$ could vary zone to
zone, and it represents the inhomogeneity of material components in the cleft.

Figure 4: A, B. The synaptic cleft space is divided into two zones. The
diffusion coefficients assume two values in two regions($\Omega^{+}$ and
$\Omega^{-}$) respectively, which represent slow and fast motion of
neurotransmitters in different material composition of the cleft space. Front
view of diffusion process with two different zones when center release(A) or
edge release(B) occur. C, D. Ratio of Max($P_{open}$) as a function of
receptor distance for diffusion inhomogeneity(base, high affinity center, and
high affinity edge) when center release(C) or edge release(D) occur.

In Figure 4, among three lines for both center(evoked) and edge(spontaneous)
release, the high affinity center model has the sharper downward slope when
the distance is apart. However, from 0 to 150nm range of distance, all three
lines are virtually constants as shown in Figure 4, and this implies that this
high affinity center is not enough for a small synapse to house the
independent currents from two modes transmissions.

###  3.3 Effect of narrow fusion pores

One possibility is to reduce the amount of glutamate molecules released per
vesicle in the partial release (called kiss-and-go) scenarios, and/or the
glutamate is released in slower release rate due to partial opening of
vesicles in the membrane. When simulating the vesicular diffusion,
$D_{glut}=0.15\mu m^{2}/ms$ and $D_{glut}=0.0375\mu m^{2}/ms$ are taken for
10nm and 2nm fusion pore respectively. As shown in Figure 5, the limited
vesicle fusion rate has impact on reducing the crosstalk of two currents from
center and edge releases than that of instantaneous model (our base model).
They did not impact as much to perturb the independent signaling of synapse on
large synapses, although slower releases did promote more independence. Figure
5E shows that for small synapse (200nm x 200nm) with instantaneous release,
the ratio of maximal opening probability is close to 1 and is consistent up to
90nm when center release occurs, this implies that they have high probability
of having crosstalk of two neurotransmissions. In fact, for small size of
synapse, there is not much difference between the maximal opening probability
at $R_{C}$ and $R_{E}$ when center release or edge release occurs. This
limited vesicle fusion model benefits small size synapses substantially in
achieving independent signaling. This can be verified in the graph of Figure
5E that as glutamate molecules release through 10nm or 2nm vesicle fusion
pore, the ratio of maximal opening probability has the sharper downward slope
when distance goes away. The ratio achieves 10-fold reduction at 90nm
distance, giving plausibility for independent currents from two transmissions.
For edge release, limited vesicle fusion is a redundant factor for activating
two independent fusions since the ratio drastically decreases and becomes
close to zero at 100nm distance for all size of synapses as shown in Figure
5B, 5D, and 5F. This is because when two release sites are located to the two
opposite corners, then most of glutamate molecules flush out of the cleft very
fast and rarely reach to the other site of receptor.

Figure 5:  Ratios of maximum NMDA receptor opening probabilities as functions
of receptor distance for different release speed (slow, 2 nm fusion pore -
$\triangle$, regular, 10 nm fusion pore- $\ast$, and instantaneous -$\circ$)
of glutamate vesicle release. The open probabilities were calculated by the
kinetics equation, when glutamates are released above the center location in
A, C, E and the edge location in (B, E, F) respectively. The sizes of the
synapses are $600\mbox{nm}\times 600\mbox{nm}$ in (A, B), $400\mbox{nm}\times
400\mbox{nm}$ in (C, D), and $200\mbox{nm}\times 200\mbox{nm}$ in (E, F),
respectively.

###  3.4 Simulation results compared with experimantal finding of PSD-95
enrichment

An alignment of presynaptic and postsynaptic nanoscale subdomains, called
nanocolumn was proposed. Evoked fusion occurs in confined areas by protein
gradient with higher local density of Rab3-interacting molecules (RIM) within
the presynaptic active zoneTang2016 . These RIM nanoclusters align with
concentrated postsynaptic receptors. Evoked neurotransmitter release prefer to
occur at sites directly opposing postsynaptic receptor guided by the
nanoachitecture of the active zones. They estimated that majority(72-82%) of
evoked signals arose from single vesicle fusion. The concentration of vesicle
priming proteins in nanoclusters prefers to evoked fusion in the subregion of
the active zone. Three RIM 1/2 nanoclustes and three PSD-95(Post Synaptic
Density) nanoclusters are well aligned for two pairs and not aligned for one
pair. They used two independent approaches to estimate the relationship
between active zone and postsynaptic density(PSD) protein distributions. In
order to figure out of the trend, they measured RIM 1/2 localization densities
as a function of radial distance from the centres of PSD-95 nanoclusters as
translated across the synaptic cleft. Similarly, they estimated PSD-95 protein
enrichment densities as a function of the center of RIM 1/2 nanoclusters.

Figure 6: A. PSD-95 enrichment as a function of distance relative to RIM 1/2
nanoclusters Tang2016 . B. Comparsion with the rescaled data with PSD-95
Enrichment from A(red) and simulation results (center, 200nm & 2nm) with the
measurement (blue, dashed).

Through this results, they defined an enrichment index as the average
molecular density of the opposed protein (n=265-272) within a 60nm radius from
the nanocluster center. They verified from this results if synapses are trans-
synaptically aligned on the nanoscale level, the distribution of protein on
side of the synapse may predict protein density in the opposing neuron. The
experimental findings seemed to be relevant to our mathematical results, so we
compared the measured data with the results from our mathematical measurement
for small synapse(200nm by 200nm). The data Tang2016  was obtained and fitted
with the exponential model using MATLAB. Figure 6A has trends of exponentially
decreasing. The measured data of PSD-95 enrichment on the postsynaptic neuron
as a function of distance relative to the center of RIM 1/2 nanocluster on
presynaptic terminal is approximating to the exponential function as below:

$f_{PSD-95}(x)=a_{1}\cdot\exp(b_{1}\cdot x)$ (4)

The coefficients with 95 % confidence bounds of $a_{1}$ and $b_{1}$ were
estimated to 104 (87.06, 121) and -0.02038 (-0.02518, -0.01558), respectively.

We tested our simulation and suggested that we may obtain the independency of
two currents from evoked and spontaneous neurotransmitter release when if
small synapses have a geometry of a narrow fusion pore (2nm)(Figure 5E, 5F).
The ratio achieves 10-fold reduction at 90nm distance, giving plausibility for
independent currents from two modes of transmissions. This corresponds to the
result of an enrichment index within a 60nm radius from the nanocluster
centre. Thus, we also consider the graph of ratio function of peak opening
probability as a function of distance from the center synapse for small
synapse and evoked narrow fusion pore release in Figure 5E. Then we fit the
graph to the exponential model similarly. The coefficients with 95 %
confidence bounds of $a_{2}$ is 1.044 (0.9322, 1.156) and $b_{2}$ is
approximating to -0.02791 (-0.03409, -0.02171). We obtained an exponential
function for our simulation,

$f_{2nm-sim}(x)=a_{2}\cdot\exp(b_{2}\cdot x).$ (5)

The coefficients of $f_{PSD-95}($4) were rescaled and plotted the rescaled
function of $f_{PSD-95}($4) with the function of $f_{2nm-sim}$(5) together as
shown in Figure 6.

##  4 Conclusion

In this study, we aimed to address the biophysical constraints that may
determine independent activation on N-methyl-D-asparate (NMDA) receptor
mediated currents in response to evoked and spontaneous glutamate molecules
releases though simulating a mathematical model. In this way, we described the
boundaries of different factors, including size and geometry of synaptic
cleft, the neurotransmitter release rate s from presynaptic terminals,
different rates glutamate mobility that permit independent synaptic events in
the limited space of a synapse cleft. The previous research has shown the
evidence that spontaneous and evoked vesicles originate from different pool of
glutamate molecules stores, and after releasing, neurotransmitters activate
non-overlapping postsynaptic NMDA receptors populations for a large
synapseSara2005 ; Atasoy2008 ; Kavalali2011 . However, it is still unclear the
mechanism of evoked and spontaneous neurotransmission that activate on
postsynaptic terminals.Thus, we constructed a mathematical model of two modes
of neurotransmission, and simulated spontaneous and evoked release process
with different factors including a size, geometry of synaptic cleft, different
glutamate mobility rate in the cleft, and the release rate of
neurotransmitters from presynaptic terminals.

###  4.1 Discussion

The geometry of synapse, especially for a small size, has not been described
clearly with the limitation of microscopy. For large synapses, experimental
evidence supports the hypothesis that spontaneous and evoked currents are
resulted from glutamate vesicle releases in different pools and after
releasing, neurotransmitters activate non-overlapping postsynaptic NMDA
receptors populations Reese2016 ; Atasoy2008 . Based on this hypothesis, the
maximal open probability is most sensitive to the distance from the release
sites. Thus, the independence of currents from two modes of releases mainly
resulted from a large ratio of peak open probability at evoked receptors(maybe
occurred at the center) or spontaneous receptor(maybe occurred around the
edge) over the different release sites($R_{C}$ or $R_{E}$). This is a
measurement for independence, when we tested the model on a few conditions
such as glutamate release site/receptor distance change or glutamate release
speed change, as well as other changes in geometry, diffusion inhomogeneity.
Based on the criteria from our previous research Atasoy2008 ; Reese2016 , we
assumed 5-fold(0.5, see in Figure 3) is the threshold ratio as a good
indicator for independent currents. However, it is hard to assure independency
of evoked and spontaneous neurotransmission for medium ($0.16\mu m^{2}$) and
small synapses ($0.04\mu m^{2}$) in our current base model. For medium synapse
($400\mbox{nm}\times 400\mbox{nm}$), if two forms of release locate towards
opposite corners of the synaptic cleft, we might achieve sufficient low level
of crosstalk and possibly obtain the independence of spontaneous and evoked
neurotransmissions Atasoy2008 . For small synapse $200\mbox{nm}\times
200\mbox{nm}$, there are apparently have more crosstalk between evoked and
spontaneous releases.

We propose the two possible scenarios to find favorable conditions where the
signals from two neurotransmissions could be independent on the postsynaptic
neurons. First, it is to assume the components in the cleft to be different
compositions. We constructed the model with the center area to be less
diffusive for glutamate molecules than the peripheral region inside cleft. In
Figure 4, among three lines for both center(evoked) and edge(spontaneous)
release, the high affinity center model has the sharper downward slope when
the distance is apart. This is compatible with the recent experimental results
that the evoked release is guided by the protein gradient and prefer to occur
in confined area with in high local density of Rab3-interacting
molecule(RIM)Tang2016  in center area. However, from 0 to 150nm range of
distance, all three lines are virtually constants as shown in Figure 4A, and
this implies that this high affinity center is not enough for a small synapse
to house the independent currents from two modes transmissions. The second
scenario is to reduce the amount of glutamate molecules released per vesicle
in the partial release (called kiss-and-go) scenarios, and/or the glutamate is
released in slower release rate due to partial opening of vesicles in the
membrane. The instantaneous release of 4000 glutamate molecules is an
approximation for the actual situation. The release of glutamate from vesicles
in presynaptic terminals is a complex process that includes elevation of Ca2+,
binding of SNARE protein to the membrane and a sequence of events of
biochemical reactionsSudhof2011 . As shown in Figure 5, the limited vesicle
fusion rate has less impact on the crosstalk of two currents from center and
edge releases than that of instantaneous model (our base model). They did not
impact as much to perturb the independent signaling of synapse on large
synapses, although slower releases did promote more independence. Slow release
model with narrow fusion model is fitted with a small size of synapse and
substantially more in achieving independent signaling. Figure 5E and 5F show
that small synapse (200nm by 200nm) might not have independent signaling when
evoked glutamate release occurs instantaneously because evoked and spontaneous
release sites are not far away from each other and thus they have high
probability of having crosstalk between each other. In fact, there is not much
difference between the peak opening probabilities at a receptor for evoked
release and the one for spontaneous release. This could be verified in the
graph of Figure 5E, the ratio of two maximum open probabilities is close to 1,
and the peak open probability in NMDA receptors is consistent up to 90nm far
from the receptor molecules opposing evoked release site. However for small
synapses, as glutamate release through 10nm and 2nm vesicle fusion pore, the
open probability ratio decreases more drastically and becomes close to zero,
and in 2 nm pore, the ratio achieves 10-fold reduction at 90nm distance,
giving plausibility for independent currents from two transmissions. Thus, we
suggested that we may obtain the independency of two currents from evoked and
spontaneous neurotransmitter release when if small synapses have a geometry of
a narrow fusion pore (2nm)(Figure 5F). Simulation results showed that the
ratio achieves 10-fold reduction at 90nm distance, giving plausibility for
independent currents from two modes of transmissions. This corresponds to the
result of an enrichment index within a 60nm radius from the nanocluster
centreTang2016 . We then focused on the graph the a ratio function of peak
opening probability as a function of distance from the center synapse for
small synapse and evoked narrow fusion pore release in Figure 5E. The
coefficients of $f_{PSD-95}$(4) were rescaled and plotted with the function of
$f_{2nm-sim}$(5) together(Figure 6). The results are well-matched to each
other, and this indicates that small synapses might be conducted dynamic
functional modules and possibly hold the segregation of sites for spontaneous
versus evoked neurotransmission within individual synapses. Tang and colleages
found how distribution of presynaptic vesicle sites corresponds to the
receptors in the postsynaptic neuron. They supported that action-potential-
evoked fusion is guided by scaffolding proteins, called nanocolumn, which were
likely aligned near the centre of synapses than near the edge. Also this
nanocolumn theory proposed that the active zone for evoked vesicle fusion
occurs at sites directly opposing postsynaptic receptorsTang2016 . These
findings indicate that the segregation of sites for spontaneous and evoked
neurotransmission with nanoscale subdomains connecting presynaptic and
postsynaptic terminals. Therefore, it is possible even for small synapses to
possess this neural functional dynamics.

This study had several limitations. First, the diffusion process may not
regard inhomogeneous structure in the cleft in the current study, and we may
consider the diffusion process in the cleft by adding an advection term in our
future study. Secondly, the small number of glutamate molecules in the vesicle
challenged in implementing mathematical models of the release process. In
addition, the scale of neurotransmission is hundreds nanometer ranges, and it
is laid between classical mechanics and quantum mechanics.

In conclusion, we developed a mathematical model to analyze those results for
independent signaling of spontaneous and evoked glutamate releases in a single
synapse, comparing with the experimental and theoretical prediction. We
defined a measurement of independency and set a criterion of a 5 fold ratio as
a reasonable boundary for the independence. From those results we suggested
two possibilities for small synapses to be less crosstalk from spontaneous and
evoked neurotransmitter currents on postsynaptic terminals. It validated
through comparisons with experimental findings. The results were well-matched
to each other and this implies that small synapses might be conducted of
dynamic functional modules and possibly hold the segregation of sites for
spontaneous versus evoked neurotransmission within individual synapses. These
results show our mathematical modeling may refer to the neuroscience
questioning of distribution and separation of NMDA receptors for two
independent spontaneous and evoked release process.

Acknowledgments We would like to thank Prof. Ege Kavalali for introducing this
problem to us and for his advice and discussion in neuroscience. This research
was partially supported by the National Research Foundation of Korea(NRF)
grant funded by the Korea government, MSIP; Ministry of Science, ICT & Future
Planning (2020R1G1A1A0100746911).

## References

  * (1) Anson LC, Chen PE, Wyllie DJ, Colquhoun D, Schoepfer R (1998) Identification of amino acid residues of the NR2A subunit that control glutamate potency in recombinant NR1/NR2A NMDA receptors. J Neurosci 18(2):581-589.
  * (2) Atasoy D, Ertunc M, Moulder KL, Blackwell J, Chung C, Su J, Kavalali ET (2008) Spontaneous and evoked glutamate release activates two populations of NMDA receptors with limited overlap. J Neurosci 28(40):10151-10166.
  * (3) Bauerfeind R, Huttner WB, Almers W, Augustin GJ (1994) Quantal neurotransmitter release from early endosomes? Trends Cell Biol. 4:155-156.
  * (4) Clements JD (1996) Transmitter timecourse in the synaptic cleft: its role in central synaptic function. Trends Neurosci. 19(96):163-171.
  * (5) Collingridge GL, Volianskis A, Bannister N, France G, Hanna L, Mercier M, Tidball P, Fang G, Irvine MW, Costa BM, Monaghan DT, Bortolotto ZA, Molnár E, Lodge D, Jane DE (2013) The NMDA receptor as a target for cognitive enhancement. Neuropharmacology. 64:13-26. doi: 10.1016/j.neuropharm.2012.06.051.
  * (6) Diamond JS, Jahr CE (1997) Transporters buffer synaptically released glutamate on a submillisecond time scale. J Neurosci 17(12):4672-4687.
  * (7) Frerking M, Wilson M (1995) Saturation of postsynaptic receptors at central synapses? Curr. Opin 6:395-403.
  * (8) Hill SM (1998) Receptor crosstalk: communication through cell signaling pathways. The Anatomical record 253(2)42-48.
  * (9) Hyman JM (1983) Accurate monotonicity preserving cubic interpolation. SIAM J. Sci. Comput.4:645-654.
  * (10) Kavalali ET, Chung C, Khvotchev M, Leitz J, Nosyreva E, Raingo J, Ramirez D (2011) ASpontaneous neurotransmission: an independent pathway for neuronal signaling? Physiology 26(1):45-53.
  * (11) Kavalali ET (2014) The mechanisms and functions of spontaneous neurotransmitter release. Nat. Rev. Neurosci. 16(1):5-16.
  * (12) Kenny A, Cary NRB, Murphy D, Shapiro LM (1994) Intraoperative epicardial echocardiography with a miniature high-frequency transducer: imaging techniques and Scanning Planes. American Society of Echocardiography. 7(2):141-149.
  * (13) Kumar A (2015) NMDA Receptor Function During Senescence: Implication on Cognitive Performance. Front. Neurosci. 9:473. doi: 10.3389/fnins.2015.00473.
  * (14) LeDoux J (2003) Synaptic Self: How Our Brains Become Who We Are. Penguin Publishing Group.
  * (15) Lewy H, Friedrichs K, Courant R (1928) Uber die partiellen Differenzengleichungen der mathematischen Physik. Mathematische Annalen 100:32-74.
  * (16) Melom JE, Akbergenova Y, Gavornik JP, Littleton JT (2013) Spontaneous and evoked release are independently regulated at individual active zones. J Neurosc 22(44):17253-17263.
  * (17) Newcomer JW, Farber NB, Olney JW (2000) NMDA receptor function, memory, and brain aging. Dialogues Clin Neurosci. 2(3):219-32. doi: 10.31887/DCNS.2000.2.3/jnewcomer.
  * (18) Nielsen TA, DiGregorio DA, Silver RA (2004) Modulation of glutamate mobility reveals the mechanism underlying slow-rising AMPAR EPSCs and the diffusion coefficient in the synaptic cleft. Neuron 42(5):757-771.
  * (19) Patneau DK, Mayer ML (1990) Structure-activity relationships for amino acid transmitter candidates acting at N-methyl-D-aspartate and quisqualate receptors. J Neurosc 10(7):2385-2399.
  * (20) Peled ES, Newman ZL, Isacoff EY (2014) Evoked and spontaneous transmission favored by distinct sets of synapses. Curr. Biol. 24(5):484-493.
  * (21) Popescu G, Robert A, Howe JR, Auerbach A (2004) EReaction mechanism determines NMDA receptor response to repetitive stimulation. Nature 430(7001):790-793.
  * (22) Reese AL, Kavalali ET (2016) Single synapse evaluation of the postsynaptic NMDA receptors targeted by evoked and spontaneous neurotransmission. eLife 5:e21170.
  * (23) Sara Y, Virmani T, Deak F, Liu X, Kavalali ET (2005) An isolated pool of vesicles recycles at rest and drives spontaneous neurotransmission. Neuron 45(4):563-573.
  * (24) Schneggenburger R, Rosenmund C (2015) Molecular mechanisms governing $Ca^{2+}$ regulation of evoked and spontaneous release. Nat. Neurosci 18(7):935-941.
  * (25) Schuz A, Palm G (1989) Density of neurons and synapses in the cerebral cortex of the mouse 286(4):442-55.
  * (26) Seo SB (2017) Evoked and spontaneous neurotransmitter releases for independent synaptic currents: mathematical modeling and analysis(Unpublished doctoral dissertation). University of Texas at Arlington, Texas, USA.
  * (27) Seo SB (2019) A note on numerical approaches for heat-diffusion equation with heterogeneous media and its applications. East Asian Math. J. 35(1):099-108. doi: http://dx.doi.org/10.7858/eamj.2019.013.
  * (28) Shampine LF, Reichelt MW (1997) The MATLAB ODE Suite. SIAM J Sci Comput 18(1):1-22.
  * (29) Sudhof TC, Rizo J (2011) Synaptic vesicle exocytosis. Cold Spring Harb. Perspect. Biol. 3(12):a005637.
  * (30) Sutton MA, Wall NR, Aakalu GN, Schuman EM (2004) Regulation of dendritic protein synthesis by miniature synaptic events. Science 304(5679):1979-1983.
  * (31) Tang AH, Chen H, Li TP, Metzbower SR, MacGillavry HD, Blanpied TA (2016) A trans-synaptic nanocolumn aligns neurotransmitter release to receptors. Nature 536(7615):210-4. doi: 10.1038/nature19058.
  * (32) Chung ST, Morris RL (2008) Vesicle association and exocytosis at ribbon and extraribbon sites in retinal bipolar cell presynaptic terminals. PNAS 105(12):4922-2927.


# MLM: A Benchmark Dataset for Multitask Learning with Multiple Languages and
Modalities

Jason Armitage
University of Bonn
Germany Denotes equal contribution to this research Endri
Kacupaj11footnotemark: 1
University of Bonn
Germany  Golsa Tahmasebzadeh11footnotemark: 1
TIB – Leibniz Information
Center for Science and
Technology
Germany  Swati
Jožef Stefan Institute
Slovenia  Maria Maleshkova
University of Bonn
Germany  Ralph Ewerth
TIB – Leibniz Information
Center for Science and
Technology
Germany  Jens Lehmann
University of Bonn
Fraunhofer IAIS
Germany

###### Abstract

In this paper, we introduce the MLM (Multiple Languages and Modalities)
dataset - a new resource to train and evaluate multitask systems on samples in
multiple modalities and three languages. The generation process and inclusion
of semantic data provide a resource that further tests the ability for
multitask systems to learn relationships between entities. The dataset is
designed for researchers and developers who build applications that perform
multiple tasks on data encountered on the web and in digital archives. A
second version of MLM provides a geo-representative subset of the data with
weighted samples for countries of the European Union. We demonstrate the value
of the resource in developing novel applications in the digital humanities
with a motivating use case and specify a benchmark set of tasks to retrieve
modalities and locate entities in the dataset. Evaluation of baseline
multitask and single task systems on the full and geo-representative versions
of MLM demonstrate the challenges of generalising on diverse data. In addition
to the digital humanities, we expect the resource to contribute to research in
multimodal representation learning, location estimation, and scene
understanding.

## 1 Introduction

At the core of recent research in machine learning is the proposal that
systems trained on large sets of data will display forms of the human ability
to generalise. Learning algorithms will enable systems to infer abstract
representations from raw data and tackle a wide range of tasks [7]. The web
provides an abundance of data for training models that enable functions in
application pipelines. Data availability then is no impediment to the aims of
machine learning in relation to these applications - but there remain two
important ways in which systems fail to generalise.

The first of these is the inability for systems to cope with diversity in
real-world data [6, 53]. Even small changes in the distributions of data lead
to steep declines in the performance of machine learning models. Information
on the web and maintained in archives is present in a diverse range of
modalities including language, images, and numerical data. Variety is also
observed in each modality taken in isolation. Consider text in documents
stored and studied by cultural and academic institutions. A system that is
designed to predict contextual information - in this case, locations
referenced in the documents [1] \- may encounter text written in several
languages and writing systems. Additional inputs with relevant information can
be stored in databases or as structured text in a knowledge base. Machine
learning systems that are robust to disparities between and within modalities
can make use of a higher proportion of the information present across a range
of materials [5].

The second way in which current systems fail to generalise is in using
representations learned during training to perform new tasks. Analysis of
machine learning in application workflows reveals a host of disparate models -
each trained on homogeneous data to conduct specialised tasks [41]. In this
approach, learning is siloed with the result that computation and data
required for training and inference multiply - and there is limited sharing of
parameters and representations across pipelines. Systems that conduct multiple
tasks hold the promise of simplifying application architectures and improving
efficiency in the use of resources. In the case of researchers conducting
cultural and historical analysis, this also presents the advantages of
studying or accessing materials in multiple formats [40] such as textual
documents and images that reference the same subject.

The Multiple Languages and Modalities (MLM) dataset consists of text in three
languages (English, French, and German), images, location data, and knowledge
graph triple classes (ie component entities of triples for the sample entity
that are stored in Wikidata). The resource is designed to evaluate multitask
systems in relation to single task alternatives when trained simultaneously on
sequences of tasks - in this case, a system that performs both cross-modal
retrieval and location estimation. Multitask learning aims to benefit from
commonalities in input data [13] and MLM is derived from knowledge graph
entities that connect all of the samples in the data. Relationships between
entities are further manifested in triple classes.

### 1.1 Contributions

We identify four contributions resulting from this research.

  * •

The first contribution is the generation process, which provides a consistent
and reproducible approach to build a resource of diverse data with semantic
relationships. As detailed below, this process will accelerate the release of
future versions of MLM.

  * •

The second contribution is the release of a novel dataset with a range of
modalities and languages that - to our knowledge - is unique in the research
literature. The resource tests the ability for multitask systems to exploit
both formal and indirect relationships between samples.

  * •

The third contribution is the specification of a benchmark series of tasks to
evaluate multitask systems that characterise applications used and built by
researchers and developers in the digital humanities. The resource presents a
unique option to this community to develop novel applications that make use of
multimodal frameworks for the assessment of cultural and historical phenomena
[40].

  * •

The fourth contribution is building a multitask system that combines several
methods to perform in concurrence cross-modal retrieval and location
estimation on highly diverse data.

### 1.2 Use Case

We present a use case that applies to the benchmark evaluation specified in
this research and characterises the novel applications enabled by a multitask
approach to diverse data in a real-world scenario. The MLM resource and
benchmark tasks also extends to use cases in several other areas highlighted
in Section 2.

Relations between media and geospatial location underpin applications and
projects developed for cultural and historical analysis. The Edinburgh
Geoparser [1] and Perdido [37] are recent instances of applications that
retrieve locations from multilingual literary texts sourced in books and
journals. Identifying locations depicted in visual materials is also a core
function in applications that georeference historical images on landscapes and
urban areas [9, 20]. In the multitask system framework presented below (see
Figure 3), we enable geoparsing and georeferencing in two novel ways. First
the researcher has the option of starting a query either with a text or an
image. During the retrieval task (see Figure 1), related materials in the
missing modality are returned. This creates the opportunity to conduct a
multimodal analysis on the entity of interest [40]. Second a multitask
framework returns a predicted location for both inputs.

Figure 1: Cross-modal retrieval and location estimation performed by the
Multitask IR+LE framework.

## 2 Impact

In this section, we highlight research areas where the MLM resource enables
innovation and assess the benefits of the dataset in relation to existing
resources for evaluating multitask systems.

We have outlined above new approaches in the digital humanities that are
enabled by the dataset and benchmark evaluation tasks. Multitask learning
systems that learn on multimodal data are also an active area of research in
relation to multimodal representation learning, location estimation, and scene
understanding [5, 28]. MLM is further designed to evaluate the ability for
multitask systems to leverage relationships between constituent entities in
data and knowledge graph properties used in the generation process. Multitask
learning systems that exploit these relationships as a signal are positioned
to deliver additional benefits to applications that rely on semantic data and
knowledge graphs, which include recommender systems, mobile information
retrieval, and bioinformatics [2].

Resources for training and evaluating multitask learning systems are
characterised by homogeneity both in modalities and languages. The Office-Home
and Computer Survey are examples of resources to train text-only systems [55].
Similarly multitask systems for computer vision turn to benchmark datasets
where images are the primary modality [12, 32]. Multimodal datasets employed
for multitask learning consist predominantly of two modalities [30, 16] and a
single language. Although ImageCLEF runs challenges (see Table 1) with textual
samples in multiple languages, these instances also focus on bimodal learning
[49]. The Multiple Language and Modalities dataset, which we propose,
approximates the diversity of online data and those stored in digital archives
or databases in presenting samples in three modalities with multilingual text
and semantic data. To our knowledge, this resource is unique in presenting an
opportunity to train multitask systems on visual, textual, spatial and
semantic web inputs for multimodal, and multilingual learning.

Geographical location is central to both the resource and task presented in
this research. We are motivated by on-going investigations on representation
bias in machine learning datasets to present a second geo-representative
version of our resource. Representation bias has been found to penalise
minority classes in classification tasks [45]. MLM-irle-gr is a subset of the
full data with balanced sample sizes of entities for the 28 countries in the
European Union. This enables governmental and commercial organisations that
provide applications and services in this region to minimise the impact of
representation bias resulting from training data when evaluating multitask
learning systems.

Table 1: Multimodal and Multilingual Datasets Resource | Data Types | Language | Samples
---|---|---|---
MLM |  Text, Images, Geocoordinates Triple Classes | DE, EN, FR | 236k
ImageCLEF | Text, Images | DE, EN, FR | 237k
IKEA | Text, Images | DE, EN, FR | 4k

## 3 MLM - Multiple Languages and Modalities

In this research we propose a benchmark to evaluate the strengths of multitask
systems to generalise on diverse data. This section starts by defining the
tasks constituting the evaluation, continues with an introduction of the
dataset, and concludes with details on the generation process.

### 3.1 Benchmark Evaluation

The benchmark evaluation is composed of two tasks that are performed in
conjunction to retrieve and locate entities represented by multimodal and
multilingual inputs. The first task consists of cross-modal retrieval where a
representation in an input modality $x_{i}$ returns the target $y_{i}$ in a
corresponding modality. In this case, the inputs are one from a pair
$P=\\{(x^{u}_{i},x^{t}_{i})\\}^{n}_{i}=1$, where $x^{u}_{i}$ is an image and
$x^{t}_{i}$ is a combination of structured $x^{r}_{i}$ and unstructured
$x^{s}_{i}$ textual inputs. The objective is to return the corresponding
target $y_{i}$ at the lowest position in the ranking of returned predictions.
The objective of the second task is to use the same pair of inputs
$P=\\{(x^{u}_{i},x^{t}_{i})\\}^{n}_{i}=1$ to perform a classification on a set
of geo-cells $y=(y_{1},y_{2},...y_{n})$ \- each of which groups a set of
entities based on location given as geocoordinates. The final objective of the
evaluation is to return results that surpass the baseline system presented
below on both tasks. Additional details of the constituent tasks are included
in Section 5.1.

### 3.2 Dataset

The Multiple Languages and Modalities comprises data points on 236k human
settlements for evaluating and optimising multitask learning systems. MLM
presents a dataset with a high level of diversity in terms of modality and
language. For each entity, we have extracted text summaries, images,
coordinates, and their respective triple classes. Text summaries are available
in three languages (English, French, and German) with each entity having
between one and three language entries.

Human settlements from all continents are provided in the full dataset with
72% located in Europe (see Figure 2). MLM-irle is a version of the dataset
generated for the benchmark evaluation tasks and features geo-cells required
for location estimation. To serve organisations that focus on the European
Union, we have also created a version of the dataset - MLM-irle-gr (ie geo-
representative) - that provides a geographically balanced coverage of human
settlements in this region. MLM-irle-gr contains data on 24k human settlements
across the EU weighted in relation to the population
count111https://population.un.org/wpp/Download/Standard/Population/ for each
of the 28 countries.

Table 2: MLM - Dataset Details Num. of | MLM | MLM-irle | MLM-irle-gr
---|---|---|---
Entities | 236496 | 218681 | 22501
Images | 412422 | 314533 | 31621
Summaries | 497899 | 462328 | 47508
Triple classes | 1685 | 1655 | 452

[step=40,max=200,width=0.3scale=0.8] [label=Europe]171 [label=North America]40
[label=Asia]13 [label=Rest of World]12

Figure 2: MLM: Number of Entities by Continent (000).

### 3.3 Generation Process

In this first release, a core aim of the research was to develop a generation
process that enables fast and consistent scaling for future versions. The
process outlined also results from a core requirement for the resource: to
evaluate the ability of multitask systems to incorporate information on
entities with semantic relations. In this section, we detail stages in the
framework for dataset generation. Our selection of the Wikidata Knowledge
Graph222https://www.wikidata.org/ as the primary source was conditioned on the
requirement for data with semantic relations between entities and the ability
enabled by SPARQL to target data that met our research aims.

Figure 3: MLM dataset generation process.

#### 3.3.1 Define base entity type.

The first step consisted of defining the base entity type that best supported
developing applications that make use of geographical location. We selected
the Wikidata entity human settlement (ID Q486972)333IDs are Wikidata
identifiers https://www.wikidata.org/wiki/Wikidata:Identifiers based on the
number of entities returned relative to other candidate base entity types. In
the second step, instances of human settlement base type with targeted
modalities were extracted resulting in over half a million entities. We
applied constraints to extract entities with a Wikipedia article in at least
one target language (EN, DE, FR), a minimum of one image, and a set of
geocoordinates resulting in 40k human settlements.

#### 3.3.2 Generate a list of classes.

We increased the scope and granularity of human settlements by extracting
entity types that are subclass of the the base type. Entities are returned for
triples upto three hops from the starting entity. An example of additions is
the commune of Italy (ID Q747074) - a subclass of the entity municipality (ID
Q15284), which is a subclass of the human settlement base type. This stage
resulted in a set of 910 unique classes where all entities derive from the
base type by either one, two, or three hops.

#### 3.3.3 Extract entities.

In the extraction stage, we run queries on all entities that are an instance
of at least one of the 910 unique classes in the generated list. This returned
a set of 2.5M entities. A final application of the constraints related to
targeted data types generated a collection of data that constituted 240k
entities.

#### 3.3.4 Download images and summaries.

The final stage consists of downloading images and text summaries for the
extracted entities. While querying Wikidata we were able to also extract
WikiMedia Commons444https://commons.wikimedia.org/wiki/Main_Page links of
images and the corresponding Wikipedia link for each entity. Text summaries
were downloaded using the MediaWiki
API555https://www.mediawiki.org/wiki/API:Main_page, while images using the
WikiMedia link. The process was accelerated by chunking the data and running
multiple sessions. Download errors led to a small loss of entities and the
final raw dataset is in Table 2.

#### 3.3.5 Geographical cell generation.

In this stage, entities along with their images and summaries are assigned to
geographical cells. To this end, we first exclude all SVG images since they
mostly contain maps and signs which are non-descriptive for our tasks. Next,
we divide the earth into non-overlapping geographical cells $C$ using the S2
Geometry Library666https://code.google.com/archive/p/s2-geometry-library/. In
this library, initial cells are generated by projecting the earth’s surface on
a cube with six sides. Based on geocoordinates of the entities, a hierarchical
subdivision is applied [52], where each cell is the node of a quad-tree.
Starting at the root nodes, the respective quad-tree is subdivided recursively
until all cells contain a maximum of $T_{max}=500$ entities. Finally, in order
to have more uniform cells relative to the entities, all generated cells
containing less than $T_{min}=20$ entities are removed.

## 4 Availability

Versions of MLM listed in Table 2 are available for direct download and use.
To support findability and sustainability, the MLM dataset is published as an
on-line resource at  https://doi.org/10.5281/zenodo.3885753. A separate page
with detailed explanations and illustrations is available at
http://cleopatra.ijs.si/goal-mlm/ to promote ease-of-use. The project GitHub
repository contains the complete source code for the system and generation
script is available at  https:/github.com/GOALCLEOPATRA/MLM. Documentation
adheres to the standards of FAIR Data
principles777http://www.nature.com/articles/sdata201618/ with all relevant
metadata specified to the research community and users. It is freely
accessible under the Creative Commons Attribution 4.0 International license,
making it reusable for almost any purpose.

### 4.1 Updating and Reusability

MLM is supported by a team of researchers from the University of Bonn, the
Leibniz Information Center for Science and Technology, and Jožef Stefan
Institute. The resource is already in use for individual projects and as a
contribution to the project deliverables of the Marie Skłodowska-Curie
CLEOPATRA Innovative Training Network. In addition to the steps above that
make the resource available to the wider community, usage of MLM will be
promoted to the network of researchers in this project. Awareness among
researchers and practitioners in digital humanities will be promoted by
demonstrations and presentations at domain-related events. The range of
modalities and languages present in the dataset also extends its application
to research on multimodal representation learning, multilingual machine
learning, information retrieval, location estimation, and the Semantic Web.
MLM will be supported and maintained for three years in the first instance. A
second release of the dataset is already scheduled and the generation process
outlined above is designed to enable rapid scaling.

## 5 Multitask Learning Framework

In this section, we specify a multitask learning framework that characterises
the use case above by performing cross-modal retrieval and location estimation
tasks. The section starts with formulations of the tasks evaluated on MLM and
concludes with details of the system and methods.

Figure 4: Multitask IR+LE framework.

### 5.1 Tasks

#### 5.1.1 Cross-modal Retrieval.

We build on the approach proposed by Marin et al [35] to perform cross-modal
information retrieval on visual $u$ and textual $t$ inputs. Embeddings $v^{u}$
and $v^{t}$ are learned as described below. A mapping from the visual and
textual embeddings to a common space is learned using a cosine embedding loss,
which minimises the distance between $v^{u}$ and $v^{t}$ based on cosine
similarity and labels $s=\\{1,-1\\}$

$L_{cos}(v^{u},v^{t},l)=\left.\begin{cases}1-cos(v^{u}_{1},v^{t}_{1}),&\text{if
}s=1\\\
max(0,cos(v^{u}_{1},v^{t}_{1})-margin),&\text{if}s=-1\end{cases}\right\\}$ (1)

where $v^{u}$ and $v^{t}$ are representations of the visual and textual inputs
respectively. At inference time, an image or textual sample is presented with
the aim of returning samples for the corresponding modality.

#### 5.1.2 Location estimation.

In this architecture, location estimation is regarded as a classification
problem. To this end, geo-cells (described in Section 3.3.5) are regarded as
classes. To formulate the location estimation task, we have $q$ geo-cells
$y=(y_{1},y_{2},...y_{n})$ for each of which there are $n$ number of entities
represented by images, summaries and triples. Each entity is labeled with its
corresponding GPS coordinates in (latitude, longitude) pairs. The task is to
predict $y$ given $v^{u}$ and $v^{t}$ based on a cross-entropy loss:

$L(v,y)=-v[y]+log\left(\sum_{j}exp(v[j])\right)$ (2)

#### 5.1.3 Multitask.

The multitask learning framework (see Figure 4) trains a combined pipeline to
perform cross-modal retrieval and location estimation in conjunction. The
system learns over the cross-entropy and cosine embedding losses with a
weighted average. In order to account for the difference in magnitude between
losses, log variance is calculated over the tasks and converted to log
standard deviation $log(\sigma)$. This weight is applied to the losses before
computing the mean $\mu$ over the products.

### 5.2 System

We apply state-of-the-art embedding methods in computer vision and NLP to
generate representations of the inputs. Two types of descriptors are utilized
to better represent location information of the images when generating Visual
embeddings $v^{u}$. One of the descriptors is taken from a ResNet [21] model
pre-trained on the Places365 dataset [57], where the task is to recognize 365
distinct places - examples include beach, stadium, street. The second
descriptor is taken from the model [38] based on ResNet101 [21] aimed at
predicting the geolocation information of an image. This model is pretrained
on a subset of the Yahoo Flickr Creative Commons 100 Million dataset
(YFCC100M) [47]. The subset includes around five million geo-tagged images
with geolocation labels, and was introduced for the MediaEval Placing Task
2016 (MP-16) [29]. Each descriptor results in a feature vector of size 2048,
so eventually we concatenate both vectors and get a vector of size 4096 for
each image. To generate textual embeddings $v^{t}$ for summaries and triples
we use BERT [19] to extract word vectors for all sentences in text. Since text
content of news articles is long, we select maximum 500 characters at a time
and extract word vectors from BERT, and average the vectors obtained from each
token of the sentence.

An integrated architecture (see Figure 3) takes the visual and text embeddings
to perform the two tasks in concurrence. Visual embeddings $v^{u}$ are fed
into a 1D convolutional layer (conv1d) with ReLU activation $max(0,x)$ \- and
a linear layer with hyperbolic tangent $tanh(u_{i}^{l})$. An LSTM layer
replaces the conv1d for text summary $v^{s}$ and triple class $v^{r}$
embeddings. Regularisation consists of dropout for textual embeddings and norm
with a clamp $min=1*10^{-12}$ for all modalities. Linear layers with dropout
receive the outputs of the norm in location estimation. In both tasks, text
summary and triple classes are fused with concatenation $v^{t}=[v^{s},v^{r}]$.

## 6 Experiments

Experiments assess the performance of the multitask learning system in
relation to single task pipelines. We align with benchmark cross-modal
retrieval tasks [35] in selecting median rank (medR) as the primary metric.
Median rank also takes into account both the performance of the retrieval and
the position of the retrieved instance. To compute medR, a subset of 500
instances are randomly sampled from the test set. Then the experiments are
repeated 10 times and report the mean results. Next, for each input image or
text, items in the corresponding modality are ranked. The objective is to
retrieve the sample for the target modality at the highest rank. The retrieved
lists are further assessed using recall percentage at top K (R@K). This
measures the percentage of queries where the matching item is ranked among the
top K returned samples. The formulation of location estimation as a multiclass
classification task motivates the selection of Precision (P) (Eq.3a), Recall
(R) (Eq.3b) and weighted F1 score (Eq.3c) as measures of performance for the
second task.

$\displaystyle Precision$ $\displaystyle=\frac{TP}{TP+FP}$ (3a) $\displaystyle
Recall$ $\displaystyle=\frac{TP}{TP+FN}$ (3b) $\displaystyle F_{1}^{weighted}$
$\displaystyle=\frac{1}{Q^{2}}\sum_{j=1}^{Q}Q_{j}\frac{2\leavevmode\nobreak\
\times P_{j}\times R_{j}}{\leavevmode\nobreak\ P_{j}+R_{j}}$ (3c)

In the above definitions, $TP$, $FP$ and $FN$ stand for True Positive, False
Positive and False Negative respectively and $Q$ is the number of geo-cells,
and $j$ is the $jth$ instance of geo-cells.

### 6.1 System Configuration and Training

Pipelines for single task and multitask learning are trained for 100 epochs on
a single GeForce GTX 1080 Ti GPU, using training/validation sets of 175k/22k
and 18k/2k for MLM-irle and MLM-irle-gr respectively. Full details on the
dataset splits are available at  http://cleopatra.ijs.si/goal-mlm/. Although
we trained for a number of epochs adopted for comparable systems [35], we note
that validation loss was still declining for both multitask and single task
systems at the end of training. For all tasks, consistent hyperparameter
settings are maintained to enable comparative analysis of the results. The
batch size is set at 128 with a learning rate of 0.0001 and a probability of
0.1 in dropout layers.

Table 3: Cross-modal retrieval results for MLM |  | Image to Text | Text to Image
---|---|---|---
|  | medR | R@1 | R@5 | R@10 | medR | R@1 | R@5 | R@10
MLM-irle | Single task | 43.1 | 0.02 | 0.07 | 0.14 | 32.5 | 0.04 | 0.14 | 0.24
Multitask | 19.9 | 0.05 | 0.20 | 0.34 | 11.9 | 0.11 | 0.34 | 0.47
MLM-irle-gr | Single task | 53.6 | 0.01 | 0.06 | 0.13 | 43.2 | 0.03 | 0.10 | 0.18
Multitask | 27.9 | 0.03 | 0.14 | 0.25 | 21.8 | 0.07 | 0.23 | 0.35
Table 4: Location estimation results for MLM

|  | Image-based | Text-based
---|---|---|---
|  | F1 | P | R | F1 | P | R
MLM- | Single task | 0.16 | 0.17 | 0.17 | 0.59 | 0.61 | 0.60
irle | Multitask | 0.15 | 0.17 | 0.16 | 0.58 | 0.61 | 0.59
MLM- | Single task | 0.28 | 0.30 | 0.28 | 0.67 | 0.70 | 0.68
irle-gr | Multitask | 0.25 | 0.27 | 0.26 | 0.67 | 0.69 | 0.68

### 6.2 Results

In this section evaluation results are discussed for multitask and single task
pipelines on the MLM-irle and MLM-irle-gr (ie geo-representative) datasets
containing 22k and 2k test instances respectively.

#### 6.2.1 Cross-modal Retrieval

In Table 3, we report medR and Recall@K for image to text (first four columns)
and text to image retrieval (the remaining columns). The multitask system
attained a strong improvement on the single task equivalent in the first of
these with medR at 19.9. Qualitative checks on samples consisted of extracting
a random sample of 500 entities from the test set. Two examples of triple
classes where the multitask system scored strongly relative to the single
equivalent are “town of the United States” and “Ortsteil” (see Figure 1). This
corresponds to increases of 13% on R@5 and 20% points on R@10. Although R@1
also showed a marked increase, scores for both multitask and single task are
at low levels. The multitask system also outperformed strongly on the geo-
representative dataset with a 26-point improvement on medR relative to the
single task pipeline. Multitask and single task systems attained stronger
scores on text to image retrieval for both datasets with double the R@K values
and medR at 11.9 - underling the superiority of the multitask approach in
cross-modal retrieval.

#### 6.2.2 Location Estimation

A summary of performance for multitask and single task systems on location
estimation is presented in Table. 4. Results are broken down by performance on
predicting geo-cells using images and text consecutively. As with cross-modal
retrieval, both systems return stronger scores when inputs are textual
relative to visual. A notable difference with the first task is that
performance improves on the geo-representative dataset. This difference is
most marked in image-based estimation: the F1 score for the multitask system
is 10 points higher on MLM-irle-gr relative to the larger dataset. Scores for
the multitask system on both image- and text-based location estimation are
closely aligned with the single task system - indicating there are only modest
gains from using the former approach on the second task.

## 7 Related Research

In the first instance, researchers designing multitask systems that learn from
multimodal data turn to existing multimodal resources for benchmarking. A
reliance on image captioning datasets has led to multitask learning proposals
limited to paired image and text samples [8, 34]. Datasets in the medical and
bioinformatics areas that have inspired multitask approaches also use these
two modalities [39, 27]. Several proposals pair natural language and knowledge
graph embeddings for training multitask systems [33]. In contrast to our
research, tasks in this area are concentrated on applications related to NLP
and knowledge graphs - and use text-only data. Multitask learning on
multimodal data is an area of research in robotics [11, 44] \- although the
objective of training autonomous agents is distinct from the use cases for
MLM. In all of the cited research, text inputs are English-only. Multilingual
resources and benchmarks for training multitask architectures are motivated
primarily to solve NLP tasks and are correspondingly unimodal [22, 31]. A
multitask solution was proposed for the multilingual IKEA dataset with a
specific objective in machine translation [58]. As with previously noted
cases, this resource is bimodal and so applies to a smaller subset of
constituent tasks relative to MLM.

Research on information retrieval for multimodal data has an extensive history
and we focus here on resources and benchmark tasks where samples are present
in multiple languages. The Cross-Language Evaluation Forum (CLEF) evolved into
ImageCLEF (from 2003), GeoCLEF (from 2005) and VideoCLEF (from 2008) tracks.
Resources in the LifeCLEF tracks of ImageCLEF have included geocoordinates -
but the focus is primarily on images and is aimed at researchers in
disciplines related to biology [23]. Information retrieval on video content
has extended to multilingual scenarios where either queries, content, or both
are in several languages [3]. Datasets in this area are mostly composed of
comparatively small numbers of samples. Constituent modalities (ie video,
audio) vary from those in MLM and place an emphasis on the spatiotemporal
context of the inputs [51]. Video retrieval is also differentiated by an
emphasis on domain-specific sub-tasks including shot segmentation [18] and
detection of objects or scenes [51]. All of these resources exclude semantic
Web elements.

Location estimation methods related to this research breakdown into three main
groups [10]: natural [4, 56], city-scale [24, 14] and global [26, 48]. Several
methods in the published research rely on a single modality [52, 38, 36] and
we focus here on multimodal approaches. MediaEval benchmark placing datasets
[29, 15] include more than five million instances with images, videos and
metadata used to estimate capturing locations represented in multimedia.
Kordopatis-Zilos et al. [25] build on this dataset by combining visual
features with a language model based on word frequency in the text for
geolocation estimation. In [50] a dataset containing image and text is
collected from Wikipedia combined with around 890K satellite images and is
used for interpreting satellite images. BreakingNews [42] is a resource with
100K instances in the domains of sports, politics, arts, healthcare or local
news. The researchers apply a baseline that combines a Convolutional Neural
Network and Recurrent Neural Networks for multimodal geolocation estimation.
In addition there are other approaches that use geolocation information for
image recognition [46, 17] and image understanding [54]. In [54] the authors
leverage season and location context with a probabilistic framework to help
improve region recognition in images. In [17] the effectiveness of using
geolocation on fine-grained recognition is examined. All the aforementioned
methods are based on datasets in English only. All these resources are
intended for single task scenarios for the corresponding task. In contrast,
this paper provides a multilingual dataset with a great number of modalities.
The objective also differs in the focus on systems that perform geolocation
estimation as one of a sequence of tasks.

Research on different forms of bias in data - and their impact on machine
learning systems - underlines the value of resources that provide balanced
coverage of their domains. Representation bias in relation to geographical
location is a form of bias that has high relevance to MLM and has been
identified in open source data sets used widely to train machine learning
systems [43]. Representation bias results in poor classification performance
on samples for classes that are under-sampled in the data. In another part of
the machine learning pipeline, evaluation bias results when the test data
fails to match the population of interest [45]. We are motivated by this
research to address representation and evaluation bias in this resource.

## 8 Conclusion and Future Work

In this research, we introduced the MLM resource for evaluating multitask
systems on diverse data. In addition, we specified a set of benchmark tasks
that characterise the specific use case of geoparsing and georeferencing
information encoded in multiple modalities and languages. A baseline multitask
system that makes use of both cross-modal and multimodal fusion was specified
and evaluated against single task systems. The paper details a process that
generates a resource comprising linked entities. The generation process also
enables consistent and rapid scaling of the resource for future releases.
Finally we also ensured that the dataset meets FAIR principles and released a
geo-representative version of the dataset to serve cultural and academic
institutions with a focus on the European Union. We believe that the resource
will improve generalisation in multitask learning systems and so streamline
the pipelines of applications that learn from materials stored in digital
archives. In upcoming research, we will release future versions with data in
additional languages, add entries from other data sources, design systems for
specific functions, and develop an application that further exploits
contextual data.

## 9 Acknowledgments

The project leading to this publication has received funding from the European
Union’s Horizon 2020 research and innovation programme under the Marie
Skłodowska-Curie grant agreement No 812997.

## References

  * [1] Beatrice Alex, Kate Byrne, Claire Grover, and Richard Tobin.  Adapting the Edinburgh geoparser for historical georeferencing.  International Journal of Humanities and Arts Computing, 9(1):15–35, 2015.
  * [2] Mehdi Ali, Hajira Jabeen, Charles Tapley Hoyt, and Jens Lehmann.  The KEEN universe.  In International Semantic Web Conference, pages 3–18. Springer, 2019.
  * [3] Yusuf Aytar, Mubarak Shah, and Jiebo Luo.  Utilizing semantic word similarity measures for video retrieval.  In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8. IEEE, 2008.
  * [4] Georges Baatz, Olivier Saurer, Kevin Köser, and Marc Pollefeys.  Large scale visual geo-localization of images in mountainous terrain.  In European conference on computer vision, pages 517–530. Springer, 2012.
  * [5] Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency.  Multimodal machine learning: A survey and taxonomy.  IEEE transactions on pattern analysis and machine intelligence, 41(2):423–443, 2018.
  * [6] Tilman Becker, Edward Curry, Anja Jentzsch, and Walter Palmetshofer.  New Horizons for a Data-Driven Economy: Roadmaps and Action Plans for Technology, Businesses, Policy, and Society.  In José María Cavanillas, Edward Curry, and Wolfgang Wahlster, editors, New Horizons for a Data-Driven Economy: A Roadmap for Usage and Exploitation of Big Data in Europe, pages 277–291. Springer International Publishing, Cham, 2016.
  * [7] Yoshua Bengio.  Learning Deep Architectures for AI.  Foundations and Trends in Machine Learning, 2:71, 2009.
  * [8] Alexander Binder, Wojciech Samek, Klaus-Robert Müller, and Motoaki Kawanabe.  Enhanced representation and multi-task learning for image annotation.  Computer Vision and Image Understanding, 117(5):466–478, 2013.
  * [9] Nicolas Blanc, Timothée Produit, and Jens Ingensand.  A semi-automatic tool to georeference historical landscape images.  Technical report, PeerJ Preprints, 2018.
  * [10] Jan Brejcha and Martin Čadík.  State-of-the-art in visual geo-localization.  Pattern Analysis and Applications, 20(3):613–637, 2017.
  * [11] Simon Brodeur, Ethan Perez, Ankesh Anand, Florian Golemo, Luca Celotti, Florian Strub, Jean Rouat, Hugo Larochelle, and Aaron C. Courville.  Home: a household multimodal environment.  In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Workshop Track Proceedings. OpenReview.net, 2018.
  * [12] Barbara Caputo, Henning Müller, Jesus Martinez-Gomez, Mauricio Villegas, Burak Acar, Novi Patricia, Neda Marvasti, Suzan Üsküdarlı, Roberto Paredes, Miguel Cazorla, et al.  Imageclef 2014: Overview and analysis of the results.  In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 192–211. Springer, 2014.
  * [13] Rich Caruana.  Multitask learning.  Machine learning, 28(1):41–75, 1997.
  * [14] David M Chen, Georges Baatz, Kevin Köser, Sam S Tsai, Ramakrishna Vedantham, Timo Pylvänäinen, Kimmo Roimela, Xin Chen, Jeff Bach, Marc Pollefeys, et al.  City-scale landmark identification on mobile devices.  In CVPR 2011, pages 737–744. IEEE, 2011.
  * [15] Jaeyoung Choi, Claudia Hauff, Olivier Van Laere, and Bart Thomee.  The placing task at MediaEval 2015.  In MediaEval 2015, Wurzen, Germany, 14-15 September 2015; Ceur Workshop Proceedings 1436, 2015. CEUR, 2015.
  * [16] Jaeyoung Choi, Martha Larson, Gerald Friedland, and Alan Hanjalic.  From Intra-Modal to Inter-Modal Space: Multi-task Learning of Shared Representations for Cross-Modal Retrieval.  In 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM), pages 1–10, Singapore, Singapore, September 2019. IEEE.
  * [17] Grace Chu, Brian Potetz, Weijun Wang, Andrew Howard, Yang Song, Fernando Brucher, Thomas Leung, and Hartwig Adam.  Geo-aware networks for fine-grained recognition.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [18] Bertrand Delezoide and Hervé Le Borgne.  Semanticvox: A multilingual video search engine.  In Proceedings of the 6th ACM international conference on Image and video retrieval, pages 81–84, 2007.
  * [19] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [20] Mouna Harrach, Alexandre Devaux, and Mathieu Brédif.  Interactive image geolocalization in an immersive web application.  International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences, 2019.
  * [21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Identity mappings in deep residual networks.  In European conference on computer vision, pages 630–645. Springer, 2016.
  * [22] Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson.  Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization.  arXiv preprint arXiv:2003.11080, 2020.
  * [23] Alexis Joly, Hervé Goëau, Hervé Glotin, Concetto Spampinato, Pierre Bonnet, Willem-Pier Vellinga, Julien Champ, Robert Planqué, Simone Palazzo, and Henning Müller.  Lifeclef 2016: multimedia life species identification challenges.  In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 286–310. Springer, 2016.
  * [24] Hyo Jin Kim, Enrique Dunn, and Jan-Michael Frahm.  Learned contextual feature reweighting for image geo-localization.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3251–3260. IEEE, 2017.
  * [25] Giorgos Kordopatis-Zilos, Symeon Papadopoulos, and Ioannis Kompatsiaris.  Geotagging text content with language models and feature mining.  Proceedings of the IEEE, 105(10):1971–1986, 2017.
  * [26] Giorgos Kordopatis-Zilos, Symeon Papadopoulos, and Yiannis Kompatsiaris.  In-depth exploration of geotagging performance using sampling strategies on yfcc100m.  In Proceedings of the 2016 ACM Workshop on Multimedia COMMONS, pages 3–10, 2016.
  * [27] Tomasz Kornuta, Deepta Rajan, Chaitanya Shivade, Alexis Asseman, and Ahmet S. Ozcan.  Leveraging medical visual question answering with supporting facts.  In Linda Cappellato, Nicola Ferro, David E. Losada, and Henning Müller, editors, Working Notes of CLEF 2019 - Conference and Labs of the Evaluation Forum, Lugano, Switzerland, September 9-12, 2019, volume 2380 of CEUR Workshop Proceedings. CEUR-WS.org, 2019.
  * [28] Ryohei Kuga, Asako Kanezaki, Masaki Samejima, Yusuke Sugano, and Yasuyuki Matsushita.  Multi-task learning using multi-modal encoder-decoder networks with shared skip connections.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 403–411, 2017.
  * [29] Martha Larson, Mohammad Soleymani, Guillaume Gravier, Bogdan Ionescu, and Gareth JF Jones.  The benchmarking initiative for multimedia evaluation: Mediaeval 2016\.  IEEE MultiMedia, 24(1):93–96, 2017.
  * [30] Qing Li, Qingyi Tao, Shafiq Joty, Jianfei Cai, and Jiebo Luo.  Vqa-e: Explaining, elaborating, and enhancing your answers for visual questions.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 552–567, 2018.
  * [31] Ying Lin, Shengqi Yang, Veselin Stoyanov, and Heng Ji.  A multi-lingual multi-task architecture for low-resource sequence labeling.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 799–809, 2018.
  * [32] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and S Yu Philip.  Learning multiple tasks with multilinear relationship networks.  In Advances in neural information processing systems, pages 1594–1603, 2017.
  * [33] Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi.  Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3219–3232, Brussels, Belgium, 2018. Association for Computational Linguistics.
  * [34] Junyu Luo, Ying Shen, Xiang Ao, Zhou Zhao, and Min Yang.  Cross-modal image-text retrieval with multitask learning.  In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 2309–2312, 2019.
  * [35] Javier Marin, Aritro Biswas, Ferda Ofli, Nicholas Hynes, Amaia Salvador, Yusuf Aytar, Ingmar Weber, and Antonio Torralba.  Recipe1m+: A dataset for learning cross-modal embeddings for cooking recipes and food images.  IEEE transactions on pattern analysis and machine intelligence, 2019\.
  * [36] Stuart E Middleton, Giorgos Kordopatis-Zilos, Symeon Papadopoulos, and Yiannis Kompatsiaris.  Location extraction from social media: Geoparsing, location disambiguation, and geotagging.  ACM Transactions on Information Systems (TOIS), 36(4):1–27, 2018\.
  * [37] Ludovic Moncla, Mauro Gaio, Thierry Joliveau, and Yves-François Le Lay.  Automated geoparsing of Paris street names in 19th century novels.  In Proceedings of the 1st ACM SIGSPATIAL Workshop on Geospatial Humanities, pages 1–8, 2017.
  * [38] Eric Muller-Budack, Kader Pustu-Iren, and Ralph Ewerth.  Geolocation estimation of photos using a hierarchical model and scene classification.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 563–579, 2018.
  * [39] Binh D Nguyen, Thanh-Toan Do, Binh X Nguyen, Tuong Do, Erman Tjiputra, and Quang D Tran.  Overcoming data limitation in medical visual question answering.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 522–530. Springer, 2019.
  * [40] Luc Pauwels.  A multimodal framework for analyzing websites as cultural expressions.  Journal of Computer-Mediated Communication, 17(3):247–265, 2012\.
  * [41] Miguel De Prado, Jing Su, Rabia Saeed, Lorenzo Keller, Noelia Vallez, Andrew Anderson, David Gregg, Luca Benini, Tim Llewellynn, Nabil Ouerhani, Rozenn Dahyot, and Nuria Pazos.  Bonseyes ai pipeline—bringing ai to you: End-to-end integration of data, algorithms, and deployment tools.  ACM Trans. Internet Things, 1(4), August 2020.
  * [42] Arnau Ramisa, Fei Yan, Francesc Moreno-Noguer, and Krystian Mikolajczyk.  Breakingnews: Article annotation by image and text processing.  IEEE transactions on pattern analysis and machine intelligence, 40(5):1072–1085, 2017.
  * [43] Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D. Sculley.  No classification without representation: Assessing geodiversity issues in open data sets for the developing world.  In NIPS 2017 workshop: Machine Learning for the Developing World, 2017.
  * [44] Devendra Singh Chaplot, Lisa Lee, Ruslan Salakhutdinov, Devi Parikh, and Dhruv Batra.  Embodied multimodal multitask learning.  arXiv preprint arXiv:1902.01385, 2019.
  * [45] Harini Suresh and John V Guttag.  A framework for understanding unintended consequences of machine learning.  arXiv preprint arXiv:1901.10002, 2019.
  * [46] Kevin Tang, Manohar Paluri, Li Fei-Fei, Rob Fergus, and Lubomir Bourdev.  Improving image classification with location context.  In Proceedings of the IEEE international conference on computer vision, pages 1008–1016, 2015.
  * [47] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li.  Yfcc100m: The new data in multimedia research.  Communications of the ACM, 59(2):64–73, 2016.
  * [48] Michele Trevisiol, Hervé Jégou, Jonathan Delhumeau, and Guillaume Gravier.  Retrieving geo-location of videos with a divide & conquer hierarchical multimodal approach.  In Proceedings of the 3rd ACM conference on International conference on multimedia retrieval, pages 1–8, 2013.
  * [49] Theodora Tsikrika, Adrian Popescu, and Jana Kludas.  Overview of the wikipedia image retrieval task at imageclef 2011.  In CLEF (Notebook Papers/Labs/Workshop), volume 4, page 5, 2011\.
  * [50] Burak Uzkent, Evan Sheehan, Chenlin Meng, Zhongyi Tang, Marshall Burke, David B Lobell, and Stefano Ermon.  Learning to interpret satellite images using wikipedia.  In IJCAI, pages 3620–3626, 2019.
  * [51] Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang.  Vatex: A large-scale, high-quality multilingual dataset for video-and-language research.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4581–4591, 2019.
  * [52] Tobias Weyand, Ilya Kostrikov, and James Philbin.  Planet-photo geolocation with convolutional neural networks.  In European Conference on Computer Vision, pages 37–55. Springer, 2016.
  * [53] Xue-Wen Chen and Xiaotong Lin.  Big Data Deep Learning: Challenges and Perspectives.  IEEE Access, 2:514–525, 2014.
  * [54] Jie Yu and Jiebo Luo.  Leveraging probabilistic season and location context models for scene understanding.  In Proceedings of the 2008 international conference on Content-based image and video retrieval, pages 169–178, 2008.
  * [55] Yu Zhang and Qiang Yang.  A Survey on Multi-Task Learning.  arXiv:1707.08114 [cs], July 2018.  arXiv: 1707.08114.
  * [56] Yan-Tao Zheng, Ming Zhao, Yang Song, Hartwig Adam, Ulrich Buddemeier, Alessandro Bissacco, Fernando Brucher, Tat-Seng Chua, and Hartmut Neven.  Tour the world: building a web-scale landmark recognition engine.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 1085–1092. IEEE, 2009.
  * [57] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Places: A 10 million image database for scene recognition.  IEEE transactions on pattern analysis and machine intelligence, 40(6):1452–1464, 2017.
  * [58] Mingyang Zhou, Runxiang Cheng, Yong Jae Lee, and Zhou Yu.  A visual attention grounding neural model for multimodal machine translation.  arXiv preprint arXiv:1808.08266, 2018.