capbtabboxtable[][]

# Spatially concentrated social capital of urban residents

Ádám J. Kovács1,∗, Sándor Juhász1,2, Eszter Bokányi1,2, Balázs Lengyel1,2

(1ELKH Centre for Economic and Regional Studies, Agglomeration and Social
Networks Lendület Research Group
Budapest, H-1097, Hungary
2Corvinus University of Budapest; Laboratory for Networks, Technology and
Innovation
Budapest, H-1093, Hungary
∗Corresponding author<EMAIL_ADDRESS>
)

###### Abstract

Social connections that span across diverse urban neighborhoods can help
individual prosperity by mobilizing social capital in cities. Yet, how the
detailed spatial structure of social capital varies in lower- and higher-
income urban neighborhoods is less understood. This paper demonstrates that
the social capital measured on social networks is spatially more concentrated
for residents of lower-income neighborhoods than for residents of higher-
income neighborhoods. We map the micro-geography of individual online social
connections in the 50 largest metropolitan areas of the US using a large-scale
geolocalized Twitter dataset. We then analyze the spatial dimension of
individual social capital by the share of friends, closure, and share of
supported ties within circles of short distance radiuses (1, 5, and 10 km)
around users’ home location. We compare residents from below-median income
neighborhoods with above-median income neighborhoods, and find that users
living in relatively poorer neighborhoods have a significantly higher share of
connections in close proximity. Moreover, their network is more cohesive and
supported within a short distance from their home. These patterns prevail
across the 50 largest US metropolitan areas with only a few exceptions. The
found disparities in the micro-geographic concentration of social capital can
feed segregation and income inequality within cities harming social circles of
low-income residents.

##  1 Introduction

Social networks are essential channels to share information, knowledge, and
opportunities. Connections can help people in getting a job ([1]), in
achieving higher wealth ([2]) or career progress ([3]). One possibility to
theorize processes behind these widely observed phenomena is the concept of
social capital that stresses the importance of resources that individuals can
mobilize through their social connections ([4, 5, 6]). The structure of social
networks is key in characterizing social capital ([7]). For example, network
cohesion - often measured by triadic closure - is an important building block
of social capital as it facilitates trust and the emergence of norms, while
decreasing misbehavior ([8, 9]). In economics, common partners are thought to
support the sustainability of cooperation links; thus, such supported ties are
claimed essential for the accumulation of social capital ([10]).

An early review of [11] stresses that geography is also an important factor in
shaping social capital. For example, increasing geographical distance
decreases both the probability of social ties ([12, 13]) and the probability
of closed triads ([14]), implying the existence of spatial limits of social
capital. [15] and [16] discuss the importance of geographically non-local ties
that are thought to boost economic prosperity by providing access to new
knowledge and opportunities. Applying a weighted network approach, evidence
suggests that individuals with spatially diverse social networks are wealthier
than individuals whose connections concentrate in certain locations ([2]).
Yet, this previous research does not consider structural measures of social
capital. Thus, how triadic closure and supported ties at geographical distance
are related to the economic prosperity of individuals is still unknown.

In this paper, we aim to establish the empirical link between the structural
measures of the social capital of spatially projected social networks and
income characteristics of home neighborhoods. This is done by using large-
scale social media data that enables us to investigate individual social
networks across a wide range of cities. We demonstrate that neighborhood-level
income is related to the spatial structure of social capital in the vast
majority of investigated cities. In particular, we show that social capital of
residents in low-income neighborhoods are spatially more concentrated than
that of residents living in high-income neighborhoods.

Cities have been long looked at as major places of the accumulation of social
capital (for a comparison between individual and collectivist social capital
in cities, see [17]). [18] has stressed the spatial dimension of social
capital very early by arguing that social ties of residents concentrate around
their home neighborhood. This early conjecture has recently been confirmed
with Facebook data from New York City: on average, around 40% of users’
friends live within 10 miles from their homes ([19]). Paired with the
prevailing assortative mixing of individuals in cities ([20, 21, 22]), this
massive spatial concentration of social capital in cities has far-reaching
consequences on neighborhood dynamics and can be associated with segregation
and inequalities. In those cities where income difference is a major factor of
assortative mixing such that social capital of the rich and poor are hardly
overlapping, income inequalities rise ([23]). Related case studies from
Bangladesh ([24]) and Hungary ([25]) show that the concentration of social
capital in poor urban neighborhoods facilitates trust and stronger social
norms, but at the same time, limits social mobility. Such patterns can also be
observed in large-scale data: the spatial concentration of Facebook
friendships in New York City is stronger in areas with a lower average income
and a lower level of education ([19]).

We contribute to the above discussion by comparing the spatial dimension of
individual social capital - captured by structural measures - in relatively
poorer versus relatively richer neighborhoods across the 50 largest
metropolitan areas in the United States. This is done with a large Twitter
dataset on geotagged tweets from which the home location of users, their
mutual follower network, and the home location of their connections can be
identified. We project the individual level ego-networks of users on census
tracts that enables us to infer the average household income of the ego and
alters by census tract level information from the American Community Survey.
This way, we can map the structure and geography of social ties for more than
80,000 users across 50 US cities. Finally, we measure three aspects of
individual social capital concentration in urban space within a circle of
short distance radius (1, 5, and 10 km) around the home location: the share of
connections within the circle, triadic closure within the circle, and the
share of supported ties within the circle.

Findings confirm that users living in neighborhoods of below median household
income have spatially more concentrated social networks. Residents of lower
income neighborhoods are embedded in social networks that are significantly
more cohesive and include significantly more supported ties within a close
proximity. These patterns prevail, with few exceptions, in the 50 largest
metropolitan areas of the United States. A significant negative correlation
between the continuous value of neighborhood income and spatial concentration
of social capital give further support to the finding. Furthermore, spatial
concentration of social capital is associated with income assortativity,
because most spatially close connections of the poor are residents of
disadvantaged neighborhoods, among whom triadic closure is also more
prevalent.

The study highlights that the social network of people from lower income
neighborhoods can offer limited access to opportunities in cities. We provide
novel insights into individual level social capital measures by mapping the
concentration of triadic closure and supported ties in urban social networks.
Moreover, we present individual level social network features that can feed
the segregation and inequality patterns observed inside cities.

##  2 Data and technical approach

We use a unique database obtained from the online social network site Twitter.
A fraction of tweets (i.e. short user messages on the site) are ‘geotagged’
meaning that they have meta-information on the location from which they were
sent. These tweets originate from users who enabled the exact geolocation
option on their smartphones. Our dataset contains tweets collected between
2012 and 2013, and due to careful sample selection described in [26], this
dataset enables the study of spatial patterns of human behavior, see ([27, 28,
29, 30, 31]).

Figure 1 Identification of home location and the spatial projection of
individual social networks. (a) Home locations are identified as the places
with the highest share of tweets in the morning (0:00-8:00) and in the evening
hours (20:00-24:00). The histograms illustrate the process through an example
user. The selected home cluster is marked by red. (b) Spatially projected
social network of an example user. The home of ego is marked with red and home
of alters are marked with blue. Home locations are characterized by the
relation of the average household income in the respective census tract to the
median income in the city.

To identify the home location of users, the Friend-of-Friend algorithm ([32])
was used to cluster messages in space. Any two tweet coordinates are
considered to belong to the same cluster, if their separation is less than 1
km. For each cluster, the first two moments of the coordinate distribution are
determined. Before calculating the mean coordinates of the cluster, data
points are trimmed until all points are inside a 3$\sigma$ radius to eliminate
outliers. For more details about the process, see ([26, 29]). We focus on the
three highest cardinality clusters with minimum 50 tweets on weekdays. The
cluster with the highest share of tweets sent between 8PM and 8AM on weekdays
is identified as the home location of users. Figure 1a illustrates this
process through an example user.

The online social network of users is defined by their mutual followership
ties on the site. To concentrate on meaningful ego networks we only consider
users with at least 10 geolocated friends in the same metropolitan area. The
spatially projected social network of an example user is visualized in Figure
1b. It is important to note that ties ranging outside the focal metropolitan
area are disregarded because this allows a clearer comparison of social
capital around home location of the rich and the poor. Additionally, we use
American Community Survey (2012) data on average household income and
population at census tract level to proxy the socio-economic characteristics
of home locations. For simplification, we calculate the median income of
census tracts for each metropolitan area and categorize each tract as above-
or below the median income. Our final sample consist of 86,177 users. The
distribution of users across the 50 metropolitan areas are available on Figure
S1 in the Supplementary Material.

We illustrate the spatial concentration of social ties with three different
measures using concentric circles around the home locations. First, we
calculate the share of friends (number of friends divided by the total number
of friends) within these circles cumulatively, similarly to ([19]). Second, we
also consider the change in this measure by taking its first derivative.
Third, we count the number of additional friends within each subsequent
concentric annulus normalized by their area to obtain spatial density.
Notably, we exclude all ties of users that are farther away than 10 km
distance and all measures are then calculated on subnetworks of users.

We also consider two approaches to quantify the structural cohesion of
connections close to home. We first measure the local clustering coefficient
of ego users in their subnetworks within concentric circles around their
identified home locations. The formula of this metric is given by:

$Clustering_{r}=\frac{2L_{i}}{k_{i}(k_{i}-1)},$ (1)

where $L_{i}$ stands for the number of links between the $k_{i}$ neighbors of
node $i$ within given radius $r$. This allows us to capture the degree to
which the neighbors of a given node link to each other. Though neighborhood is
meant in terms of social distance, these nodes represent people who actually
live in the close neighborhood in physical space as well.

Second, we use the supported tie measure related to individual social capital
introduced by ([10]). Contrary to local clustering that is calculated for
nodes, support is an edge characteristic. A social tie is considered to be
supported, if the two nodes linked by the relationship have at least one
common partner. The formula for the share of supported ties in the subnetwork
of users is the following:

$Tie\,support_{r}=\frac{|{j\in N_{i}(g):[g^{2}]_{ij}>0}|}{d_{r}},$ (2)

where the numerator is the absolute count of the number of friends $j$ of
$i$’s network $g$, who are supported by at least one friend in common within
radius $r$, and $d_{r}$ is the degree of the node (total number of friends)
within radius $r$.

|  Full graph |  Income $<$ median |  Income $>$ median
---|---|---|---
Degree in 10 km | 8 | 4 | 4
Share of ties in 1 km | 0.125 | 0.25 | 0
Share of ties in 5 km | 0.375 | 0.25 | 0.5
Share of ties in 10 km | 1 | 1 | 1
Clustering in 1 km | - | - | -
Clustering in 5 km | 0.333 | - | 1
Clustering in 10 km | 0.143 | 0.6 | 0.6
Support in 1 km | - | - | -
Support in 5 km | 0.667 | - | 1
Support in 10 km | 1 | 1 | 1

Table 1 Illustration of our key variables on an example graph. The red node
represents our focal user. The color of the nodes refers to the income level
at the census tract of users. The table describes the social capital related
measures calculated within 1-5-10 km for the focal user on the example graph.

Table 1 illustrates our key measures with the help of an example graph. To
construct these variables we take the subnetwork of each users’ ego network by
different distance thresholds. The focal user has a total degree of 8 within a
10 kilometer distance from the identified home location. 37.5% of the user’s
friends live within 5 kilometers distance and one of the friends (12.5%)
within 1 kilometer. The example user has 3 friends inside 5 kilometers, 2 of
whom follow each other. Therefore, the local clustering coefficient inside 5
kilometers is 0.333, while in 10 kilometers, it is 0.143. Within 10
kilometers, the share of supported ties is 1 as all of them are supported by
at least one friend. 66.7% of the ties in the example graph are supported
within 5 kilometers.

To consider assortative tie formation, we construct all these variables on
income-based subnetworks of users. More precisely, we create two more ego
networks for each individual, where we only consider friends living in above
median or below median income level neighborhoods. This allows us to measure
whether these structural features are more prominent inside income groups.
Table 1 illustrates that the example user has 25% of ties in only 1 km towards
lower income friends, but the network is more clustered and supported amongst
wealthier people in 5 kilometers.

##  3 Results

To illustrate the concentration of individual social capital inside cities and
to compare low- and high-income neighborhoods in this respect, we use three
approaches in Figure 2 that quantify concentration of social connections
within concentric circles around home locations. As Figure 2a illustrates, an
average user living in either a below- or above-median income neighborhood has
between 40% and 50% of their ties within a 10 kilometers radius around the
home location. We find that concentration of ties is stronger for people
living in lower income neighborhoods (home income $<$ median in the respective
city) than for people living in higher income areas. This difference is more
prevalent in Figure 2b, where we plot share of ties binned into distance
categories from home that enables us to evaluate typical distances of
concentration. Residents of lower-income neighborhoods are found to have the
highest share of ties who live 2.5 km away while residents of higher-income
neighborhoods have most friends 4.5 km away. In Figure 2a, the average
cumulative share of ties decreases sharply as we focus on small radius circles
around home that is due to the fact that smaller circles include lower number
of residents. Therefore, we measure the density of ties within circles of
distance $r$ around home (measurement is described in Data and Technical
Approach section) and find a monotonously decreasing probability of ties as
distance grows in Figure 2c. This monotonous decrease is less steep for lower-
income areas than for higher-income ones in the distance regime where Figure
2b also shows the separation of the two income classes. This shows that there
is a general trend towards the spatial concentration individual social
networks in micro-space.

Figure 2 Concentration of social ties around the home location of users in the
top 50 metropolitan areas of the US. (a) Cumulative share of connections in 10
km distance from home location. (b) Average share of connection at the
distances from home in 10 km. (c) Probability density of ties in 10 km.
(Normalized by the respective areas). All three figures represent average
values for users across the top 50 metropolitan areas of the US. We only
consider users with at least 10 connections with identified home location.
Median income is calculated for each metropolitan area separately.

Figure 3 Structural features of individual social networks inside 10 km. (a)
Average cumulative share of connections in 1/5/10 km for above and below
median income user groups. (b Average local clustering coefficient for higher
and lower income groups in 1/5/10 km. (c) Average tie support for users in
above or below median income neighborhoods in 1/5/10 km from their home. Error
bars represent 95$\%$ confidence intervals from bootstrap sampling.

In Figure 3, we proceed by comparing the $95\%$ confidence intervals of social
capital measures of lower- and higher-income neighborhood residents. In
general, there is a clear pattern that users from lower-income neighborhoods
have a spatially more concentrated ego-network, and on average, their ties are
more cohesive and involve a higher share of supported relationships in close
proximity to their home. Figure 3a illustrates that the share of ties is
significantly higher in lower-income neighborhoods in 1, 5 and 10 km circles.
Measuring the average of local clustering in the ego-networks of individuals
trimmed to concentric circles around their home tells us in Figure 3b that
higher-income neighborhoods concentrate significantly more closed triads in 1
km than lower-income neighborhoods. Because triadic closure is negatively
correlated with degree, the lower-income neighborhoods having more connections
within the 1k̃m circle might lead to this difference. Therefore, it is even
more striking that triadic closure is significantly higher in the 5 and 10 km
circles for lower-income neighborhoods despite a higher share of friends for
the same distance thresholds, with more than 10% of the triads being closed.
Concerning the share of supported ties, Figure 3c documents that spatial
concentration of supported ties is the highest among the social capital
indicators. More than 40% of supported ties are within 1 km and more than 60%
within 10 km. Residents of lower-income neighborhoods have significantly
higher shares of supported ties in all distance categories.

In Figure 4a-c, we illustrate the relationship between the income category of
home neighborhood and the spatial concentration of social capital in each of
the 50 metropolitan areas. The precise description of this controlled
correlation can be found in Section S3 of the Supplementary Material. Positive
coefficients mean that users in the respective city with above median income
tend to have higher share of their friends, higher triadic closure, and higher
share of supported ties in 10 km from their home. Results suggest that in most
of the cities residents of below-median income neighborhoods have higher share
of connections (Figure 4a) and higher number of supported ties (Figure 4c) in
10 kilometer distance. In case of the local clustering coefficient (Figure
4b), the picture is more mixed (19/50 cases the coefficient is positive) but
exceptions from the general trend are only few in case of share of friends
(7/50) and the share of supported ties (12/50). Interesting exceptions are San
Francisco, Detroit, Baltimore, and New Orleans, which suggests that both
prosperous and segregated cities can deviate from the trend. However, the
general tendency is clear that people living in lower income neighborhoods
have more concentrated social capital. Strong relationships along all three
dimensions are observed for both metropolitan areas with high population such
as Los Angeles and Miami, and at metro areas with smaller population such as
Buffalo, Raleigh and Salt Lake City.

Figure 4 Controlled correlation between above median home income and social
capital related measures in 10 km from home location. Dots represent linear
regression coefficients for each metro areas (see Section S3 of the
Supplementary Material for more details on the models). The color of nodes
indicate the sign of coefficients (gray ) and horizontal lines represent
95$\%$ confidence intervals. Nodes with lighter colors indicate statistically
insignificant coefficients.

To further increase the robustness of these findings, Table 2 presents linear
multivariate regression models, in which we test the correlation between the
continuous value of neighborhood average income and social capital
concentration within concentric circles around the home location. The table
contains models in which the dependent variable is social capital within 10 km
and further models including social capital concentration in 1 and 5 km are
presented in Table S1 in the Supplementary Material. In all regressions, the
respective metropolitan areas are used as fixed effects that allows for
sorting out the role of income differences across cities. We find that home
income level has a statistically significant negative association with the
spatial concentration of social capital. This result strengthens our claim
that lower-income neighborhoods tend to have spatially more concentrated
social capital than higher-income neighborhoods.

We include the population of the census tract as a control variable because
these are not uniform even within metropolitan areas. As expected, population
is significantly associated with the social capital measures. The negative
correlation between population and share of friends within 10 km implies that
people in populous neighborhoods tend to be connected to others in distant
neighborhoods. On the contrary, the positive correlations between population
and triadic closure and share of supported ties suggest that the size of
neighborhoods facilitates spatial concentration of social capital by enabling
network cohesion. (Population is negatively associated with triadic closure
and has no significant relationship with supported ties in the 1 km circle, as
reported in Supplementary Material.)

Moreover, we also include the overall degree of users (irrespective of the
distance of friends) as a further control variable. This way, we make sure
that our results on the spatial concentration of the social network measures
are not confounded by some users simply having an excess number of friends
compared to others. The negative correlation between degree and share of
friends within 10 km distance implies that having more connections goes
together with a more widespread social network. A similar negative
relationship is found for the clustering coefficient, which is unsurprising
due to more friends having a lower probability of knowing each other.
Conversely, the share of supported ties correlates positively with degree,
implying that having more connections increases the probability of having at
least one common partner with any one friend.

Further generalized robustness checks of the results in the form of
regressions with dependent variables being the same measures within 1 and 5 km
distance from the home locations of users can be found in Supplementary
Material Section S2.

| In 10 km from home location
---|---
| Share of friends | Clustering | Tie support
| (1) | (2) | (3)
Home income (log) | $-0.106^{***}$ | $-0.031^{***}$ | $-0.075^{***}$
| $(0.004)$ | $(0.003)$ | $(0.006)$
|  |  |
Home population (log) | $-0.057^{***}$ | $0.013^{***}$ | $0.061^{***}$
| $(0.004)$ | $(0.003)$ | $(0.006)$
|  |  |
Degree | $-0.001^{***}$ | $-0.001^{***}$ | $0.002^{***}$
| $(0.0001)$ | $(0.00004)$ | $(0.0001)$
|  |  |
Constant | $1.042^{***}$ | $0.242^{***}$ | $0.638^{***}$
| $(0.024)$ | $(0.017)$ | $(0.033)$
|  |  |
Metro FE | Yes | Yes | Yes
Observations | 86,177 | 74,900 | 74,900
R2 | 0.055 | 0.039 | 0.027
Adjusted R2 | 0.054 | 0.038 | 0.026
Note: |  ∗p$<$0.1; ∗∗p$<$0.05; ∗∗∗p$<$0.01

Table 2 Linear regression models with metropolitan area fixed effects. The
number of observations is smaller in case of clustering and tie support,
because their calculation requires at least 2 ties within 10 km distance

Besides their structural features, we also investigate the homogeneity of the
individual ego networks in terms of income level. To this end, we decompose
individual ego-networks analyzed above to income-based sub-networks that
contain friends who live in either above or below median income level
neighborhoods. This enables us to compare the structural characteristics of
social networks across income groups.

Figure 5 shows income segregation in all three approaches of social capital
measurement. In Figure 5a, we find that residents of higher income
neighborhoods have on average almost 60% of their ties to users living also in
higher income neighborhoods. Income homophily has an even greater role in the
case of residents of lower-income neighborhoods: 70% of their friends live in
lower-income neighborhoods. Similar patterns are found in terms of the
clustering coefficient, and the share of supported ties. Users living in
relatively poor census tracts tend to have higher closure in their networks
among their friends who also come from lower income neighborhoods. The tie
support measure further corroborates this observation, most of the ties users
living in less affluent neighborhoods have to likewise people are supported
relationships. Though homophily is also apparent in both of these measures
among those coming from higher-income neighborhoods, its extent is always
superior in case they are from lower-income neighborhoods. Taken together,
income segregation is prevalent in social networks that span across
neighborhoods in cities and can harm residents of lower-income neighborhoods
who are typically connected to and accumulate social capital with lower-income
residents.

Figure 5 Homophily among the social networks of users within 10 km distance.

##  4 Discussion

Social capital is key for prosperity in life. This work contributes to the
literature on social capital by illustrating that social network features of
individuals measuring some aspects of social capital are spatially
concentrated in urban areas. This adds to the discussion in several points.

First, we show that connections of individuals concentrate within only a few
kilometers around the place of residence and these connections tend to be
closed and supported. We find that spatial concentration of social capital is
stronger for people living in lower-income neighborhoods, which is a general
notion across the top 50 metropolitan areas of the US. The spatial
concentration of social ties and social capital inside metropolitan areas also
shows that like in ([19]), edge formation probability is distance dependent
within cities as well as on a larger scale ([12, 13]) making spatial
constraints important in the formation of the social fabric of cities. The
physical proximity of home neighborhoods are clearly crucial places for social
capital accumulation, especially for the unprivileged. Therefore, social
housing and redevelopment policies have to focus on changes in access to
resources and norms embedded in people’s social networks when relocating
individuals.

Second, we adopt the novel individual social capital related network measure
of ([33]) to a spatial social network. Results suggest that similarly to the
clustering of social ties, supported relationships are also highly
concentrated inside 10 kilometers. The negative coefficient of income
predicting the share of supported ties in Table 3 is able to support the
relationship between lower income and spatially concentrated social capital
similarly to the clustering coefficient.

Third, the pattern that the social capital of people living in lower income
areas is more concentrated in urban space connects to several other works on
capturing societal challenges through online social media data ([19, 22, 20,
21, 31]). Multiple studies address the problem that online social networks and
interactions mirror offline segregation and inequality within metropolitan
areas ([23]). Our work is able to show the micro-foundations of these
segregation patterns at the individual ego-network level. It is important to
note here that our measures are not directly dependent on the structure of the
whole network of the metropolitan area, neither are they calculated from area-
based metrics. Therefore, this large-scale ego-network approach is unique,
especially at its scale, and we hope that future research will be able to make
use of the anonymized network metrics published alongside this paper.

Our approach has some limitations. The definition of a social connection in
this paper is based on mutual followership on the Twitter online social
networking platform. While mutual followership signifies a mutual attention,
due to biases both in the user-base and the sampling of the data, these
relationships might not fully capture offline social relationships. Also, the
definition itself might be unable to include other types of relationships that
could be significant in the measurement of social capital. Information is not
available on the nature of social ties i.e. whether two mutual followers are
co-workers, friends from school or college. Future research might look into
the concentration of social capital by investigating labeled and weighted
social networks.

While the number of users within metropolitan areas included in our analysis
is proportional to the population, the user samples can be unrepresentative of
the whole population regarding age, gender, income, and ethnicity. Within US
Twitter users, African Americans are overrepresented ([34]) while other
ethnicities might be underrepresented ([35, 36]). The users in our sample are
predominantly young and well-educated ([37, 38]). Therefore, we might not be
able to generalize our findings to the whole population of these metropolitan
areas. ([39]) suggests that the free 1% sample from Twitter Streaming API that
was used for the initial data collection is prone to errors because of bot
activity. By imposing strict count limits, spatio-temporal constraints and
mutual followership for ties, we tried to make our sample less distorted in
this aspect. Also, ([40]) and ([41]) confirm that tweets filtered to
containing GPS coordinates are retrieved to almost 90% of the time when
compared to the full dataset.

Our study uses neighborhood-level information from the American Community
Survey to infer the socio-economic status of users by matching their
identified home location to census tracts. The survey was conducted in 2012,
while the data from Twitter was collected during 2012-2013, which again allows
for some degree of bias. The socio-economic characteristics of the census
tracts may be subjects to change over time and the individuals can also move
to other neighborhoods. Finally, we did not consider deviations of income from
the mean.

This study does not follow changes in the network structure or socio-economic
indicators of people over time. A promising extension of this line of research
would be to track how social capital changes when moving to different areas of
a city. Also, adding context to social ties, e.g. coworker ties, school ties
might be possible from different data source, such as social networks inferred
from register data. Tie context might also be derived based on natural
language processing of the messages sent on the Twitter platform.

## Acknowledgements

Eszter Bokányi was supported by the ÚNKP-20-4 New National Excellence Program
of the Ministry for Innovation and Technology from the source of the National
Research, Development and Innovation Fund of Hungary. We thank for the usage
of ELKH Cloud (https://science-cloud.hu/) that significantly helped us
achieving the results published in this paper.

## Conflict of interest

The authors declare no conflict of interests.

## References

  * [1] M S Granovetter  “Getting a job: a study of contacts and careers”  University of Chicago Press, Chicago, 1995
  * [2] Nathan Eagle, Michael Macy and Rob Claxton  “Network Density and Economic Development”  In _Science_ 328.., 2010, pp. 1029–1031  DOI: 10.1126/science.1186605
  * [3] S E Seibert, M L Kraimer and R C Liden  “A social capital theory of career success”  In _Academy of Management Journal_ 44.2, 2001, pp. 219–237
  * [4] James S Coleman  “Social capital in the creation of human capital”  In _American journal of sociology_ 94 University of Chicago Press, 1988, pp. S95–S120
  * [5] Robert D. Putnam  “Bowling Alone: America’s Declining Social Capital”  In _Culture and Politics_ New York: Palgrave Macmillan US, 2000, pp. 223–234  DOI: 10.1007/978-1-349-62965-7˙12
  * [6] Nan Lin  “Social Capital: A Theory of Social Structure and Action”  New York: Cambridge University Press, 2001
  * [7] S P Borgatti, C Jones and M G Everett  “Network measures of social capital”  In _Connections_ 21.2, 1998, pp. 27–36
  * [8] James S Coleman  “Foundations of Social Theory”  The Belknap Press of Harvard University Press., 1990
  * [9] M S Granovetter  “The Impact of Social Structure on Economic Outcomes”  In _Journal of Economic Perspectives_ 19.1, 2005, pp. 33–50
  * [10] Matthew O. Jackson, Tomas Rodriguez-Barraquer and Xu Tan  “Social capital and social quilts: Network patterns of favor exchange”  In _American Economic Review_ 102.5, 2012, pp. 1857–1897  DOI: 10.1257/aer.102.5.1857
  * [11] Giles Mohan and John Mohan  “Placing social capital”  In _Progress in Human Geography_ 26.2 Sage Publications Sage CA: Thousand Oaks, CA, 2002, pp. 191–210  DOI: 10.1191/0309132502ph364ra
  * [12] David Liben-Nowell et al.  “Geographic routing in social networks”  In _Proceedings of the National Academy of Sciences_ 102.33 National Academy of Sciences, 2005, pp. 11623–11628  DOI: 10.1073/pnas.0503018102
  * [13] Balázs Lengyel et al.  “Geographies of an online social network”  In _PLoS ONE_ 10.9 Public Library of Science, 2015, pp. e0137248  DOI: 10.1371/journal.pone.0137248
  * [14] Renaud Lambiotte et al.  “Geographical dispersal of mobile communication networks”  In _Physica A: Statistical Mechanics and its Applications_ 387.21, 2008, pp. 5317–5325  DOI: 10.1016/j.physa.2008.05.014
  * [15] Harald Bathelt, Anders Malmberg and Peter Maskell  “Clusters and knowledge: Local buzz, global pipelines and the process of knowledge creation”  In _Progress in Human Geography_ 28.1 Sage publications Sage CA: Thousand Oaks, CA, 2004, pp. 31–56  DOI: 10.1191/0309132504ph469oa
  * [16] Johannes Glückler  “Economic geography and the evolution of networks”  In _Journal of Economic Geography_ 7.5 Oxford University Press, 2007, pp. 619–634  DOI: 10.1093/jeg/lbm023
  * [17] Alejandro Portes  “The Two Meanings of Social Capital”  In _Sociological Forum_ 15.1, 2000, pp. 1–12  Springer  DOI: 10.1023/A:1007537902813
  * [18] J Jacobs  “The Death and Life of Great American Cities.”  Random House, New York., 1961
  * [19] Michael Bailey, Patrick Farrell, Theresa Kuchler and Johannes Stroebel  “Social connectedness in urban areas”  In _Journal of Urban Economics_ 118, 2020, pp. 103264  DOI: 10.1016/j.jue.2020.103264
  * [20] Alfredo J. Morales, Xiaowen Dong, Yaneer Bar-Yam and Alex ‘Sandy’ Pentland  “Segregation and polarization in urban areas”  In _Royal Society Open Science_ 6.10, 2019, pp. 190573  DOI: 10.1098/rsos.190573
  * [21] Xiaowen Dong et al.  “Segregated interactions in urban and online space”  In _EPJ Data Science_ 9.1, 2020, pp. 20  DOI: 10.1140/epjds/s13688-020-00238-7
  * [22] Qi Wang, Nolan Edward Phillips, Mario L. Small and Robert J. Sampson  “Urban mobility and neighborhood isolation in America’s 50 largest cities”  In _Proceedings of the National Academy of Sciences of the United States of America_ 115.30, 2018, pp. 7735–7740  DOI: 10.1073/pnas.1802537115
  * [23] Gergő Tóth et al.  “Inequality is rising where social network segregation interacts with urban topology”  In _Nature Communications_ 12.1, 2021, pp. 1143  DOI: 10.1038/s41467-021-21465-0
  * [24] Toriqul Bashar and Glen Bramley  “Social capital and neighbourhood cooperation: Implications for development of the urban poor in LDCs”  In _Urban Studies_ 56.13 SAGE Publications Sage UK: London, England, 2019, pp. 2727–2745  DOI: 10.1177/0042098018797945
  * [25] Boglárka Berki, György Málovics, Janka Tóth and Creţan Remus  “The role of social capital and interpersonal relations in the alleviation of extreme poverty and spatial segregation of Romani people in Szeged”  In _Journal of Urban and Regional Analysis_ 9, 2017, pp. 33–50  DOI: 10.37043/JURA.2017.9.1.2
  * [26] Laszlo Dobos et al.  “A multi-terabyte relational database for geo-tagged social network data”  In _2013 IEEE 4th International Conference on Cognitive Infocommunications (CogInfoCom)_ IEEE, 2013, pp. 289–294  DOI: 10.1109/CogInfoCom.2013.6719259
  * [27] Eszter Bokányi et al.  “Race, religion and the city: twitter word frequency patterns reveal dominant demographic dimensions in the United States”  In _Palgrave Communications_ 2.1 Nature Publishing Group, 2016, pp. 16010  DOI: 10.1057/palcomms.2016.10
  * [28] Zsófia Kallus, Norbert Barankai, János Szüle and Gábor Vattay  “Spatial Fingerprints of Community Structure in Human Interaction Network for an Extensive Set of Large-Scale Regions”  In _PLOS ONE_ 10.5, 2015, pp. e0126713  DOI: 10.1371/journal.pone.0126713
  * [29] Zsófia Kallus et al.  “Video Pandemics: Worldwide Viral Spreading of Psy’s Gangnam Style Video”  In _ICT Innovations 2017: Data-Driven Innovation. 9th International Conference, ICT Innovations 2017, Skopje, Macedonia, September 18-23, 2017, Proceedings_ Cham: Springer International Publishing, 2017, pp. 3–12  DOI: 10.1007/978-3-319-67597-8˙1
  * [30] Eszter Bokányi, Zoltán Lábszki and Gábor Vattay  “Prediction of employment and unemployment rates from Twitter daily rhythms in the US”  In _EPJ Data Science_ 6.1, 2017, pp. 14  DOI: 10.1140/epjds/s13688-017-0112-x
  * [31] Eszter Bokányi, Sándor Juhász, Márton Karsai and Balázs Lengyel  “Universal role of commuting in the reduction of social assortativity in cities”  In _arXiv_ , 2021  arXiv: https://arxiv.org/abs/2105.01464
  * [32] J P Huchra and M.. Geller  “Groups of galaxies. I - Nearby groups”  In _The Astrophysical Journal_ 257, 1982, pp. 423  DOI: 10.1086/160000
  * [33] Matthew O Jackson  “A typology of social capital and associated network measures”  In _Social Choice and Welfare_ 54.2-3, 2020, pp. 311–336  DOI: 10.1007/s00355-019-01189-3
  * [34] Eszter Hargittai and Eden Litt  “The tweet smell of celebrity success: Explaining variation in Twitter adoption among a diverse group of young adults”  In _New Media & Society_ 13.5, 2011, pp. 824–842  DOI: 10.1177/1461444811405805
  * [35] Alan Mislove et al.  “Understanding the Demographics of Twitter Users”  In _Int’l AAAI Conference on Weblogs and Social Media (ICWSM)_ , 2011, pp. 554–557
  * [36] Momin M. Malik, Hemank Lamba, Constantine Nakos and Jürgen Pfeffer  “Population bias in geotagged tweets”  In _AAAI Workshop - Technical Report_ WS-15-18, 2015, pp. 18–27
  * [37] Tom Webster  “Twitter Usage In America : 2010”  In _Edison Research/ Arbitron Internet and Multimedia Study._ , 2010
  * [38] Luke Sloan, Jeffrey Morgan, Pete Burnap and Matthew Williams  “Who Tweets? Deriving the Demographic Characteristics of Age, Occupation and Social Class from Twitter User Meta-Data”  In _Plos One_ 10.3, 2015, pp. e0115545  DOI: 10.1371/journal.pone.0115545
  * [39] Jürgen Pfeffer, Katja Mayer and Fred Morstatter  “Tampering with Twitter’s Sample API”  In _EPJ Data Science_ 7.1 The Author(s), 2018, pp. 50  DOI: 10.1140/epjds/s13688-018-0178-0
  * [40] Kenneth Joseph, Peter M Landwehr and Kathleen M Carley  “Two 1%s Don’t Make a Whole: Comparing Simultaneous Samples from Twitter’s Streaming API”  In _Association fo the Advanced of Artificial Intelligence_ , 2014, pp. 75–83  DOI: 10.1007/978-3-319-05579-4˙10
  * [41] Fred Morstatter, Jürgen Pfeffer and Huan Liu  “When is it biased?”  In _Proceedings of the 23rd International Conference on World Wide Web - WWW ’14 Companion_ New York, New York, USA: ACM Press, 2014, pp. 555–556  DOI: 10.1145/2567948.2576952

## Supplementary Material

### S1 - Distribution of users across the US

Map of the selected 50 metropolitan areas in the United States and the number
of users we follow in each metropolitan areas. Every user in the data have at
least 10 friends with identified home location in the respective metropolitan
area.

Figure S1 Top 50 metropolitan areas of the US and number of users in each
metropolitan area.

### S2 - Robustness checks

Additional regression models that supplement the models in Table 2 of the main
text.

| Share of friends
---|---
| In 10 km | In 5 km | In 1 km
| (1) | (2) | (3)
Home income (log) | $-0.106^{***}$ | $-0.092^{***}$ | $-0.007^{***}$
| $(0.004)$ | $(0.003)$ | $(0.001)$
|  |  |
Home population (log) | $-0.057^{***}$ | $-0.043^{***}$ | $-0.021^{***}$
| $(0.004)$ | $(0.003)$ | $(0.001)$
|  |  |
Degree | $-0.001^{***}$ | $-0.001^{***}$ | $-0.0002^{***}$
| $(0.0001)$ | $(0.00005)$ | $(0.00002)$
|  |  |
Constant | $1.042^{***}$ | $0.750^{***}$ | $0.137^{***}$
| $(0.024)$ | $(0.019)$ | $(0.007)$
|  |  |
Metro FE | Yes | Yes | Yes
Observations | 86,177 | 86,177 | 86,177
R2 | 0.055 | 0.061 | 0.027
Adjusted R2 | 0.054 | 0.060 | 0.026
Note: |  ∗p$<$0.1; ∗∗p$<$0.05; ∗∗∗p$<$0.01

Table S1 Linear regression models with metropolitan area fixed effects.
Dependent variable: Share of friends within 10, 5 and 1 km respectively.
Independent variables: natural logarithm of average income level and
population in users’ home neighborhoods, overall degree of users.

| Clustering coefficient
---|---
| In 10 km | In 5 km | In 1 km
| (1) | (2) | (3)
Home income (log) | $-0.031^{***}$ | $-0.035^{***}$ | $0.006^{**}$
| $(0.003)$ | $(0.003)$ | $(0.003)$
|  |  |
Home population (log) | $0.013^{***}$ | $-0.001$ | $-0.019^{***}$
| $(0.003)$ | $(0.003)$ | $(0.003)$
|  |  |
Degree | $-0.001^{***}$ | $-0.001^{***}$ | $-0.0005^{***}$
| $(0.00004)$ | $(0.00003)$ | $(0.00003)$
|  |  |
Constant | $0.242^{***}$ | $0.247^{***}$ | $0.080^{***}$
| $(0.017)$ | $(0.014)$ | $(0.016)$
|  |  |
Metro FE | Yes | Yes | Yes
Observations | 74,900 | 59,993 | 14,109
R2 | 0.039 | 0.045 | 0.075
Adjusted R2 | 0.038 | 0.044 | 0.071
Note: |  ∗p$<$0.1; ∗∗p$<$0.05; ∗∗∗p$<$0.01

Table S2 Linear regression models with metropolitan area fixed effects.
Dependent variable: Local clustering coefficient within 10, 5 and 1 km
respectively. Independent variables: natural logarithm of average income level
and population in users’ home neighborhoods, overall degree of users.

| Tie support
---|---
| In 10 km | In 5 km | In 1 km
| (1) | (2) | (3)
Home income (log) | $-0.075^{***}$ | $-0.094^{***}$ | $-0.066^{***}$
| $(0.006)$ | $(0.007)$ | $(0.015)$
|  |  |
Home population (log) | $0.061^{***}$ | $0.059^{***}$ | $0.005$
| $(0.006)$ | $(0.007)$ | $(0.017)$
|  |  |
Degree | $0.002^{***}$ | $0.002^{***}$ | $-0.0003$
| $(0.0001)$ | $(0.0001)$ | $(0.0002)$
|  |  |
Constant | $0.638^{***}$ | $0.672^{***}$ | $0.704^{***}$
| $(0.033)$ | $(0.042)$ | $(0.097)$
|  |  |
Metro FE | Yes | Yes | Yes
Observations | 74,900 | 59,993 | 14,109
R2 | 0.027 | 0.016 | 0.015
Adjusted R2 | 0.026 | 0.015 | 0.011
Note: |  ∗p$<$0.1; ∗∗p$<$0.05; ∗∗∗p$<$0.01

Table S3 Linear regression models with metropolitan area fixed effects.
Dependent variable: Share of supported ties within 10, 5 and 1 km
respectively. Independent variables: natural logarithm of average income level
and population in users’ home neighborhoods, overall degree of users.

### S3 - Linear models behind Figure 4

Figure 4 illustrates how the dummy variables (0/1) on home income $>$ median
income in the respective metro area ($HI_{I>m}$) correlates with share of
friends ($SoF_{d}$=10)), local clustering coefficient ($Clust_{d=10}$) and
supported ties ($Supp_{d=10}$) inside 10 kilometer ($d=10$) from the home
location of individual users. We estimate the relationship between home income
and the different network variables by controlling for the log transferred
population ($Pop$) of the home census tract. Figure 4 visualizes the
$\beta_{1}$ coefficients of the linear regression models below:

$SoF_{d=10}=\beta_{0}+\beta_{1}HI_{I>m}+\beta_{2}Pop$ (1)

$Clust_{d=10}=\beta_{0}+\beta_{1}HI_{I>m}+\beta_{2}Pop$ (2)

$Supp_{d=10}=\beta_{0}+\beta_{1}HI_{I>m}+\beta_{2}Pop$ (3)


# Deep Similarity Learning Loss Functions in Data Transformation for Class
Imbalance

Damian Horna
Poznan University of Technology
<EMAIL_ADDRESS>Mateusz Lango
Poznan University of Technology, Institute of Computing Science,
60-965 Poznan, Poland
Charles University, Faculty of Mathematics and Physics,
Prague, Czech Republic<EMAIL_ADDRESS>Jerzy Stefanowski
Poznan University of Technology, Institute of Computing Science,
60-965 Poznan, Poland<EMAIL_ADDRESS>
###### Abstract

Improving the classification of multi-class imbalanced data is more difficult
than its two-class counterpart. In this paper, we use deep neural networks to
train new representations of tabular multi-class data. Unlike the typically
developed re-sampling pre-processing methods, our proposal modifies the
distribution of features, i.e. the positions of examples in the learned
embedded representation, and it does not modify the class sizes. To learn such
embedded representations we introduced various definitions of triplet loss
functions: the simplest one uses weights related to the degree of class
imbalance, while the next proposals are intended for more complex
distributions of examples and aim to generate a safe neighborhood of minority
examples. Similarly to the resampling approaches, after applying such
preprocessing, different classifiers can be trained on new representations.
Experiments with popular multi-class imbalanced benchmark data sets and three
classifiers showed the advantage of the proposed approach over popular pre-
processing methods as well as basic versions of neural networks with classical
loss function formulations. Keywords multiple classes imbalanced data, deep
learning, triplet loss function, data difficulty factors
This is an extended version of the paper presented at LIDTA’2023 workshop at
ECMLPKDD2023 [15].

##  1 Introduction

Class imbalance is a common challenge in many domains, which severely degrades
the predictive ability of classifiers. It has been extensively studied in the
last decades and as a result, many methods have been proposed to improve
classifiers, see e.g. [4, 9, 13, 22]. However, most of the proposed methods
considered different modifications of the data or algorithms for fairly
classic classifiers, such as k-NN, decision trees, SVM, ensembles or similar
ones. Despite their effectiveness, especially for the processing of tabular
data, the situation has begun to change with the development of deep neural
networks. As noted by [20], one of the reasons for the success of deep
learning is the ability to map the original data to an embedded new
representation, where learning complex relationships can be much easier.

An increase in research interest in deep networks can also be observed in the
class imbalance community, see e.g. [6, 35]. However, the number of such works
is still insufficient. Moreover, most of them are concerned with the use of
known preprocessing techniques in combination with selected neural network
architectures, or with the modification of the error or loss functions to make
them more sensitive to minority class errors.

In our opinion, the impact of learning a new embedded representation in the
last layers of the network is still insufficiently exploited, especially for
tabular data. It should be directed to obtaining better separation of minority
classes examples from majority classes ones and linking this to learning
further more efficient classifiers. In this paper, we follow inspirations from
the contrastive learning paradigm [17]. The deep neural network is learned by
comparing the feature representations of selected examples and directing the
learning process of the network to find the differences and similarities
between examples from different classes.

According to [32] one can use triples of examples and the corresponding
contrast loss function, so-called triplet loss, so that examples from the same
class are brought closer to each other and at the same time are moved away
from an example from another class in the embedded representation. Such an
approach was mainly used to improve image recognition. Despite interesting
extensions, such as [16], it has not been sufficiently explored so far for
difficult imbalanced data. Therefore, in this paper, we will study their
usefulness for multi-class imbalanced data.

Recall that the multi-class imbalanced data are recognized as harder to learn
than two-class ones due to more complex relationships between many classes and
local difficulty characteristics for single examples from these classes [25].
Instead of looking for yet another method of solving such difficult multi-
class problems in the original data representations, we will examine the
possibility of learning new representations of these problems using
appropriately modified triplet loss functions and, then, building classifiers
in these embedded representations.

The contributions of our paper are the following:

  1. 1.

to introduce modifications of the triplet loss functions to deal with
imbalanced classes and to propose new ones to handle local difficulty of
learning examples;

  2. 2.

to present modifications of learning deep neural networks with them for
processing multiple imbalanced tabular data;

  3. 3.

to carry out the experimental analysis of proposed variants of these triplet
loss functions and to compare the best of them against using previously known
methods for dealing with multi-class imbalanced data.

##  2 Related works

The problems of learning from imbalanced data have been studied by many
researchers - see e.g. comprehensive reviews, such as [4, 9]. Most of the
current research concerns binary imbalanced data, i.e. a single minority class
versus the majority one [19]. Multi-class imbalanced classification is more
difficult and less frequently studied than its binary counterpart. The
specialized classification methods for this problem are usually divided into
two groups: decomposition-based approaches (the transformation of the multi-
class problem into a set of binary ones) and dedicated multi-class approaches
(mainly various pre-processing methods). It is often postulated that such
approaches should take into account both global class inter-relationships and
local data difficulty factors [29, 25, 33].

Multi-class imbalanced classification is more difficult and less frequently
studied than its binary counterpart. According to recent work, this may be due
to many factors: overlapping between multiple classes, its interaction with
various imbalance ratios, different configurations of class sizes and presence
of the local difficulty distribution in the neighbourhoods of examples from
particular classes [25, 31]. The specialized classification methods for this
problem are usually divided into two groups: decomposition-based approaches
and dedicated multi-class approaches. The first methods rely on transformation
of multi- class problems into a series of binary ones, which are solved
separately and later aggregated in a kind of ensembles. The other category are
usually data resampling techniques specialized for dealing with multiple
classes, see e.g. random oversampling of classes, such as Global-CS [37] and
more informed methods, e.g., Mahalanobis-distance over-sampling (MDO) [1] or
Static-SMOTE [10]. For their more comprehensive review see [9, 11]. Referring
to the identified sources of difficulties in multiple imbalanced data recall
that the new methods designed for improving classifiers should take into
account both both global class inter-relationships and local factors [23, 25,
28].

Despite advances in deep learning, multi-class imbalanced classification
problems in the context of deep neural networks were mostly addressed for
image classification or natural language processing tasks only; for their
survey see [6, 20]. Early proposals used resampling techniques before training
deep neural networks. More recent works modified the distribution of examples
in mini-batches during network training (see e.g. ReMix algorithm by [2]) or
oversampled the embedded representation. Another group of proposals is aimed
at modifying the loss function in network training to make it sensitive to
minority class errors, e.g. [34] modified the mean square error, [27]
introduced focal loss, [26] proposed Dice loss, etc. Finally, there are also
proposals of modifying the optimization task to handle different costs of
misclassifications, see e.g. [21]. In addition, the more recent proposals try
to preprocess the embedded representation, such as DeepSmote [8] which uses
the SMOTE algorithm on a hidden representation created with autoencoders or
with GAN deep networks [30]. The results of these and other proposals are also
discussed in [20].

In contrastive learning, the information about similarity or dissimilarity of
training examples is used to learn corresponding feature representation. One
of the most popular approaches is to train a neural network with triplet loss,
which has been introduced in [32]. The method was originally applied to the
face recognition task, where each face image was mapped to a compact Euclidean
space, where distances directly corresponded to a measure of face similarity.
They used a deep convolutional neural network and trained it using triplets of
matching / non-matching face patches generated using the triplet mining
method.

While using triplet loss, the neural network f, which maps a training example
$x\in X$ from the original feature space to the embedding space $f(x)\in
R^{d}$, is trained to ensure that a selected reference example $x_{i}^{a}$
(called an anchor) is closer to all other examples $x_{i}^{p}$ (positive) from
the same class than to any example $x_{i}^{n}$ (negative) from other classes
in the induced embedding space. More formally, the constraint can be described
as:

$\forall_{\left(f\left(x_{i}^{a}\right),f\left(x_{i}^{p}\right),f\left(x_{i}^{n}\right)\right)\in\mathcal{T}}\,\,\left\|f\left(x_{i}^{a}\right)-f\left(x_{i}^{p}\right)\right\|_{2}^{2}+\alpha<\left\|f\left(x_{i}^{a}\right)-f\left(x_{i}^{n}\right)\right\|_{2}^{2},$
(1)

where $\alpha$ is a margin that should be enforced between positive and
negative pairs and $\mathcal{T}$ is set of all possible triplets in the
training set. In practice, this constraint leads to optimization of the
following training objective:

$\sum_{i}^{|\mathcal{T}|}\left[\left\|f\left(x_{i}^{a}\right)-f\left(x_{i}^{p}\right)\right\|_{2}^{2}-\left\|f\left(x_{i}^{a}\right)-f\left(x_{i}^{n}\right)\right\|_{2}^{2}+\alpha\right]_{+}$
(2)

The authors of [32] claim that his formulation of the loss function may lead
to learning such embedded representations, where examples from the same class
may approach each other in subsequent epochs of network learning, forming
clusters. It worked very well in face recognitions and then it proved to be
successful especially while learning a similarity of images, in e.g., object
tracking or visual search. Such triplet loss functions were successfully
applied in image recognition, object tracking, and visual search. Recently
this idea has been modified to the quintuplet instance sampling and triple-
header hinge loss, which allows learning more complex relationships between
classes and their potential cluster in the embedded representations [7, 36,
16]. To the best of our knowledge, the triplet loss and its extensions have
not been used in the context of tabular, multi-class imbalanced data.

##  3 Modifying triplet networks for learning representation of imbalanced
data

The above-defined triplet loss, despite improving the separability of examples
from different classes in the hidden embedded representation, does not
directly take into account class imbalance or other local data difficulty
factors (e.g., class overlapping, presence of small, rare groups of examples,
decomposition of the minority class into rare sub-concepts, see [28]).
Therefore, in this paper we will present its extensions to address these
issues.

The processing pipeline is as follows: the original tabular data is fed to a
neural network (trained with the new proposed loss function) which transforms
it to a potentially easier embedded representation, produced by the last layer
of the network. Then, such preprocessed data (i.e. the constructed data
representation) is used to train the final classifier. Note that the method
modifies only the feature representation, combating the decline of
classification performance on imbalanced data by addressing local data
difficulty factors (i.e. producing safer representation) and not by modifying
the number of examples in particular classes.

We will use simple, feedforward neural networks – being sufficient for
processing tabular data, which are considered in this work. They consist of 3
layers with PReLU activations in between [14]. The numbers of units in each
layer vary between datasets as we tune these individually per dataset. We use
triplet loss (defined in Eq. 2) as the baseline loss function. To optimize the
objective, we use Adam optimizer.

Selecting triplets for the optimization of the loss function is challenging
since choosing all possible triplets is computationally infeasible. [32] used
an online sampling of triplets during the construction of mini-batches and
considering only such combinations of training examples that violate the
constraint from Eq. 1. As is still insufficient, the sampling of triplets is
performed online with mini-batches. Following inspirations from the
experiments of [32], we will experimentally study three selection strategies:

  1. 1.

selecting the hardest negative instance for each positive pair – select a
negative example that is closest from the anchor in the mini-batch;

  2. 2.

random hard negative for each positive pair – select one of the negatives that
gives positive triplet loss value (i.e. violates the constraint in Eq. (1));

  3. 3.

semi-hard negative for each positive pair – allow to select one of the
negatives that are further away from the anchor than the positive, but still
localized within the margin $\alpha$.

###  3.1 Weighted triplet loss with global imbalances

First, we propose a triplet weighting modification of Eq. 2 to better handle
the minority classes. For each anchor example, $x_{i}^{a}$ we multiply the
loss by inverse class cardinality normalized to 1 as shown in Eq. 3. As a
result, the examples from minority classes provide bigger gradients for the
neural network, than examples from the majority classes:

$\sum_{i}^{|\mathcal{T}|}\left[\left[\left\|f\left(x_{i}^{a}\right)-f\left(x_{i}^{p}\right)\right\|_{2}^{2}-\left\|f\left(x_{i}^{a}\right)-f\left(x_{i}^{n}\right)\right\|_{2}^{2}+\alpha\right]\times
w_{x_{i}^{a}}\right]_{+},$ (3)

where $\mathcal{T}$ is a set of all triplets, $w_{x_{i}^{a}}=\frac{|x\in
X:y_{x}=y_{x_{i}^{a}}|^{-1}}{\sum_{c_{i}}^{C}\left[\left|x\in
X:y_{x}=c_{i}\right|^{-1}\right]},$ $X$ denotes a set of all training
examples, $y_{x}$ is a class of example $x$, and $C$ is a set of all the
classes.

###  3.2 Dealing with the safe local neighborhood in loss functions

This modification takes into account global imbalance ratios present in the
data and, similarly to the original triplet loss, attempts to join examples
from the given class into a single cluster in the embedded representation. Now
we will also consider its variants oriented toward learning more complicated
data structures.

Firstly we want to clean the local neighborhood of examples. If the example
$x\in X$ is described in $d$-dimensional space $R^{d}$, $D$ is a distance
measure (e.q. Euclidean) and $KNN(x)$ denotes k nearest neighbors of $x$, then
the new loss function is defined as:

$min\sum_{x}\left[\sum_{x^{\prime}\in
KNN(x):y_{x}=y_{x^{\prime}}}D(x,x^{\prime})-\sum_{x^{\prime}\in
KNN(x):y_{x}\neq
y_{x^{\prime}}}min\left\\{D(x,x^{\prime})-\alpha,0\right\\}\right],$ (4)

$\alpha=\\{\begin{array}[]{rr}\max\limits_{x^{\prime}\in
KNN(x):y_{x}=y_{x^{\prime}}}D(x,x^{\prime})+1,&\text{if
}|KNN(x):y_{x}=y_{x^{\prime}}|>0\\\ 1,&\text{otherwise},\end{array}$ (5)

Intuitively, our objective (Eq. 4) is to minimize the sum of distances to
examples from the same class in the nearest neighborhood of $x$ and maximize
the sum of distances to examples from other classes in the nearest
neighborhood of $x$. Note that the maximization of distances to examples from
other classes does not go further than $\alpha$, which is dynamically set to
be greater than the distance to the farthest instance of the same class in the
neighborhood. Moreover, using $KNN(x)$ requires changing the algorithm of
constructing mini-batches: we select randomly $B$ examples from the training
set and for each of those we find their $k$ nearest neighbors, which
ultimately forms the mini-batch. The preliminary experiments indicated the
value of $k$ to be equal to 20. Batch size $B$ was equal to 16 or 32.

The next extensions are designed to exploit more information on the class
distribution inside $k$-neighborhood, similarly to earlier studies on local
safeness of the minority class examples [24, 28]. In the first extension of
the above objective, the loss value is additionally multiplied by a weight
$w_{x}$ dependent on the ratio of examples from the same class in the
neighborhood of the example $x$ (the less, the bigger is the weight). This is
to make the model less biased towards majority classes and, in particular,
focus its operation towards separating difficult (i.e. unsafe) examples:

$w_{x}=(1-\frac{|KNN(x):y_{x}=y_{x^{\prime}}|}{|KNN(x)|})$ (6)

In order to make the above loss more robust to outliers and reduce the chance
of overfitting, another proposed variant of the loss function uses the above
weight but with a cut-off below a certain threshold. Intuitively, after
reaching a certain small ratio of examples from the same class (e.g. 0.3) in
the neighborhood, we do not further increase the weight. The cutoff weight is
denoted as $c_{x}$ instead of $w_{x}$ \- see definition below.

$c_{x}=min\left\\{w_{x},0.7\right\\}$ (7)

Finally, instead of minimizing the sum of distances to examples in the
neighborhood in the loss function Eq. 4, we try minimizing (or maximizing) the
mean distance to examples from the same class (or accordingly other classes)
in the nearest neighborhood:

$min\sum_{x}\left[\frac{\sum_{x^{\prime}\in
KNN(x):y_{x}=y_{x^{\prime}}}D(x,x^{\prime})}{|KNN(x):y_{x}=y_{x^{\prime}}|}-\frac{\sum_{x^{\prime}\in
KNN(x):y_{x}\neq
y_{x^{\prime}}}min\left\\{D(x,x^{\prime})-\alpha,0\right\\}}{|KNN(x):y_{x}\neq
y_{x^{\prime}}|}\right]\times w_{x}$ (8)

As starting from a random representation and optimizing it according to any
loss function that uses KNN in the induced embedding space gives very poor
results, we decided to learn the initial representation with the autoencoder
and simple MSE loss. Then, we take the pre-trained encoder and further train
it using one of the four variants of the loss function specified above.

##  4 Experiments

###  4.1 An experimental setup

The aims of our experiments are the following: (1) to compare the newly
proposed weighted loss function vs. the original triplet loss; (2) to compare
the performance of classifiers trained on a new, triplet-based representation
versus the standard pre-processing methods for multiple imbalanced classes:
Global-CS [37], Static-SMOTE [10], MDO [1]; (3) to visualize the embedding
space of imbalanced data learned by triplet networks; (4) to evaluate
usefulness of different variants of the proposed triplet loss functions.

In all these experiments we use KNN, Decision Tree, and LDA as final
classifiers – implementations were taken from the scikit-learn library. LDA
(Linear Discriminant Analysis) is run with default parameters and KNN
classifier with k equal to 1. Decision Trees are parameterized with
min_samples_split=4, min_samples_leaf=2, and class_weight=’balanced’. For
Global-CS, Static-SMOTE or MDO, we used implementations provided by the multi
– imbalance library [12]. Before applying any preprocessing, classification,
or representation learning we performed one-hot encoding of the categorical
attributes (typical for processing in scikit learn) as well as z-score
standardization for numerical attributes.

All experiments are carried out with 17 popular multi-class imbalanced
benchmark datasets characterized by different levels of difficulty [25]. Due
to space limits, we skip the table with their detailed characteristics, the
reader can find their details e.g. in [18].

We decided to use F1-Score (macro-averaged) and G-Mean measures in their
versions for multiple classes due to their popularity in the literature (for
their definitions see [5, 3, 9]) and estimate their values with the stratified
5-fold cross-validation technique.

dataset | unweighted | weighted
---|---|---
cmc | 0.504 | 0.518
dermatology | 0.958 | 0.970
hayes-roth | 0.858 | 0.864
vehicle | 0.979 | 0.985
yeast | 0.383 | 0.294
art1 | 0.973 | 0.986
art2 | 0.779 | 0.806
art3 | 0.557 | 0.589
art4 | 0.860 | 0.880
balance-scale | 0.887 | 0.969
cleveland | 0.136 | 0.152
cleveland_v2 | 0.192 | 0.211
glass | 0.646 | 0.693
ecoli | 0.814 | 0.817
led7digit | 0.758 | 0.757
winequality-red | 0.445 | 0.472
thyroid | 0.944 | 0.951
Table 1: LDA G-Mean results comparison for representations obtained with
weighted and unweighted triplets

###  4.2 Studying weighted vs. unweighted triplets

In the first experiment, we compare the usefulness of the triplet weighting,
deined in Eq. 3 with the original, unweighted objective ( Eq. 2). For brevity,
we present the exact results only for the LDA classifier and G-mean measure in
Table 1 \- the best results are in bold.

An improvement of classification performance with the representation obtained
with triplet weighting (described in Eq. 3) is clearly visible. The
representation obtained through the original objective 2 provides better
results on the G-Mean only for two datasets: yeast and led7digit. Similar
observations are done for the F1 measure. We also compared all sampling
strategies for choosing triplets into the mini-batch in training a neural
network. Results showed that the differences between them are rather small.
Due to small computational costs, we stick to the random hard negative
strategy in the rest of experiments.

###  4.3 Comparison of pre-processing methods vs. new learned representation

Then we compare the performance of all three classifiers learned on the new
embedded representation (obtained with the weighted triplets - Eq. 2) against
using these classifiers with specialized preprocessing methods Global-CS,
Static-SMOTE and MDO (their description see [18]). Additionally, we will learn
these classifiers directly on the original, unchanged data - which is denoted
as Baseline. Again for brevity, we present the results only for one selected
classifier: KNN, see Tables 2 and 3, respectively.

dataset | Baseline | Global-CS | Static-SMOTE | MDO | New Rep.
---|---|---|---|---|---
cmc | 0.423 ± 0.015 | 0.409 ± 0.008 | 0.415 ± 0.004 | 0.415 ± 0.008 | 0.454 ± 0.008
dermatology | 0.937 ± 0.018 | 0.891 ± 0.021 | 0.910 ± 0.040 | 0.891 ± 0.020 | 0.970 ± 0.029
hayes-roth | 0.672 ± 0.113 | 0.817 ± 0.053 | 0.823 ± 0.059 | 0.834 ± 0.060 | 0.886 ± 0.042
vehicle | 0.912 ± 0.023 | 0.912 ± 0.023 | 0.909 ± 0.036 | 0.912 ± 0.023 | 0.989 ± 0.021
yeast | 0.487 ± 0.039 | 0.487 ± 0.039 | 0.458 ± 0.026 | 0.467 ± 0.036 | 0.497 ± 0.045
art1 | 0.946 ± 0.012 | 0.946 ± 0.012 | 0.950 ± 0.013 | 0.946 ± 0.017 | 0.974 ± 0.015
art2 | 0.700 ± 0.035 | 0.700 ± 0.035 | 0.702 ± 0.023 | 0.712 ± 0.012 | 0.751 ± 0.032
art3 | 0.509 ± 0.041 | 0.509 ± 0.041 | 0.479 ± 0.033 | 0.528 ± 0.049 | 0.572 ± 0.073
art4 | 0.768 ± 0.014 | 0.768 ± 0.014 | 0.769 ± 0.020 | 0.781 ± 0.034 | 0.817 ± 0.017
balance-scale | 0.561 ± 0.009 | 0.345 ± 0.044 | 0.304 ± 0.047 | 0.345 ± 0.041 | 0.954 ± 0.040
cleveland | 0.297 ± 0.042 | 0.317 ± 0.073 | 0.300 ± 0.043 | 0.328 ± 0.078 | 0.315 ± 0.105
cleveland_v2 | 0.327 ± 0.064 | 0.322 ± 0.037 | 0.332 ± 0.058 | 0.328 ± 0.025 | 0.306 ± 0.062
glass | 0.659 ± 0.065 | 0.659 ± 0.065 | 0.673 ± 0.071 | 0.666 ± 0.064 | 0.690 ± 0.080
ecoli | 0.793 ± 0.054 | 0.793 ± 0.054 | 0.776 ± 0.075 | 0.800 ± 0.062 | 0.814 ± 0.049
led7digit | 0.734 ± 0.035 | 0.706 ± 0.040 | 0.681 ± 0.044 | 0.706 ± 0.035 | 0.748 ± 0.048
winequality-red | 0.535 ± 0.029 | 0.535 ± 0.029 | 0.516 ± 0.038 | 0.537 ± 0.035 | 0.553 ± 0.055
thyroid | 0.930 ± 0.042 | 0.930 ± 0.042 | 0.937 ± 0.042 | 0.930 ± 0.042 | 0.957 ± 0.034
Table 2: Comparison of F1-Score for KNN learned on the original data (Baseline), after using pre-processing methods and for a new weighted triplet representation dataset | Baseline | Global-CS | Static-SMOTE | MDO | New Rep.
---|---|---|---|---|---
cmc | 0.413 ± 0.017 | 0.404 ± 0.008 | 0.408 ± 0.007 | 0.412 ± 0.009 | 0.448 ± 0.012
dermatology | 0.932 ± 0.024 | 0.892 ± 0.022 | 0.916 ± 0.035 | 0.892 ± 0.022 | 0.968 ± 0.031
hayes-roth | 0.636 ± 0.114 | 0.803 ± 0.059 | 0.812 ± 0.066 | 0.824 ± 0.065 | 0.881 ± 0.045
vehicle | 0.912 ± 0.018 | 0.912 ± 0.018 | 0.914 ± 0.032 | 0.912 ± 0.018 | 0.990 ± 0.021
yeast | 0.408 ± 0.088 | 0.408 ± 0.088 | 0.415 ± 0.080 | 0.364 ± 0.099 | 0.405 ± 0.099
art1 | 0.944 ± 0.012 | 0.944 ± 0.012 | 0.952 ± 0.019 | 0.966 ± 0.017 | 0.964 ± 0.029
art2 | 0.678 ± 0.052 | 0.678 ± 0.052 | 0.706 ± 0.027 | 0.728 ± 0.015 | 0.735 ± 0.033
art3 | 0.458 ± 0.049 | 0.458 ± 0.049 | 0.470 ± 0.046 | 0.526 ± 0.061 | 0.541 ± 0.104
art4 | 0.761 ± 0.020 | 0.761 ± 0.020 | 0.784 ± 0.024 | 0.802 ± 0.046 | 0.811 ± 0.029
balance-scale | 0.089 ± 0.001 | 0.064 ± 0.005 | 0.056 ± 0.006 | 0.065 ± 0.005 | 0.936 ± 0.065
cleveland | 0.078 ± 0.041 | 0.165 ± 0.158 | 0.121 ± 0.043 | 0.174 ± 0.166 | 0.180 ± 0.156
cleveland_v2 | 0.065 ± 0.042 | 0.048 ± 0.034 | 0.116 ± 0.044 | 0.049 ± 0.033 | 0.056 ± 0.035
glass | 0.554 ± 0.183 | 0.554 ± 0.183 | 0.589 ± 0.196 | 0.563 ± 0.186 | 0.573 ± 0.207
ecoli | 0.773 ± 0.058 | 0.773 ± 0.058 | 0.769 ± 0.075 | 0.799 ± 0.067 | 0.786 ± 0.067
led7digit | 0.648 ± 0.207 | 0.655 ± 0.051 | 0.647 ± 0.060 | 0.571 ± 0.182 | 0.699 ± 0.089
winequality-red | 0.475 ± 0.043 | 0.475 ± 0.043 | 0.488 ± 0.045 | 0.495 ± 0.048 | 0.453 ± 0.094
thyroid | 0.917 ± 0.080 | 0.917 ± 0.080 | 0.926 ± 0.080 | 0.917 ± 0.080 | 0.937 ± 0.061
Table 3: Comparison of G-mean for KNN learned on the original data (Baseline), after using pre-processing methods and for a new weighted triplet representation | New Rep. | MDO | Baseline | Static-SMOTE | Global-CS
---|---|---|---|---|---
KNN | 1.35 | 2.71 | 3.47 | 3.65 | 3.82
DT | 1.59 | 2.94 | 2.94 | 3.65 | 3.88
LDA | 1.41 | 3.59 | 3.82 | 3.65 | 2.53
Table 4: Average ranks of compared algorithms (the lower, the better) from Friedman tests on F1-Score for all classifiers. | New Rep. | MDO | Baseline | Static-SMOTE | Global-CS
---|---|---|---|---|---
KNN | 1.71 | 2.85 | 3.59 | 2.88 | 3.97
DT | 1.82 | 3.00 | 3.00 | 2.94 | 4.24
LDA | 1.59 | 3.65 | 4.24 | 3.53 | 2.00
Table 5: Average ranks of compared algorithms (the lower, the better) from
Friedman tests on G-Mean for various classifiers.

Results for both measures (also for Decision Trees and LDA) show that using
the new embedded representation leads to better classifier performance than
the compared pre-processing methods. We also carried out the Friedman ranked
test and rejected the null hypothesis of similar performance of compared
methods. The averaged ranks for all the classifiers are given in Tables 4 and
5 for F1 and G-mean , respectively. One can notice that our newly proposed
triplet representations are always better than the preprocessing methods,
while MDO or GlobalCS are in the second position. The post-hoc Nemenyi
analysis shows clear dominance for K-NN while for other classifiers the rank
differences are within CD. However, according to the paired Wilcoxon test the
new representation is significantly better for all classifiers in the case of
F1 score. Additionally, the performance of the proposed approach is also
statistically significantly better for KNN and DT with respect to G-mean (for
LDA $p$ = 0.9).

###  4.4 Visualization of the learned embeddings

To examine the changes in the example’s distribution inside the embedded
representation, we visualized them with PCA methods for two dimensions and
compared them to the analogous visualization of the distribution in the
original data. Below two representative datasets (easier and more difficult
ones) are presented in Figures 1 – 4.

One can notice that easier datasets such as vehicle – but also dermatology,
hayes-roth or thyroid, which we were unable to include due to page limits -
are easily transformed into single clusters of examples for each class. As
they become separable in the embedded representation it somehow solved the
overlapping in those datasets. Ecoli was a more complex data (see also its
analysis in [24]) and its separation in the embedded representation is not so
clear as before, but there is still an improvement in the differentiation of
examples from different classes. However, the situation is more difficult for
few harder datasets, such as glass or led7digit. For these datasets, the
method has difficulties in transforming the space into single clusters for
each class. Those datasets form much more complicated structures in the
original feature space (see experimental studies such as [25]), which makes
them very hard to transform by the triplet-based network using only weights
referring to the global imbalance. On the other hand the results on the
balance-scale – see Figures 5 and 6 – came as a surprise, because this dataset
seemed hard to classify with KNN, Decision Tree or LDA on original features,
but it was easily decomposed into single manifolds by the triplet-based
network. This may suggest, that non-linear feature transformations found by
the network can reduce difficulty factors even for some harder datasets.

Figure 1: Visualization of the original vehicle dataset in 2 PCA dimensions

Figure 2: Visualization of the triplet learned vehicle embeddings in 2 PCA
dimensions

Figure 3: Visualization of the original ecoli dataset in 2 PCA dimensions

Figure 4: Visualization of the triplet learned ecoli embeddings in 2 PCA
dimensions

Figure 5: Visualization of the original balance-scale dataset in 2 PCA
dimensions

Figure 6: Visualization of the triplet balance-scale embeddings in 2 PCA
dimensions

###  4.5 Experiments with new versions of triplet loss functions

dataset | Baseline | Autoencoder | Basic | Safe Weights | Cutoff | Mean Dists
---|---|---|---|---|---|---
cmc | 0.423 ± 0.015 | 0.403 ± 0.022 | 0.426 ± 0.014 | 0.430 ± 0.023 | 0.421 ± 0.011 | 0.436 ± 0.022
dermatology | 0.937 ± 0.018 | 0.920 ± 0.017 | 0.946 ± 0.030 | 0.949 ± 0.017 | 0.946 ± 0.020 | 0.953 ± 0.023
hayes-roth | 0.672 ± 0.113 | 0.807 ± 0.051 | 0.880 ± 0.054 | 0.896 ± 0.051 | 0.891 ± 0.045 | 0.891 ± 0.045
vehicle | 0.912 ± 0.023 | 0.918 ± 0.035 | 0.937 ± 0.010 | 0.975 ± 0.017 | 0.967 ± 0.022 | 0.986 ± 0.012
yeast | 0.487 ± 0.039 | 0.488 ± 0.040 | 0.488 ± 0.063 | 0.486 ± 0.050 | 0.487 ± 0.054 | 0.497 ± 0.052
art1 | 0.946 ± 0.012 | 0.945 ± 0.012 | 0.945 ± 0.017 | 0.949 ± 0.024 | 0.947 ± 0.016 | 0.955 ± 0.021
art2 | 0.700 ± 0.035 | 0.694 ± 0.024 | 0.714 ± 0.031 | 0.717 ± 0.026 | 0.711 ± 0.016 | 0.719 ± 0.038
art3 | 0.509 ± 0.041 | 0.521 ± 0.037 | 0.523 ± 0.033 | 0.523 ± 0.026 | 0.494 ± 0.061 | 0.524 ± 0.036
art4 | 0.768 ± 0.014 | 0.767 ± 0.016 | 0.778 ± 0.019 | 0.777 ± 0.028 | 0.765 ± 0.018 | 0.786 ± 0.018
balance-scale | 0.561 ± 0.009 | 0.401 ± 0.059 | 0.508 ± 0.028 | 0.839 ± 0.058 | 0.862 ± 0.054 | 0.942 ± 0.043
cleveland | 0.297 ± 0.042 | 0.321 ± 0.067 | 0.356 ± 0.083 | 0.319 ± 0.039 | 0.325 ± 0.034 | 0.335 ± 0.044
cleveland_v2 | 0.327 ± 0.064 | 0.310 ± 0.045 | 0.315 ± 0.025 | 0.314 ± 0.037 | 0.309 ± 0.015 | 0.342 ± 0.024
glass | 0.659 ± 0.065 | 0.642 ± 0.105 | 0.679 ± 0.035 | 0.690 ± 0.058 | 0.694 ± 0.056 | 0.699 ± 0.073
ecoli | 0.793 ± 0.054 | 0.776 ± 0.053 | 0.809 ± 0.048 | 0.789 ± 0.037 | 0.810 ± 0.036 | 0.802 ± 0.045
led7digit | 0.734 ± 0.035 | 0.708 ± 0.041 | 0.755 ± 0.076 | 0.748 ± 0.024 | 0.737 ± 0.079 | 0.744 ± 0.041
winequality-red | 0.535 ± 0.029 | 0.530 ± 0.027 | 0.536 ± 0.055 | 0.543 ± 0.030 | 0.542 ± 0.033 | 0.537 ± 0.027
thyroid | 0.930 ± 0.042 | 0.946 ± 0.052 | 0.952 ± 0.053 | 0.953 ± 0.038 | 0.953 ± 0.038 | 0.953 ± 0.038
Table 6: Comparison of F1-Score for KNN learned on the original data
(Baseline), and for new versions of loss functions

Finally, we tested all proposed new formulations of loss functions proposed in
Section 3 – see Eq. 4 – 8. Let us remind, that they are motivated by the
observation that data can form very complicated structures in the feature
space, which are not necessarily possible to represent as single clusters of
examples for each class by using the simpler modification of the triplet loss
function to the imbalance weighting - see also visualization results.

As mentioned before, we first train an autoencoder to minimize the MSE loss.
This is to avoid the collapse of training since the proposed loss functions
depend on kNN in the embedding space which is random at the beginning of
training. Therefore, after the autoencoder converges, we take the encoder
network and continue training under one of the loss functions specified in
Section 3. They are denoted as:

  * •

Basic – triplet loss guided to clean example’s neighborhood – Eq. 4.

  * •

Safeness Weights – basic with examples’ weights depending on their safe levels
as defined in Eq. 6.

  * •

Cutoff - another variant with safeness-based examples’ weights but with
additional cut-off at the threshold value of 0.7 as defined in Eq. 7.

  * •

Mean Dists - the last variant analysing the mean distances to the examples
from the same class as it is defined in Eq. 8.

In our experiments, we also calculated the results of classification performed
on the original representation (this is our Baseline) and on the
representation obtained from the hidden layer of autoencoder to see, how much
information is lost by such training initialization.

dataset | Baseline | Autoencoder | Basic | Safeness Weights | Cutoff | Mean Dists
---|---|---|---|---|---|---
cmc | 0.413 ± 0.017 | 0.398 ± 0.027 | 0.419 ± 0.015 | 0.424 ± 0.025 | 0.417 ± 0.014 | 0.429 ± 0.025
dermatology | 0.932 ± 0.024 | 0.927 ± 0.016 | 0.944 ± 0.027 | 0.950 ± 0.022 | 0.947 ± 0.023 | 0.953 ± 0.028
hayes-roth | 0.636 ± 0.114 | 0.793 ± 0.058 | 0.875 ± 0.058 | 0.892 ± 0.054 | 0.886 ± 0.048 | 0.886 ± 0.048
vehicle | 0.912 ± 0.018 | 0.918 ± 0.031 | 0.935 ± 0.012 | 0.976 ± 0.017 | 0.967 ± 0.023 | 0.986 ± 0.013
yeast | 0.408 ± 0.088 | 0.407 ± 0.086 | 0.376 ± 0.121 | 0.375 ± 0.110 | 0.375 ± 0.111 | 0.385 ± 0.114
art1 | 0.944 ± 0.012 | 0.942 ± 0.011 | 0.946 ± 0.017 | 0.952 ± 0.029 | 0.949 ± 0.019 | 0.956 ± 0.017
art2 | 0.678 ± 0.052 | 0.676 ± 0.033 | 0.693 ± 0.032 | 0.710 ± 0.033 | 0.695 ± 0.016 | 0.712 ± 0.047
art3 | 0.458 ± 0.049 | 0.474 ± 0.048 | 0.478 ± 0.031 | 0.478 ± 0.035 | 0.431 ± 0.062 | 0.479 ± 0.034
art4 | 0.761 ± 0.020 | 0.764 ± 0.022 | 0.767 ± 0.030 | 0.774 ± 0.032 | 0.766 ± 0.029 | 0.781 ± 0.016
balance-scale | 0.089 ± 0.001 | 0.071 ± 0.008 | 0.158 ± 0.166 | 0.827 ± 0.080 | 0.855 ± 0.076 | 0.958 ± 0.037
cleveland | 0.078 ± 0.041 | 0.129 ± 0.134 | 0.161 ± 0.145 | 0.141 ± 0.089 | 0.146 ± 0.093 | 0.194 ± 0.127
cleveland_v2 | 0.065 ± 0.042 | 0.049 ± 0.038 | 0.056 ± 0.030 | 0.069 ± 0.030 | 0.053 ± 0.028 | 0.071 ± 0.025
glass | 0.554 ± 0.183 | 0.490 ± 0.262 | 0.561 ± 0.185 | 0.577 ± 0.196 | 0.642 ± 0.064 | 0.591 ± 0.214
ecoli | 0.773 ± 0.058 | 0.739 ± 0.057 | 0.797 ± 0.060 | 0.772 ± 0.034 | 0.793 ± 0.037 | 0.774 ± 0.051
led7digit | 0.648 ± 0.207 | 0.667 ± 0.078 | 0.743 ± 0.218 | 0.736 ± 0.028 | 0.647 ± 0.231 | 0.735 ± 0.052
winequality-red | 0.475 ± 0.043 | 0.460 ± 0.028 | 0.468 ± 0.089 | 0.493 ± 0.036 | 0.492 ± 0.038 | 0.481 ± 0.041
thyroid | 0.917 ± 0.080 | 0.936 ± 0.069 | 0.937 ± 0.089 | 0.937 ± 0.058 | 0.937 ± 0.058 | 0.937 ± 0.058
Table 7: KNN G-Mean results comparison for different variants of the new loss
function

The results for k-NN classifiers (with $k=1$) are presented in Table 6. They
show that the proposed method with new loss functions outperforms Baseline
classification on original representation and also for the autoencoder
network. The comparison of different loss function variants clearly showed the
advantage of the most advanced version, i.e. Mean Dists (which wins in 12
datasets). Versions that use weighting (i.e. Safeness Weights and Cutoff) are
second in order of wins. A fairly similar dominance of the Mean Dists function
is also evident for the G-mean measure (see Table 7), although some advantages
occur for other datasets.

However, comparing this best version of the new loss function to the first
proposed function (weighted triplet as defined by Eq. 3) indicates a less
significant advantage. According to the paired Wilcoxon signed test, the $p$
values are equal to 0.029 for F1-Score and 0.589 for G-mean (i.e. here the
difference is not statistically significant). Looking more closely at the
results, the better performance of more advanced versions of the loss function
occurs for some datasets such as hayes-roth, balance scale, cleveland, glass,
ecoli, led7digit lub wine-quality. Following previous analyses of their
difficulties and complex distribution many of these datasets are identified as
more difficult to learn than dermatology, vehicles or artificial datasets art
[25].

##  5 Final remarks

We presented a new way to create an embedded representation for multi-class
imbalanced data that uses neural network training with specialized triplet
loss functions. It should be emphasized that our proposal only modifies the
distribution of features, i.e. the positions of examples in the newly learned
deep representation, and does not modify the class size or does not create any
synthetic examples (which is the most common for the currently most popular
under/oversampling processing methods).

In accordance with the inspirations of contrastive learning in deep networks,
we want to learn such a new representation of examples that their distribution
of classes in the new embedded space becomes more separated and safe in the
sense of the neighborhood of examples from a given class [28] and we do not
change the degree of imbalance in the dataset.

For this purpose, we introduced various definitions of triplet loss functions
used in training selected deep neural networks for tabular data. The first and
simplest proposal modifies the original triplet loss function [32] by
introducing weights related to the degree of class imbalance. As can be seen
from the conducted experiments (see, for example, visualizations in section
4.4), this can lead to learning disjoint clusters of objects from different
classes. Next, we put forward completely new proposals of triplet loss
functions which are intended for more complex distributions of examples, for
which it is necessary to take into account the local degrees of safeness of
the examples against their nearest neighbors.

Quite comprehensive experiments with popular benchmark multi-class imbalanced
data sets and three classifiers – K-NN, decision trees and linear discriminant
analysis – clearly showed the advantage of the proposed approach over the use
of popular pre-processing methods as well as basic versions of neural networks
with classic loss function formulations. However, it should be also noted that
for many of the analyzed datasets, the differences between the proposed loss
function variants were not statistically significant. Nevertheless, the most
advanced version, i.e. triplet loss with mean distances (Mean Dist) was by far
the winner of most comparisons, but for some datasets the weighted triplet
loss function (as defined by Eq. 3) worked better. For additional results
please refer to our external repository:
https://github.com/damian-horna/imbalanced_triplets.

Despite encouraging experimental results, we believe that in future research
it is still worth considering using and possibly extending the recent
modifications of the triplet functions. For instance, one can start by
investigating the loss functions considering the positions of four (inspired
by the work on other loss functions with four elements [7]) or even five
raining instances instead of triples [16], since they could potentially allow
for the creation of even more complicated representations for particularly
difficult imbalanced data distributions. Another line of research is to test
the applicability of the new triplet loss functions also for binary
classification. Preliminary ongoing experiments have already shown their
effectiveness.

Acknowledgments This work was partially supported by 0311/SBAD/0740. M. Lango
was supported by 0311/SBAD/0743.

## References

  * [1] Lida Abdi and Sattar Hashemi.  To combat multi-class imbalanced problems by means of over-sampling techniques.  IEEE Transactions on Knowledge and Data Engineering, 28:238–251, January 2016.
  * [2] C. Bellinger, R. Corizzo, and N. Japkowicz.  Calibrated resampling for imbalanced and long-tails in deep learning.  In Proceedings of Discovery Science Conference, pages 242–252. Springer International Publishing, 2021.
  * [3] Jerzy Błaszczyński, Jerzy Stefanowski, and Łukasz Idkowiak.  Extending bagging for imbalanced data.  In Proceedings of the 8th International Conference on Computer Recognition Systems CORES 2013, pages 269–278. Springer, 2013.
  * [4] P. Branco, L. Torgo, and R. Ribeiro.  A survey of predictive modeling under imbalanced distributions.  ACM Comput Surv, 49(2):31, 2016.
  * [5] Dariusz Brzezinski, Jerzy Stefanowski, Robert Susmaga, and Izabela Szczech.  Visual-based analysis of classification measures and their properties for class imbalanced problems.  Information Sciences, 462:242–261, 2018.
  * [6] M. Buda,, A. Maki,, and MA. Mazurkowski.  A systematic study of the class imbalance problem in convolutional neural networks.  Neural Networks, 106, 2018.
  * [7] W. Chen,, X. Chen,, J. Zhang,, and K. Huang.  Beyond triplet loss: a deep quadruplet network for person re-identification.  In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 403–412, 2017.
  * [8] Damien Dablain, Bartosz Krawczyk, and Nitesh V. Chawla.  Deepsmote: Fusing deep learning and smote for imbalanced data, 2021.
  * [9] Alberto Fernández, Salvador García, Mikel Galar, Ronaldo C. Prati, Bartosz Krawczyk, and Francisco Herrera.  Learning from Imbalanced Data Sets.  Springer, 2018.
  * [10] Francisco Fernández-Navarro, Francisco Hervás-Martínez, and Pedro Antonio Gutiérrez.  A dynamic over-sampling procedure based on sensitivity for multi-class problems.  Pattern Recognition, 44(8):1821–1833, 2011.
  * [11] Fares Grina, Zied Elouedi, and Eric Lefevre.  Evidential hybrid re-sampling for multi-class imbalanced data.  In Information Processing and Management of Uncertainty in Knowledge-Based Systems, pages 612–623. Springer International Publishing, 2022\.
  * [12] Jacek Grycza, Damian Horna, Hanna Klimczak, M. Lango, K. Plucinski, and J. Stefanowski.  multi-imbalance: Open source python toolbox for multi-class imbalanced classification.  In ECML/PKDD, pages 546–549, 2020.
  * [13] Guo Haixiang, Li Yijing, Jennifer Shang, Gu Mingyun, Huang Yuanyue, and Gong Bing.  Learning from class-imbalanced data: Review of methods and applications.  Expert Systems with Applications, 73:220–239, 2017.
  * [14] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  2015 IEEE International Conference on Computer Vision (ICCV), pages 1026–1034, 2015.
  * [15] Damian Horna, Mateusz Lango, and Jerzy Stefanowski.  Deep similarity learning loss functions in data transformation for class imbalance.  In LIDTA 2023 workshop ECMLPKDD 2023, Proceedings of Machine Learning, 2023.
  * [16] C. Huang, Y. Li, Chen Change Loy, and X. Tang.  Learning deep representation for imbalanced classification.  2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5375–5384, 2016.
  * [17] A. Jaishwal, A. R. Babu, L. Hall, M. Z. Zadeh, D. Banerjee, and F. Makedon.  A survey on contrastive self-supervised learning.  Technologies, 9(1), 2021.
  * [18] Małgorzata Janicka, Mateusz Lango, and Jerzy Stefanowski.  Using Information on Class Interrelations to Improve Classification of Multiclass Imbalanced Data: A New Resampling Algorithm.  Int. J. of Applied Mathematics and Computer Science, 29:769–781, 2019.
  * [19] Nathalie Japkowicz and Jerzy Stefanowski.  Big Data Analysis: New Algorithms for a New Society.  Springer: Studies in Big Data, 2016.
  * [20] Justin M Johnson and Taghi M Khoshgoftaar.  Survey on deep learning with class imbalance.  Journal of Big Data, 6(1):1–54, 2019.
  * [21] S. Khan, Munawar Hayat, M. Bennamoun, Ferdous Sohel, and R. Togneri.  Cost-sensitive learning of deep feature representations from imbalanced data.  IEEE Transactions on Neural Networks and Learning Systems, 29:3573–3587, 2018.
  * [22] Bartosz Krawczyk.  Learning from imbalanced data: open challenges and future directions.  Progress in Artificial Intelligence, 5(4):221–232, 2016.
  * [23] Mateusz Lango, Krystyna Napierala, and Jerzy Stefanowski.  Evaluating difficulty of multi-class imbalanced data.  In Proceedings of 23rd International Symposium Foundations of Intelligent Systems, ISMIS 2017, pages 312–322, 2017.
  * [24] Mateusz Lango, Krystyna Napierała, and Jerzy Stefanowski.  Evaluating Difficulty of Multi-class Imbalanced Data.  In Proceedings of 23rd International Symposium on Methodologies for Intelligent Systems, pages 312–322, 2017.
  * [25] Mateusz Lango and Jerzy Stefanowski.  What makes multi-class imbalanced problems difficult? an experimental study.  Expert Systems with Applications, 199:116962, 2022.
  * [26] Xiaoya Li, Xiaofei Sun, Yuxian Meng, Junjun Liang, Fei Wu, and Jiwei Li.  Dice loss for data-imbalanced NLP tasks.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 465–476, Online, July 2020. Association for Computational Linguistics.
  * [27] Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  CoRR, abs/1708.02002, 2017.
  * [28] Krystyna Napierała and Jerzy Stefanowski.  Types of Minority Class Examples and Their Influence on Learning Classifiers from Imbalanced Data.  Journal of Intelligent Information Systems, 46:563–597, 2016.
  * [29] José A Sáez, Bartosz Krawczyk, and Michał Woźniak.  Analyzing the oversampling of different classes and types of examples in multi-class imbalanced datasets.  Pattern Recognition, 57:164–178, 2016.
  * [30] Addisson Salazar, Luis Vergara, and Gonzalo Safont.  Generative adversarial networks and markov random fields for oversampling very small training sets.  Expert Systems with Applications, 163:113819, 2021.
  * [31] Miriam Seoane Santos, Pedro Henriques Abreu, Nathalie Japkowicz, Alberto Fernández, Carlos Soares, Szymon Wilk, and João Santos.  On the joint-effect of class imbalance and overlap: a critical review.  Artificial Intelligence Review, pages 1–69, 2022.
  * [32] Florian Schroff, Dmitry Kalenichenko, and James Philbin.  Facenet: A unified embedding for face recognition and clustering.  2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 815–823, 2015.
  * [33] Jerzy Stefanowski.  Dealing with data difficulty factors while learning from imbalanced data.  In J. Mielniczuk and S. Matwin, editors, Challenges in Computational Statistics and Data Mining, pages 333–363. Springer, 2016\.
  * [34] S. Wang, W. Liu, J. Wu, L. Cao, Q. Meng, and P.J. Kennedy.  Training deep neural networks on imbalanced data sets.  IEEE IJCNN, pages 4368–4374, 2016.
  * [35] Adam Wojciechowski and Mateusz Lango.  Adversarial oversampling for multi-class imbalanced data classification with convolutional neural networks.  In Proceedings of the Fourth International Workshop on Learning with Imbalanced Domains: Theory and Applications, volume 183 of Proceedings of Machine Learning Research, pages 98–111. PMLR, 2022.
  * [36] Q. Zhai,, R. Huang,, H. Cheng,, H. Zhan,, J. Li,, and Z. Liu.  Learning quintuplet loss for large-scale visual geo-localization.  IEEE MultiMedia, 27(3):34–43, 2020.
  * [37] Zhi-Hua Zhou and Xu-Ying Liu.  On multi-class cost-sensitive learning.  In Proceedings of the 21st National Conference on Artificial Intelligence - Volume 1, AAAI’06, pages 567–572, 2006.