# Bayesian Models of Functional Connectomics and Behavior

Niharika Shimona D’Souza
Department of Electrical and Computer Engineering
Johns Hopkins University
<EMAIL_ADDRESS>

###### Abstract

The problem of jointly analysing functional connectomics and behavioral data
is extremely challenging owing to the complex interactions between the two
domains. In addition, clinical rs-fMRI studies often have to contend with
limited samples, especially in the case of rare disorders. This data-starved
regimen can severely restrict the reliability of classical machine learning or
deep learning designed to predict behavior from connectivity data. In this
work, we approach this problem from the lens of representation learning and
bayesian modeling. To model the distributional characteristics of the domains,
we first examine the ability of approaches such as Bayesian Linear Regression,
Stochastic Search Variable Selection after performing a classical covariance
decomposition. Finally, we present a fully bayesian formulation for joint
representation learning and prediction. We present preliminary results on a
subset of a publicly available clinical rs-fMRI study on patients with Autism
Spectrum Disorder.

##  1 Introduction

Resting state fMRI (rs-fMRI) is a popular paradigm for assessing brain
activity and localize critical functions through steady state patterns of co-
activation 14. Network-based approaches to rs-fMRI analysis often group voxels
in the brain into regions of interest (ROIs) via a standard anatomical or
dervied functional atlas 17,18. From here, the synchrony between the regional
time courses can be summarized using a similarity matrix, which can be used as
input for further analysis. In the context of neuropsychiatric disorders such
as Autism, inter-patient variability often manifests as a spectrum of
impairments, that clinicians quantify as “behavioral score" of clinical
severity obtained from an exam. Identifying sub-networks in the brain that are
predictive of such severity can help us understand the social and behavioral
implications of the disorder for developing effective behavioral therapy.

Building predictive models at the patient level continues to remain as an open
challenge due to the high data dimensionality and considerable inter-subject
variation and noise in the resting state acquisition. From a frequentist
perspective, predictive models often follow a two step procedure. To combat
the data-dimensionality, feature selection is first applied to the raw
correlation values i.e. obtained by vectorizing the entries in the similarity
matrices. Examples approaches include graph theoretic measures
(betweenness,node degree), statistical and/or embedding features obtained from
representation learning techniques such as PCA, k-PCA or ICA 16. Next,
regression models such as Random Forests/ Support Vector Regression are
applied to the derived features to predict the clinical measures. These
strategies have shown success at modeling the group-averaged functional
connectivity across the cohort but often fail to accurately capture individual
variability. Consequently, the generalization power of these techniques is
limited 4,8 12

In an attempt to address these limitations, recent focus has shifted towards
mechanistic network models that are capable of modeling hierarchy onto
existing connectivity notions. For example, community detection techniques are
population-level models that are designed to identify interconnected subgraphs
within a larger network. These techiniques have refined our understanding of
the organization of complex systems such as brain networks 2,1. Extensions to
Bayesian community detection algorithms 24,22, 22, 21 have provided valuable
insights in characterizing the social and communicative deficits in
neurodevelopmental disorders such as schizophrenia and Autism. Unfortunately
all of the above focus on group characterizations, and even studies that
consider patient variability 15 or hierarchy in 6 have little generalization
power on new subjects.

The recent success of network decomposition models 3 in this space largely
based on their ability to simultaneously model the patient and group level
information. For example, the work of 13 introduces a common principal
components formulation, where multiple rank one matrix outer products
capturing the underlying ‘generative’ basis are combined using patient
specific coefficients. The sparse basis networks identify meaningful co-
activation patterns common to all the patients, and the coefficients model the
patient variability. Similar to the joint network optimization model in 8; 5,
this project explores the ‘discriminative’ nature of these coefficients.
Specifically, we estimate clinical severity of every subject first by
constructing bayesian regression models which map the subject-coefficients to
the behavioral domain once the decomposition is estimated, and then in an end-
to-end bayesian model. Through our experiments, we demonstrate the benefit of
this joint bayesian formulation in terms of capturing the variability in the
cohort, as well as for uncertainty quantification of the estimates.

We have organised this letter as follows 111This work was performed as a final
project for graduate level Bayesian Statistics course offered by the Applied
Mathematics and Statistics Department at Johns Hopkins University. We first
briefly describe the ASD dataset which we validate on. Next, our methods
section briefly introduces the dictionary decomposition to jointly model
group-averaged and patient-specific representations, along with the
corresponding inference algorithm. From here, we construct two bayesian
regression algorithms, the first of the vanilla variety and the second of the
variable selection (SVSS) flavour to predict clinical severity from the
subject specific coefficients and the estimation algorithms. We compare and
this performance to classical penalized linear regression. Finally, we propose
a joint heirarchical bayesian model that simultaneously infers the dictionary
representation and regression model parameters given the correlation matrices
and scores in an end-to-end fashion.

###  1.1 Dataset

##### rs-fMRI Dataset and Preprocessing.

We evaluate our method on a cohort of $52$ children with high-functioning ASD
released as a part of ABIDE 15 from the KKI site. Rs-fMRI preprocessing was
performed according to the prevalidated pipeline in 24 We use the Automatic
Anatomical Labeling (AAL) atlas to define $P=116$ cortical, subcortical and
cerebellar regions.

##### Clinical Scores.

We consider two measures of clinical severity: Autism Diagnostic Observation
Schedule (ADOS) total raw score 20, which captures the social and
communicative interaction deficits of the patient along with repetitive
behaviors (dynamic range: $0$-$30$), and the Social Responsiveness Scale (SRS)
total raw score 20 which characterizes social responsiveness (dynamic range:
$70$-$200$).

##  2 Methods

###  2.1 Dictionary Learning on rs-fMRI correlation matrices

Figure 1: Scree Plot for $\mathbf{\Gamma}_{n}$

We define $\mathbf{\Gamma}_{n}\in\mathcal{R}^{P\times P}$ as the correlation
matrix for patient $n$, where $P$ is the number of regions given by the
parcellation. We model $\mathbf{\Gamma}_{n}$ using a group average basis
representation and a patient-specific network strength term. The matrix
$\mathbf{B}\in\mathcal{R}^{P\times K}$ is a concatenation of $K$ elemental
bases vectors $\mathbf{b}_{k}\in\mathcal{R}^{P\times 1}$, i.e.
$\mathbf{B}:=\mathbf{b}_{1}\quad\mathbf{b}_{2}\quad...\quad\mathbf{b}_{K}$,
where $K\ll P$. These bases capture steady state patterns of co-activation
across regions in the brain. While the bases are common to all patients in the
cohort, the combination of these subnetworks is unique to each patient and is
captured by the non-negative coefficients $\mathbf{c}_{nk}$. We include a non-
negativity constraint $\mathbf{c}_{nk}\geq 0$ on the coefficients to preserve
the positive semi-definite structure of the correlation matrices
$\\{\mathbf{\Gamma}_{n}\\}$. The orthonormality constraint on $\mathbf{B}$
helps us learn uncorrelated sub-networks that explain the rs-fMRI data well
and implicitly regularize the optimization. Our complete rs-fMRI data
representation is:

$\mathbf{\Gamma}_{n}\approx\sum_{k}{\mathbf{c}_{nk}\mathbf{b}_{k}\mathbf{b}_{k}^{T}}\
\ \ s.t.\ \ \ \mathbf{c}_{nk}\geq 0\ \ \ \
\mathbf{B}^{T}\mathbf{B}=\mathcal{I}_{K}$ (1)

$\mathcal{I}_{K}$ is the $K\times K$ identity matrix. As seen in Eq. (1), we
model the heterogeneity in the cohort using a patient specific term in the
form of
$\mathbf{c}_{n}:=\mathbf{c}_{n1}\quad...\quad\mathbf{c}_{nK}^{T}\in\mathcal{R}^{K\times
1}$. Taking $\textbf{diag}(\mathbf{c}_{n})$ to be a diagonal matrix with the
$K$ patient coefficients on the diagonal and off-diagonal terms set to zero,
Eq. (1) can be re-written in matrix form as follows:

$\mathbf{\Gamma}_{n}\approx{\mathbf{B}\textbf{diag}({\mathbf{c}}_{n})\mathbf{B}^{T}}\
\ \ s.t.\ \ \ \mathbf{c}_{nk}\geq 0$ (2)

Overall, this formulation is similar to common principal components from the
statistics and manifold learning literature. Essentially, this strategically
reduces the high dimensionality of the data, while providing a patient level
description of the correlation matrices. We choose $K=15$ based on the
eigenspectrum on $\\{\mathbf{\Gamma}_{n}\\}$ (See Fig. 1).

####  2.1.1 Optimization

We use alternating minimization to optimize Eq. (2) with respect to
$\mathbf{B},\\{\mathbf{c}_{n}\\}$. Here, we cycle through the updates for the
dictionary $\mathbf{B}$, and loadings $\\{\mathbf{c}_{n}\\}$, to obtain a
joint solution.

We note that there is a closed-form Procrustes solution for quadratic
objectives. However, Eq. (2) is bi-quadratic in $\mathbf{B}$, so it cannot be
directly applied. Therefore, we adopt the strategy in 8, by which we introduce
the constraints of the form
$\mathbf{D}_{n}=\mathbf{B}\mathbf{diag}(\mathbf{c}_{n})$, with corresponding
augmented Lagrangian variables $\\{\mathbf{\Lambda}_{n}\\}$. Thus, our
objective from Eq. (2) now becomes:

$\mathcal{J}_{c}=\sum_{n}{||{\mathbf{\Gamma}_{n}-\mathbf{D}_{n}\mathbf{B}^{T}}||}^{2}_{F}+\sum_{n}{\Big{[}{\text{Tr}{\left[{(\mathbf{\Lambda}_{n})^{T}({\mathbf{D}_{n}-\mathbf{B}\mathbf{diag}(\mathbf{c}_{n})})}\right]}}+{{\frac{1}{2}}~{}{||{\mathbf{D}_{n}-\mathbf{B}\mathbf{diag}(\mathbf{c}_{n})}||}_{F}^{2}}}\Big{]}$
(3)

along with the constraints $\mathbf{c}_{nk}\geq 0$ and
$\mathbf{B}^{T}\mathbf{B}=\mathcal{I}_{K}$. See Algorithm 1

Result: Dictionary $\mathbf{B}$ and patient-specific coefficients
$\\{\mathbf{c}_{n}\\}$

Initialize $\mathbf{B}^{0}$ as the top eigenvectors of the mean of
$\\{\Gamma_{n}\\}$;

while  _Not converged_ do

Step 1: Compute quadratic programming solution for $\\{\mathbf{c}_{n}\\}$;

Step 2: Compute Procrustes solution for $\mathbf{B}$;

Step 3: Compute Augmented Lagrangian updates for
$\\{\mathbf{D}_{n},\mathbf{\Lambda}_{n}\\}$;

end while

Algorithm 1 Dictionary Learning on rs-fMRI correlation matrices

###  2.2 Bayesian Regression Models

We combine this representation learning with a bayesian regression models to
map to clinical severity. Our first set of models consider two classes of
Bayesian Regression frameworks to predict behavior from the coefficients
$\\{\mathbf{c}_{n}\\}$.

####  2.2.1 Bayesian Linear Regression (BLR)

Let $\mathbf{y}_{n}$ be the scalar behavioral severity scores for a patient n.
We model each
$\mathbf{y}_{n}=\mathbf{\beta}_{0}+\mathbf{\beta}^{T}\mathbf{c}_{n}+\epsilon_{n}$,
where $\epsilon_{n}\sim\mathcal{N}(0,\sigma^{2})$. In this model, we consider
that the samples are drawn iid given $\mathbf{c}_{n}$. Our likelihood function
is parametrized by $\mathbf{\beta}\in\mathcal{R}^{K\times 1}$ and takes the
form:

$\displaystyle\ell(\\{\mathbf{y}_{n}\\}|\\{\mathbf{c}_{n}\\},\mathbf{\beta},\mathbf{\beta}_{0},\sigma^{2})=\prod_{n=1}^{N}{\ell(\mathbf{y}_{n}|\mathbf{c}_{n},\mathbf{\beta},\mathbf{\beta}_{0},\sigma^{2})}=\prod_{n=1}^{N}{\mathcal{N}(\mathbf{y}_{n};\mathbf{\beta}^{T}\mathbf{c}_{n},\mathbf{\beta}_{0},\sigma^{2})}$
(4)

We impose a conjugate prior on $(\mathbf{\beta},\sigma^{2})$ of the normal
inverse-gamma form as follows:

$\displaystyle
P(\mathbf{\beta};\beta_{0}|\sigma^{2})=\mathcal{N}(\mathbf{M},\sigma^{2}\mathbf{V})\
\ \text{and}\ \ \sigma^{2}\sim\text{IG}(a,b)$

We set $\mathbf{M}=\mathbf{0}\in\mathcal{R}^{(K+1)\times 1}$ and $a=3,b=1$ as
mild assumptions on the prior. For our experiments, we apriori assume that the
entries in $\mathbf{\beta}$ are uncorrelated, i.e.
$\mathbf{V}=\sigma_{\beta}^{2}\cdot\mathcal{I}_{K+1}$.

For our experiments, we consider different values of $\sigma^{2}_{\beta}$ to
determine the model with the best fit as a grid search. We use a standard
Gibbs Sampling algorithm (Implemented using Matlab’s econometrics toolbox) to
generate pairs of samples from the posterior
$\mathbf{\beta},\mathbf{\beta}_{0},\sigma^{2}|\\{\mathbf{c}_{n},\mathbf{y}_{n}\\}$
as follows:

  * 1

Initialize $\mathbf{\beta},\mathbf{\beta}_{0},\sigma^{2}$

  * 2

Sample
$\mathbf{\beta};\mathbf{\beta}_{0}|\\{\mathbf{c}_{n},\mathbf{y}_{n}\\},\sigma^{2}\sim\mathcal{N}(\mu_{n},\mathbf{\Sigma}_{y})$
where
$\mathbf{\mu}_{n}=(\hat{\mathbf{C}}\hat{\mathbf{C}}^{T}+\sigma^{2}\mathbf{V})^{-1}(\hat{\mathbf{C}}\hat{\mathbf{C}}^{T}\hat{\mathbf{\beta}})$
and
$\mathbf{\Sigma}_{y}=(\hat{\mathbf{C}}\hat{\mathbf{C}}^{T}+\sigma^{2}\mathbf{V})$,
$\hat{\mathbf{C}}=\mathbf{1};\mathbf{C}$

  * 3

Sample
$\sigma^{2}|\\{\mathbf{c}_{n},\mathbf{y}_{n}\\},\mathbf{\beta};\mathbf{\beta}_{0}\sim\text{IG}\Big{(}a+{n}/{2},b_{0}+(\sum{\mathbf{y}^{2}_{n}}-\mathbf{\mu}_{n}^{T}\mathbf{\Sigma}_{y}\mathbf{\mu}_{n})/{2}\Big{)}$

  * 4

After a burn-in, we keep the rest of the samples as generated from the
posterior

####  2.2.2 Bayesian Stochastic Search Variable Selection (SVSS)

The goal of variable selection over the linear regression model in Eq. (4) is
to only include only those predictors supported by data in the final
regression model. However, analysing $2^{(K+1)}$ permutations of models is
computationally inefficient. Instead, Stochastic Variable Selection looks at
this problem from a Bayesian perspective. Here, if we wish to exclude a
coefficient from a model, we assign it a degenerate posterior distribution
that approximates a Dirac-Delta. Practically, if a coefficient is to be
included, we draw the coefficient from an
$\mathbf{\beta}_{k}\sim\mathcal{N}(\mathbf{0},\mathbf{V}_{1k})$, else
$\mathbf{\beta}_{k}\sim\mathcal{N}(\mathbf{0},\mathbf{V}_{2k})$, where
$\mathbf{V}_{2k}$ is small relative to $\mathbf{V}_{1k}$. We represent
inclusion vs exclusion via a binary variable $\gamma_{k}$, $k=\\{0,\dots
K\\}$. Thus the sample space $\gamma_{k}$ has cardinality $2^{(K+1)}$, and the
coefficients $\mathbf{\beta}_{0},\dots\mathbf{\beta}_{K}$ are independent
apriori. Our likelihood model takes the form:

$\displaystyle\ell(\\{\mathbf{y}_{n}\\}|\\{\mathbf{c}_{n}\\},\mathbf{\beta},\mathbf{\beta}_{0},\sigma^{2})=\prod_{n=1}^{N}{\mathcal{N}(\mathbf{y}_{n};\mathbf{\beta}^{T}\mathbf{c}_{n},\mathbf{\beta}_{0},\sigma^{2})}$
$\displaystyle\text{If}\ \
\mathbf{\gamma}_{k}=1,\mathbf{\beta}_{k}\sim\mathcal{N}(0,\sigma^{2}\mathbf{V}_{1k})$
$\displaystyle\text{If}\ \
\mathbf{\gamma}_{k}=0,\mathbf{\beta}_{k}\sim\mathcal{N}(0,\sigma^{2}\mathbf{V}_{2k})$

Given $\mathbf{\beta}_{k}$, $\gamma_{k}$ is conditionally independent of the
data. Therefore, the full conditional posterior distribution of the
probability that the variable $k$ is included in the model.

$\displaystyle
P(\mathbf{\gamma}_{k}|\mathbf{\beta}_{0},\mathbf{\beta},\sigma^{2},\mathbf{\gamma}_{\neq
k})\propto\mathbf{g}_{k}\mathcal{N}(\mathbf{\beta}_{k};0,\sigma^{2}\mathbf{V}_{1k})$
$\displaystyle\mathbf{\gamma}_{k}\sim\text{Bernoulli}(\mathbf{g}_{k})$

We impose a conjugate prior structure on
$\mathbf{\beta}_{0};\mathbf{\beta},\\{\mathbf{\gamma}\\},\sigma^{2}$ of the
following form:

$\displaystyle
p(\mathbf{\beta};\mathbf{\beta}_{0},\mathbf{\gamma},\sigma^{2})=p(\sigma^{2})\prod^{K}_{k=0}p(\beta_{k}|\gamma_{k})p(\gamma_{k})$
$\displaystyle\mathbf{\beta}_{k}|\mathbf{\gamma}_{k}\sim\mathbb{I}(\gamma_{k}=0)\mathcal{N}(0,\sigma^{2}\mathbf{V}_{2k})+\mathbb{I}(\gamma_{k}=1)\mathcal{N}(0,\sigma^{2}\mathbf{V}_{1k})$
$\displaystyle\sigma^{2}\sim\text{IG}(a,b)$

Again, we use a diffuse prior with $a=3,b=1$ and $\mathbf{g}_{k}=0.5$. For our
experiments, we consider different values of
$\\{\mathbf{V}_{1k},\mathbf{V}_{2k}\\}$ to determine the model with the best
fit as a grid search. We use a standard Gibbs Sampling algorithm (Implemented
using Matlab’s econometrics toolbox) to generate pairs of samples from the
posterior. Although a closed-form posterior exists for conjugate mixture
priors, since the prior $\\{\mathbf{\beta}\\}|\sigma^{2},\\{\gamma_{k}\\}$ is
marginalized by $\gamma$, this implementation uses MCMC to sample from the
joint posterior
$\mathbf{\beta},\mathbf{\beta}_{0},\\{\mathbf{\gamma}_{k}\\}|\\{\mathbf{c}_{n},\mathbf{y}_{n}\\},\sigma^{2}$.
For both of these methods, we run the chains for $5000$ samples as burn in and
generate $10000$ additional samples to approximate the posterior.

Figure 2: (T) Histogram for ADOS (L) true samples, (M) posterior from BLR (R)
posterior from SVSS (B) Subnetworks from SVSS. (L) Visual & Subcortical (R)
Default Mode Network Figure 3: Prior and Posterior Densities for
$\mathbf{\beta},\beta_{0}$ by Bayesian Linear Regression on ADOS. The blue
line represents the posterior, while the dotted red line is the prior

####  2.2.3 Evaluation and Results

We evaluate the generalization performance of the model using a five fold
cross validation like strategy. In each fold, eighty percent of the examples
are used for estimation, and the rest of the twenty percent as forecasting.
First, we vary the free parameters $\sigma_{\beta}^{2}$ for BLR and
$\\{\mathbf{V}_{1k},\mathbf{V}_{2k}\\}$ for SVSS and determine the parameters
that fit the estimation points the best. We then evaluate the performance on
the held-out forecasting examples. Table 1 compares the performance based on
the root mean square error (rMSE) and normalized mutual information (NMI)
metric, between the predicted and true samples. SVSS performs better than BLR
for both scores. As a baseline, we also report the performance of a classical
ridge regression on $\mathbf{c}_{n}$ to predict $\mathbf{y}_{n}$ as well.
Using bayesian approaches, we obtain more than a point estimate of each
$\mathbf{y}_{n}$ as in frequentist methods. We can also quantify the
uncertainty of the estimate around the a posteriori maximum as summary
statistics.

Table 1: Performance evaluation using root Mean Squared Error (rMSE) & Normalized Mutual Information (MI). Lower MAE & higher MI indicate better performance. Score | Method | rMSE Train | rMSE Test | MI Test
---|---|---|---|---
ADOS | BLR | 2.76 $\scriptstyle\pm$ 0.27 | 4.15 $\scriptstyle\pm$ 0.26 | 0.48
SVSS | 2.51 $\scriptstyle\pm$ 0.45 | 3.80 $\scriptstyle\pm$ 0.37 | 0.56
Ridge Regression | 2.70 $\scriptstyle\pm$ 2.32 | 3.35 $\scriptstyle\pm$ 2.11 | 0.41
SRS | BLR | 34.33 $\scriptstyle\pm$ 5.44 | 29.23 $\scriptstyle\pm$ 5.81 | 0.71
SVSS | 25.23 $\scriptstyle\pm$ 5.10 | 27.99 $\scriptstyle\pm$ 4.99 | 0.76
Ridge Regression | 19.29 $\scriptstyle\pm$ 11.11 | 24.44 $\scriptstyle\pm$ 18.18 | 0.66

In Fig. 2, we compare the histograms of the true samples against the samples
generated by one run of the BLM and SVSS models for the ADOS score, according
to the parameters selected above. The overlap with the true distribution is
indicative of how well the models are able to approximate the data generating
process. Fig. 3 compares the prior and posterior densities for the
coefficients $\mathbf{\beta},\beta_{0}$ learned for BLR. Finally, SVSS allows
us to isolate the coefficients $\beta_{k}$ which are consistently selected as
predictors. For both ADOS and SRS, $\\{\mathbf{c}_{n,7},\mathbf{c}_{n,8}\\}$
are the features that are selected most often (have the highest magnitude of
$\beta_{k}>90\%$ of the times). We plot the corresponding subnetworks
$\mathbf{B}_{k}$ for $k=7,8$ in Fig. 2. Thus, SVSS offers us interpretability
in terms of the features relevant to prediction. Altered connectivity in these
networks, both default mode and in visual processing areas has been found to
be associated with ASD previously 23.

Figure 4: Hierarchical Bayesian Model for Joint Representation Learning and
Prediction

###  2.3 A Bayesian Model for Joint Representation Learning and Prediction

In the previous two approaches, the feature extraction is combined with
prediction in a pipelined fashion, decoupling the two. However, inherently,
the two views of data are complementary to each other. In this section, we
propose to use a bayesian model which mimics this representation learning
step. At the same time, the patient-specific coefficients relate to prediction
via a bayesian linear regression. By combining the dictionary learning
directly with prediction, we expect to learn a joint rs-fMRI representation
that is more aligned with clinical prediction, similar to the principles in 8

Recall that the correlation matrices $\\{\mathbf{\Gamma}_{n}\\}$ are positive
semi-definite. Recall that we use the common principal components
decomposition
$\mathbf{\Gamma}_{n}\approx\mathbf{B}\mathbf{diag}(\mathbf{c}_{n})\mathbf{B}^{T}$
Additionally, we center $\mathbf{y}_{n}$ to have zero mean. Accordingly, our
data likelihood uses an Inverse Wishart distribution ($\mathbf{\Phi}_{W}$) on
$\mathbf{\Gamma}_{n}$ centered around
$\mathbf{B}\mathbf{diag}(\mathbf{c}_{n})\mathbf{B}^{T}$ with degrees of
freedom $\nu_{0}$. We chose $\nu_{0}=P+5$ to center
$\\{\mathbf{\Gamma}_{n}\\}$ loosely around
$\mathbf{B}\mathbf{diag}(\mathbf{c}_{n})\mathbf{B}^{T}$. Again, we predict the
clinical scores via a linear regression
$\mathbf{y}_{n}\approx\mathbf{c}^{T}_{n}\mathbf{w}$ with the linear regression
weights $\mathbf{w}\in\mathcal{R}^{K\times 1}$. Let
$\theta=(\sigma_{y}^{2},\sigma_{c}^{2},\sigma_{w}^{2})$. Given
$\\{\mathbf{w},\\{\mathbf{c}_{n}\\}\\}$, we assume that $\mathbf{y}_{n}$ are
independent, while given $\\{\mathbf{B},\\{\mathbf{c}_{n}\\}\\}$,
$\mathbf{\Gamma}_{n}$ are independent. Thus, if $Q=\nu_{0}-P-1$:

$\displaystyle\ell(\\{\mathbf{\Gamma}_{n},\mathbf{y}_{n}\\}|\\{\mathbf{c}_{n}\\},\mathbf{w},\mathbf{B},\theta)=\prod_{n=1}^{N}{\mathcal{N}(\mathbf{y}_{n};\mathbf{c}_{n}^{T}\mathbf{w},\mathbf{\sigma}_{y}^{2})}\
\mathbf{\Phi}_{W}\Bigg{[}\mathbf{\Gamma}_{n};\frac{\mathbf{B}\mathbf{diag}(\mathbf{c}_{n})\mathbf{B}^{T}}{Q},\nu_{0}\Bigg{]}$
(5)

We apriori assume that
$\mathbf{\Theta}=\\{\mathbf{B},\\{\mathbf{c}_{n}\\},\mathbf{w}\\}$ are
independent given $\mathbf{\theta}$. Therefore:

$\displaystyle
P(\mathbf{\Theta},\mathbf{\theta})=P(\sigma^{2}_{w})P(\sigma^{2}_{c})P(\sigma^{2}_{y})P(\mathbf{w}|\sigma^{2}_{w})\prod_{k=1}^{K}P(\mathbf{b}_{k})\prod_{n=1}^{N}P(\mathbf{c}_{n}|\sigma_{c}^{2})$
(6)

To approximate a basis $\mathbf{B}$ that is almost orthogonal [], we use a
multivariate normal prior of the form:

$\displaystyle
P(\mathbf{b}_{k})=\mathcal{N}(\mathbf{0},\sigma_{B}^{2}\mathcal{I}_{P})\ \ \
\text{s.t.}\ \ \sigma_{B}^{2}=\frac{1}{P}$ (7)

We use a conjugate multivariate normal-inverse gamma prior on
$\\{\mathbf{w},\sigma^{2}_{w}\\}$ and a half multivariate normal-inverse gamma
prior (for non-negativity) on $\\{\mathbf{c},\sigma^{2}_{c}\\}$ and an inverse
gamma prior on $\sigma_{y}^{2}$:

$\displaystyle
P(\mathbf{w},\sigma_{w}^{2})=P(\mathbf{w}|\sigma_{w}^{2})P(\sigma_{w}^{2})=\mathcal{N}(\mathbf{0},\sigma_{w}^{2}\mathcal{I}_{K})\text{IG}(\sigma_{w}^{2};a_{w},b_{w})$
(8) $\displaystyle
P(\mathbf{C},\sigma_{c}^{2})=P(\sigma_{c}^{2})\prod_{n=1}^{N}P(\mathbf{c}_{n}|\sigma_{c}^{2})=\text{IG}(\sigma_{c}^{2};a_{c},b_{c})\prod^{N}_{n=1}\mathcal{N}(\mathbf{0},\sigma_{c}^{2}\mathcal{I}_{K})$
(9) $\displaystyle P(\sigma^{2}_{y})=\text{IG}(\sigma_{y}^{2};a_{y},b_{y})$
(10)

Notice that the complete posterior distributions of
$\mathbf{w},\sigma_{w}^{2},\sigma_{y}^{2},\sigma_{c}^{2}$ can be derived in
closed form owing to the structure of the prior:

$\displaystyle\mathbf{w}|\mathbf{B},\\{\mathbf{c}_{n},\mathbf{\theta}\\}\sim\mathcal{N}(\mathbf{w};\mathbf{\mu}_{w},\mathbf{\Sigma}_{w})\
\ \text{s.t.}\ \
\mathbf{\Sigma}_{w}=\Bigg{[}\frac{\mathcal{I}_{K}}{\sigma_{w}^{2}}+\frac{\sum_{n}{\mathbf{c}_{n}\mathbf{c}^{T}_{n}}}{{\sigma_{y}^{2}}}\Bigg{]}^{-1}\
\ \text{and}\ \
\mathbf{\mu}_{n}=\mathbf{\Sigma}_{w}\frac{\sum_{n}{\mathbf{c}^{2}_{n}}}{\sigma_{y}^{2}}$
(11)
$\displaystyle\sigma^{2}_{w}|\mathbf{w},\mathbf{B},\\{\mathbf{c}_{n}\\},\sigma^{2}_{c},\sigma^{2}_{y}\sim\text{IG}\Bigg{(}a_{w}+\frac{K}{2},b_{w}+\frac{\sum_{k}\mathbf{w}^{2}_{k}}{2}\Bigg{)}$
(12)
$\displaystyle\sigma^{2}_{c}|\mathbf{w},\mathbf{B},\\{\mathbf{c}_{n}\\},\sigma^{2}_{w},\sigma^{2}_{y}\sim\text{IG}\Bigg{(}a_{c}+\frac{NK}{2},b_{c}+\frac{\sum_{n}\sum_{k}\mathbf{c}^{2}_{nk}}{2}\Bigg{)}$
(13)
$\displaystyle\sigma^{2}_{y}|\mathbf{w},\mathbf{B},\\{\mathbf{c}_{n}\\},\sigma^{2}_{c},\sigma^{2}_{w}\sim\text{IG}\Bigg{(}a_{y}+\frac{N}{2},b_{y}+\frac{\sum_{n}(\mathbf{c}^{T}_{n}\mathbf{w}-\mathbf{y}_{n})^{2}}{2}\Bigg{)}$
(14)

Thus, our inference algorithm performs a Gibbs-MH sampling based on the full
conditionals for these variables and random-walk like proposal distributions
to sample $\mathbf{B},\\{\mathbf{c}_{n}\\}$. Our inference algorithm is
summarised below (Algorithm 2).

Result: Posterior samples for
$\\{\mathbf{B},\\{\mathbf{c}_{n}\\},\mathbf{w},\mathbf{\theta}\\}$

Initialize $\mathbf{B}^{0},\\{\mathbf{c}^{0}_{n}\\},\mathbf{w}^{0},\theta^{0}$
, $a_{c}=a_{w}=a_{y}=3,b_{c}=b_{w}=b_{y}=1$

while  _Not converged_ do

Step 1: Sample $\mathbf{B}^{t}\sim q(\cdot|\mathbf{B}^{t-1})$ ; Determine
whether to accept-reject samples

Step 2: Sample $\mathbf{C}^{t}\sim q(\cdot|\mathbf{C}^{t-1})$ ; Determine
whether to accept-reject samples

Step 3: Sample $\mathbf{w}$ according to Eqn. (11);

Step 4: Sample $\sigma^{2}_{w}$ according to Eqn. (12);

Step 5: Sample $\sigma^{2}_{c}$ according to Eqn. (13);

Step 6: Sample $\sigma^{2}_{y}$ according to Eqn (14);

end while

Algorithm 2 Gibbs-MH Sampling for the joint model

Figure 5:  Histogram of (L) True samples and (R) Samples from Alg. 2, i.e
$\mathbf{c}^{T}_{n}\mathbf{w}$ (T): SRS (B): ADOS

###  2.4 Implementation Details and Preliminary Results

We implement Alg. 2 in $R$ on an $8$-core machine with an Intel $i7$ processor
($16$GB RAM). The approximate run time is about 10 hours to generate $10000$
samples. $8000$ of these were treated as burn-in. We experimented with several
proposal distributions and found that a normal around the previous sample with
a small variance to be the most stable (with acceptance ratio $0.21$ and
$0.22$ for the two chains respectively) and concurs with best practices 25.
Also, we fold samples $\mathbf{C}^{t}$ to maintain non-negativity.

We first examine the convergence of the chains for
$\sigma^{2}_{w},\sigma^{2}_{c},\sigma^{2}_{y}$ via the trace plots and
autocorrelation in Fig. 6. Note that examining the convergence of the other
latent variables is a less straightforward exercise. We observed that
$\sigma^{2}_{c}$ has the slowest mixing of these with high autocorrelations
between samples, even after running the chains for very long. Additionally, we
compare the posterior samples generated by our model i.e.
$\hat{\mathbf{y}}_{n}=\mathbf{c}^{T}_{n}\mathbf{w}$ against the distribution
of the true scores $\mathbf{y}_{n}$ for both scores. The overlap gives us a
sense of how well the generating process is approximated. We obtain an rMSE of
$3.78~{}\raisebox{0.86108pt}{$\scriptstyle\pm$}~{}{2.51}$ for ADOS and
$19.51~{}\raisebox{0.86108pt}{$\scriptstyle\pm$}~{}{7.51}$ for SRS when using
all the samples, which is higher than those obtained in Table 1. Additionally,
as a sanity check, we plot the inner product measure the columns of
$\mathbf{B}$ for a representative sample (Fig. 7 (a)). Indeed, we see that our
chains provide uncorrelated and nearly orthogonal bases. Finally, we plot side
by side a correlation matrix sample and the corresponding mean approximation
error over samples from the chain (Fig. 7 (b)). We notice that while a large
number of regions have relatively small approximation errors, the model has
trouble determining the prominent patterns along the band diagonal. One of the
reasons may be the scale of $\mathbf{B}$ and $\mathbf{C}$, which is difficult
to simultaneously control in a random walk as can be seen with the trace plot
for $\sigma^{2}_{c}$.

Figure 6: (T) Trace Plots and (B) Autocorrelation Plots for (L)
$\sigma_{w}^{2}$ (M) $\sigma_{y}^{2}$ (R)$~{}\sigma_{c}^{2}$ where
$\mathbf{y}_{n}$ is the SRS score (Top Set) ADOS score (Bottom Set)  Figure 7:
(a) Inner Product between columns of a representative sample of $\mathbf{B}$
(b) Comparison between a true correlation matrix sample (left), and its mean
approximation error (right)

##  3 Discussion

In this letter, we first examined the efficacy of bayesian regression models
coupled with dictionary learning to predict clinical severity from rs-fMRI
correlation data. Of these, the stochastic variable selection generalized best
and offered us with an approach to recognise features most relevant to
clinical outcomes. Next, inspired by previous results, we took a fully
bayesian approach to jointly learn an rs-fMRI representation and regression
model. From the modeling standpoint, this framework presented several design
challenges. For example, models selection in terms of order $K$, prior
parameters, eg. $(\nu_{0},a,b)$, currently chosen ad-hoc, and convergence
(diagnostics/designing good proposal distributions) of the sampling procedure.
Currently, our sampling procedure is computationally expensive and more work
needs to be done to ensure that the proposals scan the latent parameter space
better and avoid getting stuck in local models. Finally, another challenge is
the inherent non-identifiability of the model (for example, scaling-
$\\{\alpha\mathbf{B},\frac{1}{\alpha^{2}}\mathbf{C}\\}$,
$\\{\delta{C},\frac{1}{\delta}\mathbf{w}\\}$, rotations of $\mathbf{B}$),
which contributes to convergence issues when the prior parameters are
incorrectly chosen.

Future Directions An immediate future direction could be the extension of the
framework to incorporate multimodal structural and functional connectivity
data such as 11; 9; 7 for behavioral prediction. Another interesting extension
of the framework could be towards dynamic modeling of functional connectivity
10; 19; 1 as it evolves over the scan. Overall, this preliminary analysis is a
first step at exploring the nascent potential of joint bayesian modeling for
brain connectivity and behavior.

## References

  * Andersen et al. 2014 Andersen Kasper Winther, Madsen Kristoffer H, Siebner Hartwig Roman, Schmidt Mikkel N, Mørup Morten, Hansen Lars Kai.  Non-parametric Bayesian graph models reveal community structure in resting state fMRI  // NeuroImage. 2014. 100. 301–315.
  * Bardella et al. 2016 Bardella Giampiero, Bifone Angelo, Gabrielli Andrea, Gozzi Alessandro, Squartini Tiziano.  Hierarchical organization of functional connectivity in the mouse brain: a complex network approach  // Scientific reports. 2016\. 6. 32060.
  * Batmanghelich et al. 2012 Batmanghelich Nematollah K, Taskar Ben, Davatzikos Christos.  Generative-discriminative basis learning for medical imaging  // IEEE transactions on medical imaging. 2012. 31, 1. 51–69.
  * D’Souza, others 2021 D’Souza Niharika S, others .  Blending generative models with deep learning for multidimensional phenotypic prediction from brain connectivity data. 2021.
  * DSouza et al. 2018 DSouza Niharika Shimona, Nebel Mary Beth, Wymbs Nicholas, Mostofsky Stewart, Venkataraman Archana.  A Generative-Discriminative Basis Learning Framework to Predict Clinical Severity from Resting State Functional MRI Data  // International Conference on Medical Image Computing and Computer-Assisted Intervention. 2018. 163–171.
  * Dai et al. 2017 Dai Tian, Guo Ying, Initiative Alzheimer’s Disease Neuroimaging, others .  Predicting individual brain functional connectivity using a Bayesian hierarchical model  // NeuroImage. 2017. 147. 772–787.
  * Dsouza et al. 2021 Dsouza Niharika Shimona, Nebel Mary Beth, Crocetti Deana, Robinson Joshua, Mostofsky Stewart, Venkataraman Archana.  M-gcn: A multimodal graph convolutional network to integrate functional and structural connectomics data to predict multidimensional phenotypic characterizations  // Medical Imaging with Deep Learning. 2021. 119–130.
  * D’Souza et al. 2020a D’Souza NS, Nebel MB, Wymbs N, Mostofsky SH, Venkataraman A.  A joint network optimization framework to predict clinical severity from resting state functional MRI data  // NeuroImage. 2020a. 206. 116314.
  * D’Souza et al. 2021a D’Souza Niharika Shimona, Nebel Mary Beth, Crocetti Deana, Robinson Joshua, Mostofsky Stewart, Venkataraman Archana.  A Matrix Autoencoder Framework to Align the Functional and Structural Connectivity Manifolds as Guided by Behavioral Phenotypes  // International Conference on Medical Image Computing and Computer-Assisted Intervention. 2021a. 625–636.
  * D’Souza et al. 2021b D’Souza Niharika Shimona, Nebel Mary Beth, Crocetti Deana, Robinson Joshua, Wymbs N, Mostofsky Stewart H, Venkataraman Archana.  Deep sr-DDL: Deep structurally regularized dynamic dictionary learning to integrate multimodal and dynamic functional connectomics data for multidimensional clinical characterizations  // NeuroImage. 2021b. 241. 118388.
  * D’Souza et al. 2020b D’Souza Niharika Shimona, Nebel Mary Beth, Crocetti Deana, Wymbs Nicholas, Robinson Joshua, Mostofsky Stewart, Venkataraman Archana.  A deep-generative hybrid model to integrate multimodal and dynamic connectivity for predicting spectrum-level deficits in autism  // International Conference on Medical Image Computing and Computer-Assisted Intervention. 2020b. 437–447.
  * D’Souza et al. 2019 D’Souza Niharika Shimona, Nebel Mary Beth, Wymbs Nicholas, Mostofsky Stewart, Venkataraman Archana.  A coupled manifold optimization framework to jointly model the functional connectomics and behavioral data spaces  // International Conference on Information Processing in Medical Imaging. 2019. 605–616.
  * Eavani et al. 2015 Eavani Harini, Satterthwaite Theodore D, Filipovych Roman, Gur Raquel E, Gur Ruben C, Davatzikos Christos.  Identifying sparse connectivity patterns in the brain using resting-state fMRI  // Neuroimage. 2015. 105. 286–299.
  * Fox, Raichle 2007 Fox Michael D, Raichle Marcus E.  Spontaneous fluctuations in brain activity observed with functional magnetic resonance imaging  // Nature reviews neuroscience. 2007. 8, 9. 700.
  * Heinsfeld et al. 2018 Heinsfeld Anibal Sólon, Franco Alexandre Rosa, Craddock R Cameron, Buchweitz Augusto, Meneguzzi Felipe.  Identification of autism spectrum disorder using deep learning and the ABIDE dataset  // NeuroImage: Clinical. 2018. 17. 16–23.
  * Khosla et al. 2019 Khosla Meenakshi, Jamison Keith, Ngo Gia H, Kuceyeski Amy, Sabuncu Mert R.  Machine learning in resting-state fMRI analysis  // Magnetic resonance imaging. 2019. 64. 101–121.
  * Nandakumar et al. 2018a Nandakumar Naresh, D’Souza Niharika S, Craley Jeff, Manzoor Komal, Pillai Jay J, Gujar Sachin K, Sair Haris I, Venkataraman Archana.  Defining patient specific functional parcellations in lesional cohorts via Markov random fields  // International Workshop on Connectomics in Neuroimaging. 2018a. 88–98.
  * Nandakumar et al. 2018b Nandakumar Naresh, D’Souza Niharika Shimona, Sair Haris, Venkataraman Archana.  A MODIFIED K-MEANS ALGORITHM FOR RESTING STATE FMRI ANALYSIS OF BRAIN TUMOR PATIENTS, AS VALIDATED BY LANGUAGE LOCALIZATION. 2018b.
  * Nandakumar et al. 2020 Nandakumar Naresh, D’souza Niharika Shimona, Manzoor Komal, Pillai Jay J, Gujar Sachin K, Sair Haris I, Venkataraman Archana.  A multi-task deep learning framework to localize the eloquent cortex in brain tumor patients using dynamic functional connectivity  // Machine Learning in Clinical Neuroimaging and Radiogenomics in Neuro-oncology. 2020. 34–44.
  * Payakachat, others 2012 Payakachat Nalin, others .  Autism spectrum disorders: a review of measures for clinical, health services and cost–effectiveness applications  // Expert review of pharmacoeconomics & outcomes research. 2012. 12, 4. 485–503.
  * Venkataraman et al. 2015 Venkataraman Archana, Duncan James S, Yang Daniel Y-J, Pelphrey Kevin A.  An unbiased Bayesian approach to functional connectomics implicates social-communication networks in autism  // NeuroImage: Clinical. 2015. 8. 356–366.
  * Venkataraman et al. 2013 Venkataraman Archana, Kubicki Marek, Golland Polina.  From connectivity models to region labels: identifying foci of a neurological disorder  // IEEE transactions on medical imaging. 2013. 32, 11. 2078–2098.
  * Venkataraman et al. 2011 Venkataraman Archana, Rathi Yogesh, Kubicki Marek, Westin Carl-Fredrik, Golland Polina.  Joint modeling of anatomical and functional connectivity for population studies  // IEEE transactions on medical imaging. 2011. 31, 2. 164–182.
  * Venkataraman et al. 2016 Venkataraman Archana, Yang Daniel Y-J, Pelphrey Kevin A, Duncan James S.  Bayesian community detection in the space of group-level functional differences  // IEEE transactions on medical imaging. 2016\. 35, 8. 1866–1882.
  * Yildirim 2012 Yildirim Ilker.  Bayesian inference: Metropolis-hastings sampling  // Dept. of Brain and Cognitive Sciences, Univ. of Rochester, Rochester, NY. 2012.


# A Multilingual Information Extraction Pipeline
for Investigative Journalism

Gregor Wiedemann Seid Muhie Yimam Chris Biemann
Language Technology Group
Department of Informatics
Universität Hamburg, Germany
{gwiedemann, yimam<EMAIL_ADDRESS>
###### Abstract

We introduce an advanced information extraction pipeline to automatically
process very large collections of unstructured textual data for the purpose of
investigative journalism. The pipeline serves as a new input processor for the
upcoming major release of our _New/s/leak 2.0_ software, which we develop in
cooperation with a large German news organization. The use case is that
journalists receive a large collection of files up to several Gigabytes
containing unknown contents. Collections may originate either from official
disclosures of documents, e.g. Freedom of Information Act requests, or
unofficial data leaks. Our software prepares a visually-aided exploration of
the collection to quickly learn about potential stories contained in the data.
It is based on the automatic extraction of entities and their co-occurrence in
documents. In contrast to comparable projects, we focus on the following three
major requirements particularly serving the use case of investigative
journalism in cross-border collaborations: 1) composition of multiple state-
of-the-art NLP tools for entity extraction, 2) support of multi-lingual
document sets up to 40 languages, 3) fast and easy-to-use extraction of full-
text, metadata and entities from various file formats.

##  1 Support Investigative Journalism

Journalists usually build up their stories around entities of interest such as
persons, organizations, companies, events, and locations in combination with
the complex relations they have. This is especially true for investigative
journalism which, in the digital age, more and more is confronted to find such
relations between entities in large, unstructured and heterogeneous data
sources.

Usually, this data is buried in unstructured texts, for instance from scanned
and OCR-ed documents, letter correspondences, emails or protocols. Sources
typically range from 1) official disclosures of administrative and business
documents, 2) court-ordered revelation of internal communication, 3) answers
to requests based on Freedom of Information (FoI) acts, and 4) unofficial
leaks of confidential information. Well-known examples of such disclosed or
leaked datasets are the _Enron_ email dataset Keila and Skillicorn (2005) or
the _Panama Papers_ O’Donovan et al. (2016).

To support investigative journalism in their work, we have developed
_New/s/leak_ Yimam et al. (2016), a software implemented by experts from
natural language processing and visualization in computer science in
cooperation with journalists from _Der Spiegel_ , a large German news
organization. Due to its successful application in the investigative research
as well as continued feedback from academia, we further extend the
functionality of _New/s/leak_ , which now incorporates better pre-processing,
information extraction and deployment features. The new version _New/s/leak
2.0_ serves four central requirements that have not been addressed by the
first version or other existing solutions for investigative and forensic text
analysis:

Improved NLP processing: We use stable and robust state-of-the-art natural
language processing (NLP) to automatically extract valuable information for
journalistic research. Our pipeline combines extraction of temporal entities,
named entities, key-terms, regular expression patterns (e.g. URLs, emails,
phone numbers) and user-defined dictionaries.
Multilingualism: Many tools only work for English documents or a few other
‘big languages’. In the new version, our tool allows for automatic language
detection and information extraction in 40 different languages. Support of
multilingual collections and documents is specifically useful to foster cross-
country collaboration in journalism.
Multiple file formats: Extracting text and metadata from various file formats
can be a daunting task, especially in journalism where time is a very scarce
resource. In our architecture, we include a powerful data wrangling software
to automatize this process as much as possible. We further put emphasis on
scalability in our pipeline to be able to process very large datasets. For
easy deployment, _New/s/leak 2.0_ is distributed as a Docker setup.
Keyword graphs: We have implemented keyword network graphs, which is build
based on the set of keywords representing the current document selection. The
keyword network enables to further improve the investigation process by
displaying entity networks related to the keywords.

##  2 Related Work

There are already a handful of commercial and open-source software products to
support investigative journalism. Many of the existing tools such as
OpenRefine111http://openrefine.org, Datawrapper222https://www.datawrapper.de,
Tabula333http://tabula.technology, or Sisense444https://www.sisense.com focus
solemnly on structured data and most of them are not freely available. For
unstructured text data, there are costly products for forensic text analysis
such as Intella555https://www.vound-software.com. Targeted user groups are
national intelligence agencies. For smaller publishing houses, acquiring a
license for those products is simply not possible. Since we also follow the
main idea of openness and freedom of information, we concentrate on other
open-source products to compare our software to.

DocumentCloud666https://www.documentcloud.org is an open-source tool
specifically designed for journalists to analyze, annotate and publish
findings from textual data. In addition to full-text search, it offers named
entity recognition (NER) based on OpenCalais777http://www.opencalais.com for
person and location names. In addition to automatic NER for multiple
languages, our pipeline supports the identification of keyterms as well as
temporal and user-defined entities.

Overview Brehmer et al. (2014) is another open-source application developed by
computer scientists in collaboration with journalists to support investigative
journalism. The application supports import of PDF, MS Office, and HTML
documents, document clustering based on topic similarity, a simple location
entity detection, full-text search, and document tagging. Since this tool is
already mature and has successfully been used in a number of published news
stories, we adapted some of its most useful features such as document tagging,
full-text search and a keyword-in-context (KWIC) view for search hits.

The Jigsaw visual analytics Görg et al. (2014) system is a third tool that
supports analyzing and understanding of textual documents. The Jigsaw system
focuses on the extraction of entities using Gate tool suite for NLP Cunningham
et al. (2013). Hence, support for multiple languages is somewhat limited. It
also lacks sophisticated data import mechanisms.

The new version of _New/s/leak_ was built targeting these drawbacks and
challenges. With _New/s/leak 2.0_ we aim to support the journalist throughout
the entire process of collaboratively analyzing large, complex and
heterogeneous document collections: data cleaning and formatting, metadata
extraction, information extraction, interactive filtering, visualization,
close reading and tagging, and providing provenance information.

##  3 Architecture

Figure 1 shows the overall architecture of _New/s/leak_. In order to allow
users to analyze a wide range of document types, our system includes a
document processing pipeline, which extracts text and metadata from a variety
of document types into a unified representation. On this unified text
representation, a number of NLP pre-processing tasks are performed as a UIMA
pipeline Ferrucci and Lally (2004), e.g. automatic identification of the
document language, segmentation into paragraph, sentence and token units, and
extraction of named entities, keywords and metadata. ElasticSearch is used to
store the processed data and create aggregation queries for different entity
types to generate network graphs. The user interface is implemented with a
RESTful web service based on the Scala Play framework in combination with an
AngularJS browser app to present information to the journalists.
Visualizations are realized with D3 Bostock et al. (2011).

Figure 1: Architecture of _New/s/leak 2.0_

In order to enable a seamless deployment of the tool by journalists with
limited technical skills, we have integrated all of the required components of
the architecture into a Docker888https://www.docker.com setup. Via docker-
compose, a software to orchestrate Docker containers for complex
architectures, end-users can download and run locally a preconfigured version
of _New/s/leak_ with one single command. Being able to process data locally
and even without any connection to the internet is a vital prerequisite for
journalists when they work with sensitive data. All necessary source code and
installation instructions can be found on our Github project
page.999https://uhh-lt.github.io/newsleak-frontend

##  4 Data Wrangling

Extracting text and metadata from various formats into a format readable by a
specific analysis tool can be a tedious task. In an investigative journalism
scenario, it can even be a deal breaker since time is an especially scarce
resource and file format conversion might not be a task journalists are well
trained in. To offer access to as many file formats as possible in
_New/s/leak_ , we opted for a close integration with _Hoover_
,101010https://hoover.github.io a set of open-source tools for text extraction
and search in large text collections. Hoover is developed by the European
Investigative Collaborations (EIC) network with a special focus on large data
leaks and heterogeneous datasets. It can extract data from various text file
formats such as txt, html, docx, pdf, zip, tar, pst, mbox, eml, etc. The text
is extracted along with metadata from files (e.g. file name, creation date,
file hash) and header information (e.g. subject, sender, and receiver). In the
case of emails, attachments are processed automatically, too.

_New/s/leak_ connects directly to Hoover’s index to read full-texts and
metadata for its own information extraction pipeline. Through this close
integration with Hoover, _New/s/leak_ can offer information extraction to a
wide variety of data formats. In many cases, this drastically limits or even
completely eliminates the amount of work needed to clean and preprocess large
datasets beforehand.

##  5 Multilingual Information Extraction

The core functionality of _New/s/leak_ is the automatic extraction of various
kinds of entities from text to facilitate the exploration and sense-making
process from large collections. Since a lot of steps in this process involve
language-dependent resources, we put an emphasis on the work to support as
many languages as possible.

###  5.1 Preprocessing

Information extraction in _New/s/leak_ is implemented as a configurable UIMA
pipeline Ferrucci and Lally (2004). Text documents and metadata from a Hoover
collection (see Section 4) are read in parallelized manner and put through a
chain of annotators. In a final step of the chain, results from annotation
processes are indexed in an ElasticSearch index for later retrieval and
visualization.

First, we identify the language of each document. Alternatively, language can
also be determined on a paragraph level to support multi-language documents,
which can occur quite often, for instance in email leaks or bilingual
contracts. Second, we separate sentences and tokens in each text. To guarantee
compatibility with various Unicode scripts in different languages, we rely on
the ICU4J library111111http://icu-project.org/apiref/icu4j for this task.
ICU4J provides locale-specific sentence and word boundary detection relying on
a simple rule-based approach. While the quality of the segmentation and
tokenization results might be better when using specifically trained
segmentation models, the advantage of the rule-based approach in ICU4J is that
it works robustly not only for many languages but also for noisy data, which
we expect to be abundant in real-life datasets.

###  5.2 Dictionaries and RE-patterns

In many cases, journalists follow some hypothesis to test for their
investigative work. Such a proceeding can involve looking for mentions of
already known terms or specific entities in the data. This can be realized by
lists of dictionaries provided to the initial information extraction process.
_New/s/leak_ annotates every mention of a dictionary term with the respective
list type. Dictionaries can be defined in a language-specific fashion, but
also applied across documents of all languages in the corpus. Extracted
dictionary entities are displayed along with extracted named entities in the
visualization.

In addition to self-defined dictionaries, we annotate email addresses,
telephone numbers, and URLs with regular expression patterns. This is useful,
especially for email leaks to reveal communication networks of persons and
filter for specific email account related content.

###  5.3 Temporal Expressions

Tracking documents across the time of their creation or by temporal events
they mention can provide valuable information during investigative research.
Unfortunately, many document sets (e.g. collections of scanned pages) do not
come with a specific document creation date as structured metadata. To offer a
temporal selection of contents to the user, we extract mentions of temporal
expressions. This is done by integrating the HeidelTime temporal tagger
Strötgen and Gertz (2015) in our UIMA workflow. HeidelTime provides
automatically learned rules for temporal tagging in more than 200 languages.
Extracted timestamps can be used to select and filter documents.

###  5.4 Named Entity Recognition

We automatically extract person, organization and location names from all
documents to allow for an entity-centric exploration of the data collection.
Named entity recognition is done using the _polyglot-NER_ library Al-Rfou et
al. (2015). Polyglot-NER contains sequence classification for named entities
based on weakly annotated training data automatically composed from
Wikipedia121212https://wikipedia.org and
Freebase131313https://developers.google.com/freebase. Relying on the automatic
composition of training data allows polyglot-NER to provide pre-trained models
for 40 languages.141414A list of the 40 languages covered by Polyglot-NER can
be found at https://tinyurl.com/yaju7bf7

###  5.5 Keyterm Extraction

To further summarize document contents in addition to named entities, we
automatically extract keyterms and phrases from documents. For this, we
implemented a keyterm extraction library for the 40 languages also supported
in the previous step.151515https://github.com/uhh-lt/lt-keyterms Our approach
is based on a statistical comparison of document contents with generic
reference data. Reference data for each language is retrieved from the Leipzig
Corpora Collection Goldhahn et al. (2012), which provides large representative
corpora for language statistics. We employ log-likelihood significance as
described in Rayson et al. (2004) to measure the overuse of terms (i.e.
keyterms) in our target documents compared to the generic reference data.
Ongoing sequences of keyterms in target documents are concatenated to key
phrases if they occur regularly in that exact same order. Regularity is
determined with the Dice coefficient. This simple method allows to reliably
extract multiword units such as “stock market” or “machine learning” in the
documents. Since this method also extracts named entities if they occur
significantly often in a document, there can be a substantial overlap between
both types. To allow for a separate display of named entities and keywords, we
filter keyterms if they already have been annotated as a named entity. The
remaining top keyterms are used to create a brief summary of each document for
the user and to generate keyterm networks for document browsing.

##  6 User Interface

#### Browsing entity networks:

Access to unstructured text collections via named entities is essential for
journalistic investigations. To support this, we included two types of graph
visualization, as it is shown in Figure 2. The first graph, called entity
network, displays entities in a current document selection as nodes and their
joint occurrence as edges between nodes. Different node colors represent
different types such as person, organization or location names. Furthermore,
mentions of entities that are annotated based on dictionary lists are included
in the entity network graph. The second graph, called keyword network, is
build based on the set of keywords representing the current document
selection. The keyword network also includes tags that can be attached to
documents by journalists during work with the collection.

Figure 2: The entity and keyword graphs of _New/s/leak_ based on the WW2
collection (see Section 7). Networks are visualized based on the current
document selection, which can be filtered by full-text search, entities or
metadata. Visualization parameters such as the number of nodes per type or
minimum edge strength can be set by the user. Hovering over nodes and edges in
one graph highlights information present in the respective another graph to
show which entities and keywords frequently co-occur with each other in
documents.

#### Journalist in the loop:

In addition to the automatic annotation of entities and keyterms, we further
enable journalists to: 1) annotate new entity types that are not in the system
at all, 2) correct automatic annotations provided by the pipeline, e.g. to
remove false positives or false entity type labels annotated by the NER
process, 3) merge identical entities which have different forms (e.g. last
names to full names, or spelling variants in different languages), and 4)
label documents with user-defined terms called _tags_. The tags are mainly
used to annotate the document either for later reading or to share with
collaborators.

##  7 Case Study

To illustrate analysis capabilities of the new version of _New/s/leak_ , we
present an exemplary case study at https://ltdemos.informatik.uni-
hamburg.de/newsleak/ (login with ”user” and ”password”). Since we refrain from
publishing any confidential leak data, we created an artificial dataset from
publicly available documents that share certain characteristics with the data
from intended use cases in investigative journalism. It contains documents
written in multiple languages, roughly centered on one topic and is full of
references to entities.

Ca. 27.000 documents in our sample set are Wikipedia articles related to the
topic of World War II. Articles were crawled from the encyclopedia in four
languages (English:en, Spanish:es, Hungarian:hu, German:de) as a link network
starting from the article ”Second World War” in each respective language.
Preprocessing and data extraction took around 75 minutes on a moderately fast
server with 12 parallel CPU threads.

#### Analysis:

From a perspective of national history discourses and education, a certain
common knowledge about WW2 can be expected. But, the topic becomes quickly a
novel unexplored terrain for most people when it comes to aspects outside of
the own region, e.g. the involvement of Asian powers. In our test case, we
strive to fill gaps in our knowledge by identifying interesting details
regarding this question. First, we start with a visualization of entities from
the entire collection which highlights central actors of WW2 in general. In
the list of extracted location entities, we can filter for ca. 2,000 articles
referencing to Asia (en, es), Ázsia (hu) or Asien (de). In this subselection,
we find most references to China as a political power of the region followed
by India and Japan. Further refinement of the collection by references to
China highlights a central person name in the network, Chiang Kai-shek, who
raises our interest. To find out more, we start the filter process all over
again, subselecting all articles referencing this name. The resulting entity
network reveals a close connection to the organization Kuomintang (KMT).
Filtering for this organization, too, we can quickly identify articles
centrally referencing to both entities by looking at their titles and
extracted keywords. From the corresponding keyterm network and a KWIC view
into the article full-texts, we learn that KMT is the national party of China
and Kai-Shek as their leader ruled the country during the period of WW2. A
second central actor, Mao Zedong, is strongly connected with both, KMT and
Chiang Kai-shek in our entity network. From articles also prominently
referencing Zedong, we learn from sections highlighting both person names that
Kai-shek and Zedong, also a member of KMT and later leader of the Chinese
Communists, shared a complicated relationship. By filtering for both names, we
can now explore the nature of this relationship in more detail and compare its
display across the four languages in our dataset.161616A video of the
proceeding can be found at: http://youtu.be/96f_4Wm5BoU

##  8 Discussion and Future Work

In this paper, we introduced the completely renewed information extraction
pipeline of _New/s/leak 2.0_ , an open-source software to support
investigative journalism. As major requirements based on prior experiences, we
identified the automatic annotation of various entity types in very large,
multi-lingual document sets contained in heterogeneous file formats. Our
solution involves a combination of powerful NLP libraries for temporal and
named entities, own developments for keyterm and pattern extraction, and a
powerful data wrangling suite for text and metadata extraction. The pipeline
is capable to process information extraction in 40 languages.

_New/s/leak_ has been in use successfully at the German news organization Der
Spiegel. It recently has also been introduced as an open-source tool to the
community of investigative journalists at respective conferences. We expect to
collect more user feedback and experiences from case studies in the near
future to further improve the software.

As a new main feature, we plan to extend the information extraction pipeline
for user-defined categories into the direction of adaptive and active machine
learning approaches. Currently, while reading the full-texts, users can
manually annotate new entity types in the text or tag the entire documents. In
combination with an adaptive and active learning approach, users will be able
to train automatic tagging of documents and extraction of information while
working with the data in the user interface.

## References

  * Al-Rfou et al. (2015) Rami Al-Rfou, Vivek Kulkarni, Bryan Perozzi, and Steven Skiena. 2015.  Polyglot-NER: Massive Multilingual Named Entity Recognition.  In _Proc. SIAM/ICDM-2015_ , pages 586–594.
  * Bostock et al. (2011) Michael Bostock, Vadim Ogievetsky, and Jeffrey Heer. 2011.  D3 Data-Driven Documents.  _IEEE Trans. Visualization & Comp. Graphics_, 17(12):2301–2309.
  * Brehmer et al. (2014) M. Brehmer, S. Ingram, J. Stray, and T. Munzner. 2014.  Overview: The Design, Adoption, and Analysis of a Visual Document Mining Tool for Investigative Journalists.  _IEEE Trans. Visualization & Comp. Graphics_, 20(12):2271–2280.
  * Cunningham et al. (2013) Hamish Cunningham, Valentin Tablan, Angus Roberts, and Kalina Bontcheva. 2013.  Getting more out of biomedical documents with GATE’s full lifecycle open source text analytics.  _PLoS computational biology_ , 9(2):e1002854.
  * Ferrucci and Lally (2004) David Ferrucci and Adam Lally. 2004.  UIMA: An architectural approach to unstructured information processing in the corporate research environment.  _Natural Language Engineering_ , 10(3-4):327–348.
  * Goldhahn et al. (2012) Dirk Goldhahn, Thomas Eckart, and Uwe Quasthoff. 2012.  Building large monolingual dictionaries at the Leipzig corpora collection: From 100 to 200 languages.  In _Proc. LREC 2012_ , pages 759–547.
  * Görg et al. (2014) Carsten Görg, Zhicheng Liu, and John Stasko. 2014.  Reflections on the evolution of the Jigsaw visual analytics system.  _Information Visualization_ , 13(4):336–345.
  * Keila and Skillicorn (2005) P. S. Keila and D. B. Skillicorn. 2005.  Detecting unusual email communication.  In _Proc. CASCON 2005_ , pages 117–125.
  * O’Donovan et al. (2016) James O’Donovan, Hannes F. Wagner, and Stefan Zeume. 2016.  The value of offshore secrets evidence from the panama papers.  _SSRN Electronic Journal_.
  * Rayson et al. (2004) Paul Rayson, Damon Berridge, and Brian Francis. 2004.  Extending the Cochran rule for the comparison of word frequencies between corpora.  In _Proc. JADT ’04_ , pages 926–765.
  * Strötgen and Gertz (2015) Jannik Strötgen and Michael Gertz. 2015.  A Baseline Temporal Tagger for all Languages.  In _Proc. EMNLP 2015_ , pages 541–547.
  * Yimam et al. (2016) Seid Muhie Yimam, Heiner Ulrich, Tatiana von Landesberger, Marcel Rosenbach, Michaela Regneri, Alexander Panchenko, Franziska Lehmann, Uli Fahrer, Chris Biemann, and Kathrin Ballweg. 2016.  new/s/leak – Information Extraction and Visualization for Investigative Data Journalists.  In _Proc. ACL 2016 System Demonstrations_ , pages 163–168.