# Construction of a Quality Estimation Dataset for Automatic Evaluation of
Japanese Grammatical Error Correction

###### Abstract

In grammatical error correction (GEC), automatic evaluation is an important
factor for research and development of GEC systems. Previous studies on
automatic evaluation have demonstrated that quality estimation models built
from datasets with manual evaluation can achieve high performance in automatic
evaluation of English GEC without using reference sentences. However, quality
estimation models have not yet been studied in Japanese, because there are no
datasets for constructing quality estimation models. Therefore, in this study,
we created a quality estimation dataset with manual evaluation to build an
automatic evaluation model for Japanese GEC. Moreover, we conducted a meta-
evaluation to verify the dataset’s usefulness in building the Japanese quality
estimation model.

Keywords: corpus construction, automatic evaluation, quality estimation,
Japanese grammatical error correction

languageresourceLanguage Resources

Construction of a Quality Estimation Dataset for Automatic Evaluation of
Japanese Grammatical Error Correction

$\text{Daisuke Suzuki}^{\text{1}}$, $\text{Yujin Takahashi}^{\text{1}}$,
$\text{Ikumi Yamashita}^{\text{1}}$, $\text{Taichi Aida}^{\text{1}}$,
$\text{Tosho Hirasawa}^{\text{1}}$,
---
$\text{Michitaka Nakatsuji}^{\text{1}}$, $\text{Masato Mita}^{\text{2,1}}$,
$\text{Mamoru Komachi}^{\text{1}}$
$\text{Tokyo Metropolitan University}^{\text{1}}$
$\text{RIKEN}^{\text{2}}$
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>aida-
<EMAIL_ADDRESS><EMAIL_ADDRESS>
nakatsuji-michitaka@ed.,}tmu.ac.jp<EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Grammatical error correction (GEC) is the task of correcting different kinds
of errors in text such as spelling, punctuation, grammatical, and word choice
errors. Automatic evaluation is considered as an important factor for research
and development of GEC systems.

There are two types of automatic evaluation systems for GEC: reference-based
and reference-less. The former has been used extensively in the GEC community,
while the latter has gained much attention recently.

One of the problems of reference-based methods is the diversity of references.
In a study by bryant-ng-2015-far, a dataset with corrections was created by 10
annotators on the CoNLL-2014 test set ng-etal-2014-conll. They found that the
number of error types corrected by each annotator varied significantly,
indicating the existence of a variety of corrections for a sentences
containing error. However, with a limited number of reference sentences, it is
difficult to cover such a variety of corrected sentences in GEC. So reference-
based automatic evaluation methods may give unreasonably low scores to system
outputs that are validly corrected but unseen in the references.

On the other hand, reference-less automatic evaluation methods (quality
estimation models) do not have this problem because it estimates the quality
of the system’s output without requiring gold-standard references. ?) and ?)
built a quality estimation model using a dataset with manual evaluations,111?)
have built several quality estimation models, and one of them uses a dataset
of manual evaluation values. and achieves a higher correlation with manual
evaluation than with reference-based methods.

However, there is no quality estimation dataset for Japanese, although
Japanese is a language with a large number of language learners. This means no
reference-less automatic evaluation method has been proposed for Japanese GEC.
In the NAIST Lang-8 Learner Corpora mizumoto-etal-2011-mining, one of the
largest corpora of language learners, out of its composition of 580,549 essays
written by language learners, 185,991 (which is the second-largest number
after English) are written in Japanese. This means that the study on quality
estimation models for Japanese GEC can be large impact to GEC community.

Thereby, in this study, we created a quality estimation dataset with manual
evaluation to build an automatic evaluation model for Japanese GEC. The
dataset consisted of three components: source text, corrected texts, and
manually evaluated scores. The source text comes from the Lang-8 corpus, and
the corrected texts consisted of the output of four diverse GEC systems. We
built a quality estimation model by finetuning a pre-trained sentence encoder
(namely BERT [Devlin et al. (2019]) on the created dataset, and calculated the
correlation with the manual evaluation values. We also calculated correlations
with manual evaluations for reference-based automatic evaluation methods and
compared them with the correlations of the quality estimation model to meta-
evaluate the quality estimation model built using this dataset.

The main contributions of this study are as follows.

  * •

To build a quality estimation model for Japanese GEC, we created a dataset of
multiple system outputs annotated with human evaluation.

  * •

We demonstrated that the quality estimation model for Japanese GEC performs
better than the reference-based automatic evaluation by building a quality
estimation model using the created data set and evaluating its performance.

## 2\. Related Work

### 2.1. Evaluation Method

#### Reference-based methods.

In the early stage of English GEC, system outputs were evaluated by
calculating the match rate, recall rate, and F-score per word [Dale and
Kilgarriff (2011]. After the success of shared tasks in GEC, Max Match (M2),
which calculates the match rate, recall rate, and F0.5 score per phrase, has
been widely adopted in the NLP community. However, despite the prevalence of
the M2 scorer, ?) proposed I-measure, which gives a score of $-1$ to $1$,
while other evaluation methods give a score of $0$ to $1$, so that the bad
corrections will receive low scores while good corrections will gain high
scores. Further, ?) proposed GLEU, which is a modified version of BLEU
[Papineni et al. (2002] for evaluating GEC. BLEU evaluates by comparing the
translated sentence with the reference sentence, while GLEU evaluates by
comparing three sentences: the source sentence, the corrected sentence, and
the reference sentence. Among the methods that used reference sentences, GLEU
had the highest correlation with manual evaluation.

As for Japanese GEC, ?) conducted an automatic evaluation using BLEU to
compare the grammaticality of the language learner’s written sentences and the
reference sentences. More recently, ?) conducted an automatic evaluation using
GLEU with an evaluation corpus for Japanese GEC created in their research.

#### Reference-less methods.

Reference-less automatic evaluation methods, on the other hand, have been
proposed recently. ?) evaluated a GEC system using the number of errors
detected by the grammatical error detection system and showed that it
performed as well as GLEU. ?) proposed a method for evaluating correction
sentences in terms of grammaticality, fluency, and meaning preservation, using
logistic regression models for grammaticality, RNN language models for
fluency, and METEOR [Denkowski and Lavie (2014] for meaning preservation. ?)
proposed a method to optimize the three evaluation measures for manual
evaluation as an extension of ?). The method builds a quality estimation model
by finetuning a pre-trained sentence encoder (BERT) on the manual evaluation
of each measure and shows better correlation with the manual evaluation than
?).

However, reference-less automatic evaluation has not been applied to Japanese
GEC, while reference-based automatic evaluation is commonly used [mizumoto-
etal-2011-mining, koyama-etal-2020-construction]. Therefore, in order to adapt
a reference-less automatic evaluation method for the Japanese GEC, we created
a dataset with manual evaluation values for building a quality estimation
model.

### 2.2. Dataset with Human Evaluation

We introduce existing datasets with manual evaluation of corrected sentences
as related work.

The GUG dataset heilman-etal-2014-predicting is a dataset consists of 3,129
sentences randomly sampled from English language learners’ essays. The dataset
contains five grammaticality ratings for sentences written by a single English
learner, one rating by a linguistically trained annotator and five ratings by
crowdsourced annotators. The GUG dataset was created to measure the
performance of automatic evaluation methods, and it has also been used as
training data for quality estimation models. In this study, the evaluation was
done on a 5-point Likert scale, following the GUG data set.

grundkiewicz-etal-2015-human assigned human ratings to the output of 12
grammatical error correction systems that participated in the CoNLL2014 Shared
Task on English GEC ng-etal-2014-conll. In their dataset, the human ratings
are annotated relative to the source text by ranking multiple corrections.

In this study, we created a dataset of manual evaluation as the training data
for a quality estimation model for Japanese GEC. Because a dataset with manual
evaluation has not yet been created for Japanese GEC, the dataset created in
this study will be useful for evaluating the performance of automatic
evaluation methods for Japanese GEC.

## 3\. Construction of QE Dataset

To construct the dataset, we first created pairs of error-containing and
corrected sentences. To create sentence pairs, we used four diverse GEC
systems (Sec. 3.1) to generate correction sentences for two corpora of
Japanese learners (Sec. 3.2). We then manually evaluated the pairs of error-
containing and corrected sentences (Sec. 3.3).

### 3.1. Grammatical Error Correction System

We used the NAIST Lang-8 Learner Corpora mizumoto-etal-2011-mining for the
training data. It was created by collecting essays written by language
learners and their correction logs from 2007 to 2011 from Lang-8, a mutually
corrected social media platform for language learners. It contains the
language learners’ sentences and their corrections, essay IDs, user IDs,
learning language tags, and native language tags in the JSON format. The
number of essays with the Japanese learning language tag is 185,991, and the
number of sentence pairs of learners’ sentences and corrections is
approximately 1.4 million.

We employed four GEC systems to output four correction sentences for one
sentence to collect manual evaluations of various types of correction
sentences. The following four representative GEC systems were used.

#### SMT:

The model is based on statistical machine translation (SMT), a method that
learns the translation probability of each word or phrase and the probability
of the correct sequence as statistical information. Moses [Koehn et al. (2007]
was used as a tool for the SMT. KenLM [Heafield (2011] was used to train the
language model.

#### RNN:

A sequence-to-sequence transformation model based on recurrent neural network,
which is a neural network method that takes into account information about the
time series of data. The implementation is based on fairseq [Ott et al.
(2019]. Experiments were conducted using bi-directional LSTM. The number of
word dimensions was set to 512, the batch size was set to 32, and the rest of
the implementation settings followed those in [Luong et al. (2015].

#### CNN:

A sequence-to-sequence transformation model based on convolutional neural
network that learns by abstracting features of the data. The implementation is
based on fairseq. The dimensions of the encoder and decoder were set to 512,
and the remainder of the implementation settings were followed [Chollampatt
and Ng (2018].

#### Transformer:

A sequence-to-sequence transformation model consisting only of a mechanism
called attention, which represents the attention of words in a sentence. The
implementation is based on fairseq. The parameter settings followed those of
[Vaswani et al. (2017].

### 3.2. Datasets

To obtain pairs of error-containing and corrected sentences, we used the TEC-
JL koyama-etal-2020-construction. and FLUTEC 222https://github.com/kiyama-
hajime/FLUTEC as the data set with error-containing sentences.

Corpus | Essays | Sentence pairs
---|---|---
Lang-8 | 192,673 | 1,296,114
TEC-JL | 139 | 2,042
FLUTEC | 169 | 2,100
Table 1: Number of essays and number of sentence pairs in the dataset used in
the experiment. For Lang-8, it reports the number of essays and sentence pairs
in Japanese.

#### TEC-JL.

TEC-JL includes 139 essays from Lang-8 containing 2,042 sentence pairs between
learners’ sentences and corrections. The corrections were carried out by
Japanese native speakers with minimal edits. Unlike the original Lang-8
corpus, the TEC-JL is a relatively reliable dataset for evaluation. In our
experiments, we selected this dataset for comparing the performances of our
quality estimation model with the original automatic evaluation methods using
reference sentences.

#### FLUTEC.

This dataset was annotated with sentences from Japanese language learners
sampled from the Lang-8 corpus to generate multiple fluency-aware corrective
sentences. The dataset consists of dev and test data, with 1,050 sentences
sampled each. Note that in the experiment in Sec. 4, we used only the dataset
created from TEC-JL.

### 3.3. Annotation

#### Policies.

We obtained a set of pairs of Japanese learner sentences and corrected
sentences by having four GEC systems correct the sentences of Japanese
learners in the TEC-JL and FLUTEC. To create an effective dataset for building
a quality estimation model, we conducted annotation based on two policies: 1)
Holistic evaluation. 2) Annotation based on pairs of Japanese learners and
corrected sentences.

First, ?) collected human ratings of corrected sentences on three measures of
grammaticality, fluency, and meaning preservation, and fine-tuned BERT on each
measure to build a quality estimation model. As a result, the model trained on
grammaticality showed the best correlation with human ratings compared to
reference-based automatic evaluation methods. Borrowing from this, in order to
create a quality estimation dataset at a low cost, we collected human ratings
using a single grammaticality-oriented holistic rating scale.

Second, in ?), grammaticality was evaluated by examining only the corrected
sentence using a simple five-step evaluation rule was used. In the present
study, to consider meaning preservation on a single evaluation scale, the
original and corrected sentences were evaluated in pairs, and the rules were
designed based on the evaluation of both before and after correction.

#### Process.

Four GEC systems were used to generate corrected sentences for 2,042 sentences
written by Japanese learners included in TEC-JL. We asked three native
Japanese university students to evaluate 4,391 sentence pairs, excluding the
duplicated sentence pairs. For a rules agreement among the annotators, the
evaluations of up to 2,000 sentences were discussed. The cases where scores
differed by more than two were discussed among the annotators, and the
guidelines were updated to supplement the rules for cases where evaluations
were likely to differ.

Score |  Description
---|---
4 |  S2 is grammatically correct or all significant and minor errors in S1 have been corrected.
3 |  All significant errors in S2 have been corrected. Some minor errors are acceptable. (e.g., Sentences containing not wrong, but unnecessary edits. Sentences containing spaces.)
2 |  S2 is able to correct important errors, but unacceptable minor errors are not corrected. Minor erroneous corrections have been made. (e.g., Missing punctuation. Sentences ending with a reading mark or comma. N.B. These errors can have a score of 3 at annotator’s decision.)
1 |  S2 is not able to correct significant and minor errors. Serious incorrect corrections have been made. (e.g., Changes to nouns and verbs that significantly impair the meaning of the original text. Cases where the meaning has changed significantly from the original text even if they are grammatically correct.)
0 |  S1 is a sentence that is difficult to correct. Sentence that cannot be corrected. (e.g., Sentences where more than half of the text is non-Japanese)
Table 2: Description of evaluation scores. S1 is the original text and S2 is
the corrected text.

#### Guidelines.

We used a 5-point Likert scale scheme to annotate grammaticality-based
evaluation scores. The evaluation scores were determined according to the
rules described in Table 2. 333In addition, we did not consider the following
usages. First, the conversion from present tense to past tense is not treated
as an error. Second, commas and periods were treated the same as reading and
punctuation marks. Third, we ignore emoticons and symbols regardless of their
position.

In the experiment, the average ratings of the three annotators were used as
the final manual evaluation. As an exception, in order to correctly reflect
the annotator’s evaluation, cases where zero was included in the three
evaluations were handled as follows: cases when one of the three annotators
rated the case as 0, the average of the other two annotators’ ratings was
taken; when two or more annotators rated the case as 0, no average was taken
but the rating was set to 0.

### 3.4. Analysis

Source text | Corrected sentence | Scores (avg.)
---|---|---
このアルバイトを本当に楽しみにします。 | このアルバイトを本当に楽しみです。 | 2, 3, 3 (2.67)
(I will really look forward to this part-time job.) |  (I am really looking forward to this part-time job.) |
元々沖縄へ行きたいでした。 | 元々沖縄へ行きたかったです。 | 4, 4, 4 (4.00)
(Originally, I want to went to Okinawa.) |  (Originally, I wanted to go to Okinawa.) |
Table 3: Annotation examples. The numbers in the Scores column represent the
scores of the three annotators. Figure 1: Histogram of manual evaluation
scores for sentence pairs generated from TEC-JL. Figure 2: Histogram of manual
evaluation scores for sentence pairs generated from FLUTEC.

To measure the agreement rate between annotators, we used the kappa
coefficient. The value was found to be 0.49, indicating moderate agreement
[Landis and Koch (1977]. Figures 1 and 2 show histograms of manual evaluation
scores for sentence pairs created from TEC-JL and FLUTEC.444Because of the way
averages are taken, there is no rating greater than 0 and less than 1. For
both datasets, overall number of ratings is 3 to 4. In the graph for TEC-JL,
the small number of 0 ratings is probably due to the fact that low noise cases
were selected by random sampling; for FLUTEC, the number of 0 ratings is very
small due to the fact that low noise data was manually selected during
sampling.

Table 3 shows actual annotation examples. In the above annotation example, a
wrong sentence was corrected only partially. The annotator evaluated this
failure as a minor error, and rated the correction 2 or 3 at the annotator’s
discretion. In the annotation example below, all annotators gave a rating of 4
because the valid and sufficient corrections were made.

## 4\. QE Experiments

To evaluate the quality estimation performance of the Japanese GEC, we
measured the sentence-level correlation coefficients between the human
evaluation scores of the reference-based and reference-less evaluations for
the output of the GEC system.

### 4.1. Settings

To build a quality estimation model, we used the same method as ?) to fine-
tune BERT. The input of BERT is the language learner’s written and corrected
sentences, and the output is the evaluation score. Because ?) assessed
grammaticality, fluency, and meaning preservation separately, only the
corrected sentences were used as input to the quality estimation model for
grammaticality. However, in this study, because we also assessed meaning
preservation at once, both the learner’s sentences and the corrected sentences
were used as input for BERT. For the output, we changed the output layer of
the BERT to a regression model to output the ratings.

As for the proposed method, we performed a 10-fold cross-validation using the
dataset created in Section 3.3. We divided the dataset into 10 parts so that
the ratio of training, development, and testing was 8:1:1 and fine-tuned BERT
with the training data. Using dev data, we selected BERT hyperparameters with
maximum Pearson’s correlation coefficient by grid search with maximum sentence
length 128, 256, batch size 8, 16, 32, learning rate 2e-5, 3e-5, 5e-5, and
number of epochs 1-10.

For the baseline, we used GLEU as a reference-based method. Because GLEU is
evaluated with a value between 0 and 1, the evaluation was multiplied by 4 for
comparison. As reference sentences, we used two sentences from TEC-JL. For the
test data, we calculated the GLEU scores, quality estimation scores, and
measured sentence-level correlations with manual evaluations. For meta-
evaluation, we used Pearson’s correlation coefficient and Spearman’s rank
correlation coefficient between the scores of the automatic and the manual
evaluations for each sentence in the test data.

### 4.2. Results

Method | Pearson | Spearman
---|---|---
GLEU | 0.320 | 0.362
fine-tuned BERT | 0.580 | 0.413
Table 4: Results of meta-evaluation. Figure 3: Sentence-level correlation
between automatic and manual evaluation scores.

Table 4 shows the results of sentence-level comparison for each automatic and
manual evaluation score using Pearson’s correlation coefficient and Spearman’s
rank correlation coefficient. The results of the experiment show that fine-
tuned BERT is more highly correlated with human ratings than GLEU in both
Pearson’s correlation coefficient and Spearman’s rank correlation coefficient.
Figure 3 shows that there is almost no correlation between the GLEU scores and
the manual ratings, while the fine-tuned BERT model shows a weak correlation
with the manual ratings, and the model tends to do a proper quality estimation
especially for the cases rated 1.

### 4.3. Analysis

| Example 1 | Example 2
---|---|---
S | ソンあさんが好きがっているのは推理小説です。 | 堤 真一さんの演技もよかったですね。
| (Son A wants to like reading mysteries.) | (Shinichi Tsutsumi’s acting was also good.)
C | ソンあさんが好きになっているのは推理小説です。 | 写真一さんの演技もよかったですね。
| (Son A is becoming fond of mystery novels.) | (Photo 1’s performance was also good.)
R | ソンアさんが好んでいるのは推理小説です。 | 堤 真一さんの演技もよかったですね。
| (Sonna likes to read mysteries.) | (Shinichi Tsutsumi’s acting was also good.)
| ソンアさんが好きなのは推理小説です。 | 堤 真一さんの演技もよかったですね。
| (Sonna likes to read mysteries.) | (Shinichi Tsutsumi’s acting was also good.)
E | GLEU, BERT, Human / 0.59, 3.30, 3.00 | GLEU, BERT, Human / 3.14, 1.12, 1.00

Table 5: Successful examples of the quality estimation method. The first line
shows the source text (S), the second line shows the corrected sentence (C),
the third line shows the two reference sentences (R), and the fourth line
shows automatic and manual evaluations (E).

| Example 1 | Example 2
---|---|---
S | 社会人になりたくない原因が多いだ。 | ほんとですか？
| (There are many cause why I don’t want to be a working adult.) | (Really?)
C | 社会人になりたくない理由が多い。 | 本当ですか？
| (There are many reasons why I don’t want to be a working adult.) | (Really?)
R | 社会人になりたくない原因が多い。 | 本当ですか？
| (There are many cause why I don’t want to be a working adult.) | (Really?)
| 社会人になりたくない原因が多い。 | ほんとですか？
| (There are many cause why I don’t want to be a working adult.) | (Really?)
E | GLEU, BERT, Human / 2.63, 1.45, 4.00 | GLEU, BERT, Human / 3.41, 2.03, 4.00

Table 6: Failed examples of the quality estimation method. The first line
shows the source text (S), the second line shows the corrected sentence (C),
the third line shows the two Reference sentences (R), and the fourth line
shows automatic and manual evaluations (E).

To investigate the differences in the evaluations, we analyzed an example of
evaluation by the two automatic evaluation methods: the quality estimation
BERT model and the reference-based GLEU.

#### Successful cases.

Table 6 shows two examples where the quality estimation model was able to
evaluate correctly.

The corrected sentence in Example 1 is grammatically correct, but the word
“ソンあ (Sonna)”, which is supposed to be a person’s name, is expressed in a
mixture of katakana and hiragana, resulting in a manual evaluation of 3.0555In
Japanese, katakana should be used for transliteration.. The reference sentence
has two types of correction for the expression “好きがっている (wants to like)”,
however, because this is not an expression that can be corrected by the GEC
system, its evaluation widely differs from the manual evaluation. However, the
quality estimation model was relatively close to the manual evaluation.

In Example 2, the person’s name “堤 真一 (Tsutsumi Shinnichi)” is corrected to
“写真一 (Photo 1)” in the corrected text, and the manual evaluation is 1 because
the meaning of the original text is greatly impaired by this correction. In
the reference sentence, no correction was made to the source text, and the
output of the GEC system was superficially similar to the source text. Because
GLEU calculates the score by subtracting the number of n-grams that appear in
the source text but do not appear in the reference text from the number of
n-gram matches between the corrected text and the reference text, it gives a
high score to the corrected text. Meanwhile, the quality estimation model
captures the meaning changes between the source and reference sentences and
can provide an evaluation similar to a manual evaluation.

#### Failed cases.

We analyzed examples of evaluations in which the quality estimation model
could not be correctly evaluated. Table 6 shows two examples where the quality
estimation model failed to evaluate correctly.

In Example 1, the manual evaluation is 4.0 because the correct correction is
made but the quality estimation model’s evaluation widely differs from from
the manual evaluation. Meanwhile, GLEU can determine from the reference
sentences that the deletion of the “だ (copula)” at the end of the word is a
correct correction, and thus it is closer to the manual evaluation than the
quality estimation model.

In Example 2, the quality estimation model does not recognize the edit of “ほんと
(really)” to “本当 (really)” as a correct correction666“本当” is pronounced as
“ほんとう” and “ほんと” is a colloquial expression of “本当”., while GLEU can recognize
the edit as a correct correction from the reference sentence. By contrast,
GLEU recognizes that the edit is a correct correction from the reference
sentence.

## 5\. Conclusions

In this study, we constructed a dataset that includes a manual evaluation of a
grammaticality-oriented holistic approach added to the outputs of Japanese GEC
systems. This dataset consisted of three elements: source text, corrected
text, and human ratings. The source text consisted of the Lang-8 corpus, and
the corrected text consisted of the outputs of the four GEC systems. The human
ratings are annotated by students whose first language is Japanese.

Using the dataset, we optimized BERT directly on human ratings to create a
quality estimation model. To compare the performance of the reference-less and
reference-based methods, we measured the sentence-level correlation
coefficients between the evaluation scores of each method and the human
evaluation scores for the output of the GEC systems. The experimental results
showed that the quality estimation model had a higher correlation with manual
evaluation than the method using reference sentences, demonstrating the
usefulness of a reference-less automatic evaluation method for Japanese GEC.

Future developments include a more detailed analysis using error-type
annotation and research to measure the GEC performance by selecting the output
with the highest QE score from the outputs of multiple GEC systems by
reranking.

## 6\. Bibliographical References

## References

  * Asano et al. (2017 Asano, H., Mizumoto, T., and Inui, K.  (2017).  Reference-based Metrics can be Replaced with Reference-less Metrics in Evaluating Grammatical Error Correction Systems.  In Proceedings of the Eighth International Joint Conference on Natural Language Processing, pages 343–348, Taipei, Taiwan. Asian Federation of Natural Language Processing.
  * Chollampatt and Ng (2018 Chollampatt, S. and Ng, H. T.  (2018).  A Multilayer Convolutional Encoder-Decoder Neural Network for Grammatical Error Correction.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 5755–5762, New Orleans, Louisiana. Association for the Advancement of Artificial Intelligence.
  * Dale and Kilgarriff (2011 Dale, R. and Kilgarriff, A.  (2011).  Helping Our Own: The HOO 2011 Pilot Shared Task.  In Proceedings of the 13th European Workshop on Natural Language Generation, pages 242–249, Nancy, France. Association for Computational Linguistics.
  * Denkowski and Lavie (2014 Denkowski, M. and Lavie, A.  (2014).  Meteor universal: Language specific translation evaluation for any target language.  In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376–380, Baltimore, Maryland, USA. Association for Computational Linguistics.
  * Devlin et al. (2019 Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  (2019).  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Felice and Briscoe (2015 Felice, M. and Briscoe, T.  (2015).  Towards a standard evaluation method for grammatical error detection and correction.  In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 578–587, Denver, Colorado. Association for Computational Linguistics.
  * Heafield (2011 Heafield, K.  (2011).  KenLM: Faster and Smaller Language Model Queries.  In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187–197, Edinburgh, Scotland. Association for Computational Linguistics.
  * Koehn et al. (2007 Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E.  (2007).  Moses: Open Source Toolkit for Statistical Machine Translation.  In ACL 2007, Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, Prague, Czech Republic. Association for Computational Linguistics.
  * Landis and Koch (1977 Landis, J. R. and Koch, G. G.  (1977).  The Measurement of Observer Agreement for Categorical Data.  Biometrics, 33(1):159–174.
  * Luong et al. (2015 Luong, M., Pham, H., and Manning, C. D.  (2015).  Effective Approaches to Attention-based Neural Machine Translation.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, page 1412–1421, Lisbon, Portugal. Association for Computational Linguistics.
  * Napoles et al. (2015 Napoles, C., Sakaguchi, K., Post, M., and Tetreault, J.  (2015).  Ground Truth for Grammatical Error Correction Metrics.  In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 588–593, Beijing, China. Association for Computational Linguistics.
  * Napoles et al. (2016 Napoles, C., Sakaguchi, K., and Tetreault, J.  (2016).  There’s No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correction.  In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2109–2115, Austin, Texas. Association for Computational Linguistics.
  * Ott et al. (2019 Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., and Auli, M.  (2019).  fairseq: A fast, extensible toolkit for sequence modeling.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Papineni et al. (2002 Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J.  (2002).  Bleu: a Method for Automatic Evaluation of Machine Translation.  In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Vaswani et al. (2017 Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I.  (2017).  Attention is All you Need.  In Advances in Neural Information Processing Systems, pages 5998–6008. Curran Associates, Inc.
  * Yoshimura et al. (2020 Yoshimura, R., Kaneko, M., Kajiwara, T., and Komachi, M.  (2020).  SOME: Reference-less Sub-Metrics Optimized for Manual Evaluations of Grammatical Error Correction.  In Proceedings of the 28th International Conference on Computational Linguistics, pages 6516–6522, Barcelona, Spain (Online). International Committee on Computational Linguistics.

## 7\. Language Resource References

lrec2022-bib languageresource


11institutetext: Computational Radiology Laboratory (CRL), Department of
Radiology, Boston Children’s Hospital, and Harvard Medical School, USA
22institutetext: Department of Radiology, Lausanne University Hospital (CHUV)
and University of Lausanne (UNIL), Lausanne, Switzerland

# TBSS++: A novel computational method for Tract-Based Spatial Statistics

Davood Karimi 11 Hamza Kebiri 1122 and Ali Gholipour 11

###### Abstract

Diffusion-weighted magnetic resonance imaging (dMRI) is widely used to assess
the brain white matter. One of the most common computations in dMRI involves
cross-subject tract-specific analysis, whereby dMRI-derived biomarkers are
compared between cohorts of subjects. The accuracy and reliability of these
studies hinges on the ability to compare precisely the same white matter
tracts across subjects. This is an intricate and error-prone computation.
Existing computational methods such as Tract-Based Spatial Statistics (TBSS)
suffer from a host of shortcomings and limitations that can seriously
undermine the validity of the results. We present a new computational
framework that overcomes the limitations of existing methods via (i) accurate
segmentation of the tracts, and (ii) precise registration of data from
different subjects/scans. The registration is based on fiber orientation
distributions. To further improve the alignment of cross-subject data, we
create detailed atlases of white matter tracts. These atlases serve as an
unbiased reference space where the data from all subjects is registered for
comparison. Extensive evaluations show that, compared with TBSS, our proposed
framework offers significantly higher reproducibility and robustness to data
perturbations. Our method promises a drastic improvement in accuracy and
reproducibility of cross-subject dMRI studies that are routinely used in
neuroscience and medical research.

###### Keywords:

Diffusion MRI deep learning brain white matter tract-based spatial statistics
TBSS

##  1 Introduction

Diffusion of water in the brain tissue is influenced by tissue micro-
structure. Diffusion MRI (dMRI) methods can characterize this process
quantitatively [13, 5]. These methods are widely used in medicine and
neuroscience. Thousands of studies have used dMRI parameters such as
fractional anisotropy (FA) as biomarkers to assess normal brain development
and to characterize the impacts of various neurological diseases [3, 24, 14].

_Tract-specific studies_ analyze dMRI-computed biomarkers on specific white
matter tracts [25]. They may consider the same subject(s) over time
(longitudinal studies) or compare different cohorts of subjects such as
healthy and diseased (population studies). They are among the most common and
most fruitful classes of dMRI studies because many neurodevelopmental deficits
and neurological disorders are linked to damage to the tissue micro-structure
on specific tracts [17, 23, 34]. The success of these studies depends on
precise alignment of different brain scans to ensure that the same tracts are
compared between scans/subjects. This is a complex and highly error-prone
computation. Existing computational methods vary greatly in terms of spatial
accuracy and the required manual work [25]. For example, voxel-based
morphometry methods are simple, but they are incapable of accurate tract-
specific comparisons [2]. There are also semi-automatic and automatic
tractography-based methods that require performing tractography on each scan
[16, 27].

Currently, Tract-Based Spatial Statistics (TBSS) [26] is the _de-facto_ method
for tract-specific dMRI analysis. It can analyze the entire brain and consists
of five steps: (1) Compute FA images for each subject; (2) Register the FA
images to a common space; (3) Compute the voxel-wise average of the registered
FA images and create the skeleton of this mean FA; (4) Project subjects’ FA
images onto the skeleton; (5) Perform statistical analysis across subjects on
the skeleton. Despite its popularity, TBSS suffers from important shortcomings
that have been extensively reported. Here we briefly list some of them: (1)
FA-based registration is inaccurate, and subsequent steps in TBSS fail to
correct the registration errors [32]. (2) Presence of lesions reduce the
accuracy of TBSS, which is a serious limitation because such pathology may be
the main focus of the analysis [11]. (3) Results are sensitive to measurement
noise, choice of FA template, and FA threshold [15]. (4) TBSS is prone to
confusing adjacent tracts even under ideal and simulated settings [4]. (5) Its
central assumption is that the locations of interest are the tract centers,
defined as the voxels with the highest FA. This assumption may not be valid.
(6) Computational steps such as skeletonization and skeleton projection can
introduce systematic bias [7]. (7) TBSS is not suited for all tract shapes and
may produce erroneous results on small tubular tracts [4]. (8) Although TBSS
is an automatic method, it involves several settings that may influence the
results. The impacts of these settings are not fully clear, and one should
visually verify the results of different processing steps [15].

These limitations significantly compromise the validity of the results of
cross-subject dMRI studies. It has been shown that tweaking the settings of
the existing methods may lead to completely different conclusions [4, 12].
Alternative methods have been proposed to improve certain aspects of TBSS [31,
30]. However, these alternative methods have their own limitations and none of
them has reached TBSS’s popularity. Despite years of effort, existing methods
suffer from serious shortcomings that are data-dependent and difficult to
predict.

Given the increasing popularity of dMRI and the growing size of datasets,
there is an urgent need for accurate and reliable computational tools. The
goal of this paper is to develop and validate a new computational framework,
in large part based on deep neural networks, that overcomes some of the
limitations of the existing methods. We show that the new methods enable
unprecedented accuracy in tract-specific cross-subject analysis of dMRI data.
Therefore, they can have a significant impact as they dramatically enhance the
reliability and reproducibility of longitudinal and population studies with
dMRI.

##  2 Materials and methods

###  2.1 Data

We use two datasets in this work. (1) Human Connectome Project (HCP) dataset.
We use 105 adult brains from the HCP dataset [28] and manual segmentations of
72 tracts on these brains available via [29]. The dMRI scans in this dataset
include high angular resolution multi-shell (b=1000, 2000, and 3000)
measurements. In all experiments with this data, we used 70 subjects for model
development and the remaining 35 for test. (2) Calgary Preschool dataset [21,
20], which we only use for further validation. This dataset includes 396 dMRI
scans from 120 children 2-8 years of age. Each scan in this dataset has 35
single-shell (b=750) measurements.

###  2.2 Methods

Figure 1 shows the proposed computational framework. It consists of three main
components that we describe in the following three sub-sections.

Figure 1: A graphical summary of the proposed computational framework. To
simplify the illustration, we have shown only one subject and only four
tracts.

####  2.2.1 Tract segmentation method.

Many cross-subject analysis methods, including TBSS, do not segment individual
tracts, which causes two problems: (1) Tract-specific computation is not
possible. In order to conduct a tract-specific analysis, tracts will have to
be segmented either manually or with an auxiliary method. (2) Ambiguity in the
location and extent of the tracts can result in significant errors (e.g., mis-
assignment of voxels) even for prominent tracts such as corpus callosum and
cingulum [4]. Segmentation of individual tracts can overcome this fundamental
limitation. To this end, we use a fully-convolutional network similar to nnU-
Net [9] to segment the individual tracts. The segmentation method itself is
not a novel aspect of this work. In fact, similar methods have been proposed
in recent works [29].

####  2.2.2 Atlases of white matter tracts.

We computed fiber orientation distribution images for the 70 training subjects
using the MSMT-CSD method [10]. We aligned all 70 brains into a common space
using an iterative succession of rigid, affine, and non-linear registrations
computed based on fiber orientation distributions using the registration
method proposed in [18]. We then used the composite deformation fields for
each subject to align individual tract segmentations into the same space and
used voxel-wise averaging and thresholding at 0.50 to compute tract atlases.
We further applied the computed deformation fields to the diffusion MRI data
of each subject, represented in spherical harmonics basis, to compute a
corresponding atlas. These atlases would serve as an unbiased reference space
where we could register the data from individual subjects for tract-specific
analysis.

####  2.2.3 dMRI registration method.

Accurate registration of brain dMRI data from different subjects is the most
central computation in cross-subject studies. Existing methods mostly rely on
registration based on fractional anisotropy (FA) images, which can be highly
inaccurate. In our proposed framework, we require a flexible registration
technique to register the subject brains to the reference atlas. Specifically,
similar to our atlas computation described above, we use the method of [18]
that is based on computing a nonlinear deformation map based on fiber
orientation distributions.

To summarize: the combination of the tract segmentation method, the atlas, and
the registration method constitute our proposed framework. The method works as
follows (Figure 1): (i) Compute the parameter(s) of interest (e.g., FA) and
segment tracts of interest for every subject; (ii) Align all subjects into the
reference atlas space; (iii) Perform statistical analysis between desired sub-
groups of the subjects to determine the differences in the parameters on each
tract.

##  3 Results and Discussion

###  3.1 Assessment of the three components of the framework

Our main interest is on the effectiveness of the entire framework for
performing tract-specific studies. Nonetheless, we first briefly explain the
evaluation of the three components.

####  3.1.1 Assessment of the segmentation method.

Our deep learning based segmentation method achieved a DSC of $0.826\pm
0.076$, which is comparable with the state of the art method [29] on the same
dataset. However, unlike [29] that needs the data in all three shells, our
method requires only the measurements in the b=1000 shell.

####  3.1.2 Assessment of atlases of white matter tract.

Figure 2 shows several views from this atlas. All tracts were reconstructed in
the atlas with high detail. In order to quantitatively assess the accuracy of
the atlas, we divided the 70 training subjects into two groups, each with 35
subjects, generated two separate atlases, and computed the DSC between white
matter tracts in the two atlas. The mean DSC across all tracts was 0.962,
suggesting that the atlases could be used as an unbiased reference space for
registering and comparing data from individual subjects.

Figure 2: LEFT: one slice from the dMRI data and fiber orientation
distribution of the generated atlas. RIGHT (from top-left to bottom-right): a
volume showing all 72 tracts, inferior cerebellar peduncle, cingulum, anterior
commissure, frontopontine tract, and fornix.

####  3.1.3 Assessment of the registration method.

The most common approach to quantitative evaluation of registration accuracy
is in terms of the overlap of corresponding structures between moving and
fixed images [6, 33]. Following this approach, we assessed registration
accuracy in terms of DSC and average surface distance (ASD) between the warped
subject tracts and atlas tracts. For this evaluation, we considered four of
the tracts that spanned most of the brain volume: thalamo-prefrontal, striato-
parietal, and anterior midbody and isthmus of corpus calosum. The registration
accuracy in terms of DSC was $0.785\pm 0.074$, while the ASD was $0.775\pm
0.082$ mm.

###  3.2 Assessment of the overall framework for tract-specific cross-subject
analysis

In general, it is difficult to compare different computational frameworks for
cross-subject studies. The aim of such studies is to compare two or more
groups of subjects and identify the “true” differences between the groups.
However, discovering such differences, which typically have complex biological
underpinnings, is the _goal_ of these studies and they are not known in
advance. Therefore, for assessing computational methods, it is common to use
synthetic data, simulations, and test-retest/reproducibility experiments [4,
26, 22, 1, 15]. Following similar evaluation strategies, we applied our new
framework in several experiments and compared it with the official
implementation of TBSS [8]. In these experiments, described below, to enable
tract-specific comparison of our method with TBSS, we non-linearly registered
the mean FA image from TBSS to the atlas FA image.

Reproducibility. Each dMRI scan from the HCP dataset has 60-90 measurements in
the b=1000 shell. For each of the 35 test subjects from this dataset, we
selected two independent subsets of 30 measurements from this shell. We
applied our method and TBSS on these two measurement subsets and computed the
average FA on each tract for each subject. Let us denote the absolute
difference between the mean FA for tract $i$ and subject $j$ with
$\Delta\text{FA}_{\text{tract\\_}i,j}=\big{|}\text{FA}_{\text{tract\\_}i,j}^{\text{subset\\_2}}-\text{FA}_{\text{tract\\_}i,j}^{\text{subset\\_1}}\big{|}$.
Ideally, $\Delta\text{FA}_{\text{tract\\_}i,j}$ should be zero. Let us denote
the average of $\Delta\text{FA}_{\text{tract\\_}i,j}$ across all subjects with
$\Delta\text{FA}_{\text{tract\\_}i}$. We computed
$\Delta\text{FA}_{\text{tract\\_}i}$ on all tracts for our method and TBSS. On
35 out of 41 tracts, $\Delta\text{FA}_{\text{tract\\_}i}$ was smaller for our
method than for TBSS. On 25 of these 35 tracts, the difference was
statistically significant at $p=0.01$ computed using paired t-tests. On the
remaining 6 tracts, the difference was not significant. This result shows a
higher reproducibility for our proposed method. We further repeated this
experiment with only six measurements in each subset to simulate clinical
scans. In this experiment, on all 41 tracts
$\Delta\text{FA}_{\text{tract\\_}i}$ was smaller for our method than for TBSS;
22 of these differences were statistically significant at $p=0.01$. We also
plotted the profile of FA on the centerline of various tracts. The profiles
computed with our method showed higher consistency and reproducibility.

Robustness to data perturbation. For each of the 35 test subjects from the HCP
dataset, we added random Rician noise with a signal-to-noise-ratio of 20dB to
the dMRI measurements. We applied our method and TBSS on these noisy scans as
well as on the original scans and computed the difference in mean FA for tract
$i$ and subject $j$ as
$\Delta\text{FA}_{\text{tract\\_}i,j}=\big{|}\text{FA}_{\text{tract\\_}i,j}^{\text{noisy}}-\text{FA}_{\text{tract\\_}i,j}^{\text{original}}\big{|}$.
We denote the average of $\Delta\text{FA}_{\text{tract\\_}i,j}$ across all
subjects with $\Delta\text{FA}_{\text{tract\\_}i}$. On all 41 tracts,
$\Delta\text{FA}_{\text{tract\\_}i}$ was smaller for our method than for TBSS.
Paired t-tests showed that 28 of these differences were statistically
significant at $p=0.01$. This result shows that our method is more robust to
data perturbations.

Analysis of the impact of gender and age with an independent dataset. We
performed additional experiments on the Calgary Preschool dataset. With the
help of a semi-manual tract segmentation technique [19], this dataset had been
previously used to assess the impact of age and gender on the micro-structural
changes on 10 major white matter tracts [20]. We applied our proposed method
and TBSS on this dataset to determine the impact of gender and age on FA and
Mean Diffusivity (MD), and compared the results with [20]. This experiment
showed that, compared with TBSS, the results of our method were consistently
closer to the results of [20]. For example, our method showed a significantly
($p<0.01$) higher FA on pyramidal tracts and higher MD on the superior
longitudinal fasciculus for boys compared with girls, which was consistent
with the results of [20]; TBSS did not reveal these differences. We also
performed a reproducibility experiment with this dataset. This dataset
included 75 subjects (30 female) aged 3 years and 73 subjects (34 female) aged
6 years. We divided these into {Subset 1: 37 (15 female) aged 3 & 37 (17
female) aged 6 } and {Subset 2: 38 (15 female) aged 3 & 36 (17 female) aged
6}. We computed the change in FA and MD for each tract using the two subsets.
Ideally, the change computed with the two subsets should be equal. In terms of
FA for 35 of the tracts and in terms of MD for all 41 tracts the difference
between the changes computed using the two subsets was smaller with our method
than with TBSS. This experiment shows higher reproducibility for the proposed
method.

##  4 Conclusions

_The potential impact of the proposed method cannot be overstated._ Tract-
specific studies are increasingly used to answer critical research questions
in medicine and neuroscience. However, the shortcomings of existing
computational methods have seriously limited the accuracy and reproducibility
of these studies. Our experiments showed that the method proposed in this
paper offers superior reproducibility and robustness to data perturbations.
Experiments with the Calgary Preschool dataset further showed that our method
has high generalizability. Therefore, our proposed method can have a
significant impact by enhancing the accuracy, reliability, and reproducibility
of the results of tract-specific studies.

## Acknowledgment

This research was supported in part by the National Institute of Biomedical
Imaging and Bioengineering, the National Institute of Neurological Disorders
and Stroke, and Eunice Kennedy Shriver National Institute of Child Health and
Human Development of the National Institutes of Health (NIH) under award
numbers R01HD110772, R01NS128281, R01NS106030, R01EB018988, R01EB031849,
R01EB032366, and R01HD109395. This research was also partly supported by
NVIDIA Corporation and utilized NVIDIA RTX A6000 and RTX A5000 GPUs. The
content of this publication is solely the responsibility of the authors and
does not necessarily represent the official views of the NIH or NVIDIA. This
work was also supported by the Swiss National Science Foundation (project
205321-182602). We acknowledge access to the facilities and expertise of the
CIBM Center for Biomedical Imaging, a Swiss research center of excellence
founded and supported by Lausanne University Hospital (CHUV), University of
Lausanne (UNIL), Ecole polytechnique fédérale de Lausanne (EPFL), University
of Geneva (UNIGE) and Geneva University Hospitals (HUG).

## References

  * [1] Arkesteijn, G.A., Poot, D.H., Ikram, M.A., Niessen, W.J., Van Vliet, L.J., Vernooij, M.W., Vos, F.M.: Orientation prior and consistent model selection increase sensitivity of tract-based spatial statistics in crossing-fiber regions. IEEE transactions on medical imaging 39(2), 308–319 (2019)
  * [2] Ashburner, J., Friston, K.J.: Voxel-based morphometry—the methods. Neuroimage 11(6), 805–821 (2000)
  * [3] Assaf, Y., Pasternak, O.: Diffusion tensor imaging (dti)-based white matter mapping in brain research: a review. Journal of molecular neuroscience 34(1), 51–61 (2008)
  * [4] Bach, M., et al.: Methodological considerations on tract-based spatial statistics (tbss). Neuroimage 100, 358–369 (2014)
  * [5] Basser, P.J., Mattiello, J., LeBihan, D.: Mr diffusion tensor spectroscopy and imaging. Biophysical journal 66(1), 259–267 (1994)
  * [6] Dalca, A.V., Balakrishnan, G., Guttag, J., Sabuncu, M.R.: Unsupervised learning of probabilistic diffeomorphic registration for images and surfaces. Medical image analysis 57, 226–236 (2019)
  * [7] Edden, R.A., Jones, D.K.: Spatial and orientational heterogeneity in the statistical sensitivity of skeleton-based analyses of diffusion tensor mr imaging data. Journal of neuroscience methods 201(1), 213–219 (2011)
  * [8] FMRIB Analysis Group, O.U.: Tbss userguide - fslwiki. https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/TBSS/UserGuide, accessed: 2023-01-14
  * [9] Isensee, F., et al.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18(2), 203–211 (2021)
  * [10] Jeurissen, B., Tournier, J.D., Dhollander, T., Connelly, A., Sijbers, J.: Multi-tissue constrained spherical deconvolution for improved analysis of multi-shell diffusion mri data. NeuroImage 103, 411–426 (2014)
  * [11] Jones, D.K., Cercignani, M.: Twenty-five pitfalls in the analysis of diffusion mri data. NMR in Biomedicine 23(7), 803–820 (2010)
  * [12] Keihaninejad, S., et al.: The importance of group-wise registration in tract based spatial statistics study of neurodegeneration: a simulation study in alzheimer’s disease. PloS one 7(11), e45996 (2012)
  * [13] Le Bihan, D., et al.: Diffusion tensor imaging: concepts and applications. Journal of Magnetic Resonance Imaging: An Official Journal of the International Society for Magnetic Resonance in Medicine 13(4), 534–546 (2001)
  * [14] Lebel, C., et al.: A review of diffusion mri of typical white matter development from early childhood to young adulthood. NMR in Biomedicine 32(4), e3778 (2019)
  * [15] Madhyastha, T., et al.: Longitudinal reliability of tract-based spatial statistics in diffusion tensor imaging. Human brain mapping 35(9), 4544–4555 (2014)
  * [16] Pagani, E., et al.: A method for obtaining tract-specific diffusion tensor mri measurements in the presence of disease: application to patients with clinically isolated syndromes suggestive of multiple sclerosis. Neuroimage 26(1), 258–265 (2005)
  * [17] Pini, L., et al.: Brain atrophy in alzheimer’s disease and aging. Ageing research reviews 30, 25–48 (2016)
  * [18] Raffelt, D., Tournier, J.D., Fripp, J., Crozier, S., Connelly, A., Salvado, O.: Symmetric diffeomorphic registration of fibre orientation distributions. Neuroimage 56(3), 1171–1180 (2011)
  * [19] Reynolds, J., Grohs, M., Lebel, C.: White matter tractography guides. FigShare. Available at 10, m9 (2019)
  * [20] Reynolds, J.E., Grohs, M.N., Dewey, D., Lebel, C.: Global and regional white matter development in early childhood. Neuroimage 196, 49–58 (2019)
  * [21] Reynolds, J.E., Long, X., Paniukov, D., Bagshawe, M., Lebel, C.: Calgary preschool magnetic resonance imaging (mri) dataset. Data in brief 29, 105224 (2020)
  * [22] Schwarz, C.G., Reid, R.I., Gunter, J.L., Senjem, M.L., Przybelski, S.A., Zuk, S.M., Whitwell, J.L., Vemuri, P., Josephs, K.A., Kantarci, K., et al.: Improved dti registration allows voxel-based analysis that outperforms tract-based spatial statistics. Neuroimage 94, 65–78 (2014)
  * [23] Sexton, C.E., et al.: A meta-analysis of diffusion tensor imaging in mild cognitive impairment and alzheimer’s disease. Neurobiology of aging 32(12), 2322–e5 (2011)
  * [24] Shenton, M.E., et al.: A review of magnetic resonance imaging and diffusion tensor imaging findings in mild traumatic brain injury. Brain imaging and behavior 6(2), 137–192 (2012)
  * [25] Smith, S.M., Kindlmann, G., Jbabdi, S.: Cross-subject comparison of local diffusion mri parameters. In: Diffusion MRI, pp. 209–239. Elsevier (2014)
  * [26] Smith, S.M., et al.: Tract-based spatial statistics: voxelwise analysis of multi-subject diffusion data. Neuroimage 31(4), 1487–1505 (2006)
  * [27] Suarez, R.O., et al.: Automated delineation of white matter fiber tracts with a multiple region-of-interest approach. Neuroimage 59(4), 3690–3700 (2012)
  * [28] Van Essen, D.C., et al.: The wu-minn human connectome project: an overview. Neuroimage 80, 62–79 (2013)
  * [29] Wasserthal, J., Neher, P., Maier-Hein, K.H.: Tractseg-fast and accurate white matter tract segmentation. NeuroImage 183, 239–253 (2018)
  * [30] Yushkevich, P.A., et al.: Structure-specific statistical mapping of white matter tracts using the continuous medial representation. In: 2007 IEEE 11th International Conference on Computer Vision. pp. 1–8. IEEE (2007)
  * [31] Yushkevich, P.A., et al.: Structure-specific statistical mapping of white matter tracts. In: Visualization and Processing of Tensor Fields, pp. 83–112. Springer (2009)
  * [32] Zalesky, A.: Moderating registration misalignment in voxelwise comparisons of dti data: a performance evaluation of skeleton projection. Magnetic resonance imaging 29(1), 111–125 (2011)
  * [33] Zhao, S., Dong, Y., Chang, E.I., Xu, Y., et al.: Recursive cascaded networks for unsupervised medical image registration. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 10600–10610 (2019)
  * [34] Zhuang, L., et al.: White matter integrity in mild cognitive impairment: a tract-based spatial statistics study. Neuroimage 53(1), 16–25 (2010)