# Building a Large Japanese Web Corpus
for Large Language Models

Naoaki Okazaki22footnotemark: 2, Kakeru Hattori22footnotemark: 2, Hirai
Shota22footnotemark: 2, Hiroki Iida22footnotemark: 2, Masanari
Ohi22footnotemark: 2,
Kazuki Fujii22footnotemark: 2, Taishi Nakamura22footnotemark: 2, Mengsay
Loem22footnotemark: 2, Rio Yokota33footnotemark: 3, Sakae
Mizuki22footnotemark: 2
22footnotemark: 2 Department of Computer Science, School of Computing, Tokyo
Institute of Technology
33footnotemark: 3 Global Scientific Information and Computing Center, Tokyo
Institute of Technology
2-12-1 Ookayama, Meguro-ku, Tokyo, 152-8550 Japan
{okazaki@c, <EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Open Japanese large language models (LLMs) have been trained on the Japanese
portions of corpora such as CC-100, mC4, and OSCAR. However, these corpora
were not created for the quality of Japanese texts. This study builds a large
Japanese web corpus by extracting and refining text from the Common Crawl
archive (21 snapshots of approximately 63.4 billion pages crawled between 2020
and 2023). This corpus consists of approximately 312.1 billion characters
(approximately 173 million pages), which is the largest of all available
training corpora for Japanese LLMs, surpassing CC-100 (approximately 25.8
billion characters), mC4 (approximately 239.7 billion characters) and OSCAR
23.10 (approximately 74 billion characters). To confirm the quality of the
corpus, we performed continual pre-training on Llama 2 7B, 13B, 70B, Mistral
7B v0.1, and Mixtral 8x7B Instruct as base LLMs and gained consistent (6.6–8.1
points) improvements on Japanese benchmark datasets. We also demonstrate that
the improvement on Llama 2 13B brought from the presented corpus was the
largest among those from other existing corpora.

##  1 Introduction

ChatGPT, released by OpenAI in late 2022, established a milestone toward
achieving general-purpose artificial intelligence. Research topics using large
language models (LLMs) attract much attention including chain of thoughts (Wei
et al., 2022; Kojima et al., 2022; Wang et al., 2023a; Trivedi et al., 2023),
instruction tuning (Wang et al., 2023b; c), evaluation (Suzgun et al., 2023;
Zheng et al., 2023), hallucination (Rawte et al., 2023; Zhang et al., 2023),
bias (Ladhak et al., 2023; Feng et al., 2023), detecting generated text
(Mitchell et al., 2023) watermarking (Kirchenbauer et al., 2023), poisoning
(Wan et al., 2023), detecting pre-training data (Shi et al., 2024), unlearning
(Yao et al., 2023), and efficient inference (Dettmers et al., 2023).

While there are various motivations, such as raising the level of research and
development in natural language processing, elucidating the mechanisms of LLM
intelligence, the security risks of relying on a handful of foreign companies,
and achieving responsible artificial intelligence, several Japanese companies
and universities have actively been developing open LLMs that achieve good
performance on Japanese text. However, in many cases, the quality of the
training data for building Japanese LLMs was not satisfactory because they
were mostly developed overseas as a part of multilingual corpora.

Corpus | Size (en) | Size (ja) | Source | Text | Langdet
---|---|---|---|---|---
`statmt.org/ngrams` | 976 BT | — |  2012, 2013 |  WET? |  `cld2`
(Buck et al., 2014) |  Dedup: Remove lines with exact hash values (MurmurHash, 64 bit).
|  Clean: Remove email addresses and HTML fragments; normalize punctuation and
capitalization (Moses truecaser).
CC-100 (CCNet) | 532 BT | 26 BL |  2018 |  WET |  fastText
(Wenzek et al., 2020) |  Dedup: Remove paragraphs with exact hash values (SHA-1, 64 bit).
|  Clean: Use documents with high likelihood computed by the language model
trained on Wikipedia.
C4 | 156 BT | — |  April 2019 |  WET |  `langdetect`
(Raffel et al., 2020) |  Dedup: Remove exact matches in three-sentence units.
|  Clean: Keep only lines ending in a punctuation mark; remove documents with
one or two sentences; remove lines with 4- words; etc.
mC4 | 2,733 BT | 240 BL |  71 months |  WET |  `cld3`
(Xue et al., 2021) |  Dedup: Same as C4
|  Clean: Same as C4, but they use a modified rule for keeping text (because
the punctuation filter of C4 is specific to English): the text must be at
least 200 characters long and contain at least 3 lines.
OSCAR 23.01 | 523 BW | 74 BL |  Nov/Dec 2022 |  WET |  fastText
(Abadji et al., 2022) |  Dedup: TLSH
|  Clean: Remove documents with high likelihood computed by the language
models trained on harmful content or with URLs registered in Blacklists UT1.
Pile-CC | 2,312 BL | — |  2013–2020 |  `jusText` |  `cld2`
(Gao et al., 2020) |  Dedup: MinHash (Jaccard coefficient threshold is approximately 0.5).
|  Clean: Use documents similar to those in OpenWebText2 (documents with a
Reddit score of 3 or higher) based on a fastText classifier.
ROOTS | 484 BB | — |  (539 domains) |  (original) |  fastText
(Laurençon et al., 2023) |  Dedup: SimHashLSH (6-gram)
|  Clean: Remove documents with too few words, with a high percentage of
repeated characters, with a high percentage of repeated words, with too many
emojis, or with too few function words, etc.
RefinedWeb | 600 BT | — |  Until Jun 2023 |  `trafilatura` |  fastText
(Penedo et al., 2023) |  Dedup: MinHash (character 5-gram, approximate threshold 0.8) .
|  Clean: Blacklist URL filter by UT1; remove repetitions; per-document
filter; per-line modification; remove lines matching more than 50 tokens based
on suffix array.
Dolma | 2,415 BT | — |  May 2020 to Jun 2023 |  WET |  fastText
(Soldaini et al., 2023) |  Dedup: Remove documents with identical URLs; remove identical paragraphs within documents.
|  Clean: Paragraph filtering of MassiveWeb (Rae et al., 2021); punctuation
rules of C4; etc.
ClueWeb 22 | *16.7 TT | 3,301 BT |  Search engine |  `BlingFire` |  (original)
(Overwijk et al., 2022) (Non-commercial use) |  Dedup and Clean: Documents were sampled from a commercial search engine based on the distribution of page visits.
RedPajama-Data-v2 | 20.5 TT | — |  All (84 dumps) |  WET |  fastText
|  Dedup: Same as CCNet
|  Clean: Text-quality estimation rules used by C4, RefinedWeb, etc.
This study (raw) | 3,684 BL | 3,684 BL |  2020-40 to 2023-23 |  `trafilatura` |  (original)
|  Dedup and Clean: None.
This study (clean) | 312 BL | 312 BL |  2020-40 to 2023-23 |  `trafilatura` |  (original)
|  Dedup: MinHash (character 5-gram, Jaccard coefficient threshold is
approximately 0.9).
|  Clean: Repetition detection of RefinedWeb and quality filter specially
designed for Japanese text.
Table 1: Representative corpora that can be used to pre-train LLMs. BL, BW,
BT, and TT stand for billion letters, billion words, billion tokens, and
trillion tokens, respectively. Extracted from the corresponding papers,
numbers in English size are not directly comparable because of the difference
in units. The number of ClueWeb22 may include text in other languages. Numbers
in Japanese size are in Unicode characters and are directly comparable. Text,
Langdet, Dedup, and Clean explain methods for text extraction, language
detection, deduplication, and cleaning. WET in text extraction indicates that
the corpus uses the text extraction results distributed by Common Crawl in WET
format.

Table 1 lists representative corpora used for training LLMs. Some popular
corpora such as Pile-CC (Gao et al., 2020), ROOTS (Laurençon et al., 2023),
Dolma (Soldaini et al., 2023), and RedPajama-
Data-v2111https://www.together.ai/blog/redpajama-data-v2 cannot be used for
training Japanese LLMs, including no Japanese text. Therefore, CC-100 (Wenzek
et al., 2020), mC4 (Xue et al., 2021), and OSCAR 23.01 (Abadji et al., 2022)
are candidates for training Japanese LLMs, including a certain amount of
Japanese text and released with permissive licenses. However, the quality of
Japanese text in these corpora is unsatisfactory because they contain noise in
the HTML-to-text conversion (Common Crawl WET format) and because they have
incorporated no special efforts to improve text quality for Japanese. Although
we recognize the importance of efforts in building multilingual corpora, it is
difficult to build a useful and reliable Japanese corpus without the knowledge
of Japanese language.

Therefore, this paper explores a method to construct a large-scale, high-
quality Japanese web corpus that can be used for training Japanese LLMs. The
presented method includes lightweight language detection that boosts the
processing speed of text extraction in the target language. This strategy is
applicable not only to Japanese but also to other languages with less text
than English. In addition, we specially design a filtering method to find
good-quality Japanese text. The corpus developed in this study is made from 21
snapshots of Common Crawl (from CC-MAIN-2020-40 to CC-MAIN-2023-23), and the
size of the corpus after cleaning is 173,350,375 pages and 312,093,428,689
characters. To confirm the quality of the corpus, we perform continual pre-
training on Llama 2 7B, 13B, 70B (Touvron et al., 2023), Mistral 7B v0.1
(Jiang et al., 2023), and Mixtral 8x7B Instruct (Jiang et al., 2024) as base
LLMs. Experimental results demonstrate that continual pre-training
consistently improves the base model’s performance by 6.6–8.1 points on
Japanese benchmark datasets. We also demonstrate that the improvement on Llama
2 13B brought from the presented corpus was the largest among those from other
existing corpora. The models trained on the presented corpus are available on
Hugging Face222https://huggingface.co/tokyotech-llm.

##  2 Related work

A number of large corpora were built from Common
Crawl333https://commoncrawl.org/ archives. Common Crawl is a non-profit
organization that crawls websites and provides their archives. The crawled
data was initially distributed in ARC
format444https://archive.org/web/researcher/ArcFileFormat.php, but since the
summer of 2013, it has been stored in Web ARChive
(WARC)555https://iipc.github.io/warc-specifications/specifications/warc-
format/warc-1.1/ and Web Text (WET) formats. According to Statistics of Common
Crawl Monthly Archives666https://commoncrawl.github.io/cc-crawl-statistics/,
the total amount of accessible archives (data crawled since 2013 till 2023) is
251,325,355,174 pages. Compact Language Detector 2
(`cld2`)777https://github.com/CLD2Owners/cld2 estimated that about 5 percent
of these web pages are written in Japanese.

Table 1 summarizes the corpora derived from Common Crawl (except for
ClueWeb22, where documents are sampled from a commercial search engine). All
permissive corpora that contain Japanese text rely on the data in WET format
(HTML pages were converted to text by Common Crawl). The advantage of
processing data in WET format is the reduction of processing time and data
transfer size. Wenzek et al. (2020) proposed a pipeline to extract
multilingual text from Common Crawl to train a multilingual BERT model (XLM-R)
(Conneau et al., 2020), and released its implementation
(CCNet)888https://github.com/facebookresearch/cc_net and data
(CC-100)999https://huggingface.co/datasets/cc100. Xue et al. (2021)
constructed mC4, a multilingual extension of C4 (Raffel et al., 2020) to train
mT5, a multilingual variant of T5. Abadji et al. (2022) released OSCAR 23.01
with adult content filtering (based on an $n$-gram language model) and
duplicate removal (based on locality sensitive hashing) added to the previous
release.

However, the corpus construction procedure often introduces aggressive
filtering rules because the WET data usually includes irrelevant text, e.g.,
noises in HTML-to-text conversion, JavaScript codes, navigation menus, and
footers. For example, C4 (Raffel et al., 2020) had to introduce filtering
rules such as “remove lines with the word _JavaScript_ ” (to remove JavaScript
code) and “remove pages with curly brackets `{}`” (because curly brackets are
rarely used in a natural language but often used in a programming language).
Therefore, we use WARC instead of WAT as the source format of Common Crawl
archives.

We follow the design principle of RefinedWeb (Penedo et al., 2023): _scale
first_ (avoid labour intensive human curation process), _strict deduplication_
(respect the value of deduplication for large language models), and _neutral
filtering_ (avoid undesirable biases introduced by machine-learning-based
filtering). The construction procedure of RefinedWeb has much in common with
MassiveWeb (Rae et al., 2021). Collecting web pages with their own web
crawler, Rae et al. (2021) used Google’s SafeSearch filter to remove harmful
content, and extracted text based on the HTML DOM structure. They also removed
documents that contained a lot of repetitions, low-quality text, or duplicated
information. In this study, we extend their pipelines to extract high-quality
Japanese documents efficiently.

We also considered using ClueWeb22 (Overwijk et al., 2022) as a source for
building a large Japanese corpus. Although ClueWeb22 has nice and unique
properties such as real distribution (web pages are extracted by the crawler
of a commercial search engine and follow the distribution of web search
activities), large-scale quality content (content extraction pipeline based on
the search engine’s production-quality content understanding system), and rich
information (annotations of content structure, in-links/out-links of web
pages, visual features of contents, etc), we could not use ClueWeb22 in this
study because an LLM trained on ClueWeb22 cannot be released under an open
license.

##  3 Method

Figure 1: The pipeline for building a large Japanese web corpus.

Figure 1 shows the corpus building pipeline. This pipeline is roughly divided
into the following three stages: (1) Extracting Japanese text from WARC files
in Common Crawl (Section 3.1); (2) Selecting Japanese text carefully with
quality filtering (Section 3.2), deduplication (Section 3.3), and host
filtering (Section 3.4); (3) Cleaning extracted text (Section 3.5).

###  3.1 Extracting Japanese text

Common Crawl snapshots are stored as buckets in Amazon S3 and can be accessed
via S3 or the web server101010https://data.commoncrawl.org/. We extract HTML
content from WARC data using the `WARCIO`
library111111https://github.com/webrecorder/warcio (Step 1, Figure 1). We will
defer to the explanation of Step 2 until later. We then apply
Trafilatura121212https://trafilatura.readthedocs.io/ (Barbaresi, 2021) for
extracting text from HTML markups (Step 3). Step 4 detects the language of a
text based on a linear binary classifier (see Appendix A.1 for the detail) and
extracts Japanese text only.

Because about 5% of the entire Common Crawl is written in Japanese, it is
possible to reduce the processing time of Steps 3 and 4 by 95% if we can
target Japanese web pages only. However, in order to improve the accuracy of
language detection, it is desirable to remove HTML markup before applying the
language detection. Therefore, we should apply text extraction first (Step 3),
followed by language detection (Step 4). To make matters worse, text
extraction takes more processing time than language detection; we cannot
reduce the processing time in this processing order.

This is the motivation behind Step 2: we select web pages that are highly
likely to be in Japanese in a rapid manner without text extraction. A web page
proceeds to Steps 3 and 4 only when one of the following two conditions is
met: (1) the HTML declares the language as Japanese, for example, `<html
lang="ja">`; (2) the precise Japanese detection (Section A.1) recognizes the
content within `<title>` tag as Japanese.

To evaluate the validity of this rule, we compared the results of rapid and
precise language detection. More specifically, assuming that the result of the
precise language detection is correct, we measured the accuracy of rapid
language detection on 10 WARC files131313CC-
MAIN-20230527223515-20230528013515-0000?.warc.gz from 0 to 9 in CC-
MAIN-2023-23. (394,192 pages, 12,052,992,707 bytes in total with gzip
compression). The precision, recall, and F1 score were 0.888, 0.967, and
0.926, respectively. This result indicates that although rapid Japanese
language detection may discard about 3.3% of Japanese web pages, the
relatively high precision reduces the processing time for Steps 3 and 4 for
non-Japanese web pages. In a benchmark experiment using 1 CPU of Intel Xeon
Gold 6130 processor (2.1 GHz), the processing time for Steps 1–4 was about 15
times faster than Steps 1, 3, and 4 (without rapid language detection) because
Step 2 removes a number of web pages from the target.

This stage obtained 2,686,080,919 pages and 3,684,320,219,695 Japanese
characters from 21 snapshots of Common Crawl (from CC-MAIN-2020-40 to CC-
MAIN-2023-23).

###  3.2 Quality filtering

This process (1) removes web pages with many repetitions, (2) selects web
pages that contain good-quality Japanese text, and (3) removes web pages that
may contain harmful expressions. For processing (1), we adopted the rules of
Rae et al. (2021) to remove documents with many duplicated contents (see
Appendix A.2).

As for processing (2), no standard has been established for assessing the
quality of Japanese text. Therefore, we designed several rules for (2)
selecting Japanese text of good quality. This study removes a text that
satisfies either of the following conditions as irrelevant for LLMs
(thresholds in parentheses).

  1. 1.

Number of characters (less than 400 characters)

  2. 2.

Percentage of hiragana characters (less than 0.2)

  3. 3.

Percentage of katakana characters (greater than 0.5)

  4. 4.

Percentage of Japanese characters (hiragana, katakana, kanji, punctuations)
(less than 0.5)

  5. 5.

Average number of characters in a sentence (less than 20 or more than 90)

  6. 6.

Number of characters in the longest sentence (more than 200)

  7. 7.

Percentage of sentences ending with an ellipsis symbol (greater than 0.2)

For example, Rule 2 ensures that a text includes a certain amount of function
words such as _ga_ (subject case marker), _wo_ (object case marker) _ni_
(position case maker roughly corresponding to _to_ in English). Rule 3 rejects
a web page containing a lot of product or service names (e.g., found in SEO
sites), which are often expressed in katakana characters. Rule 7 removes a web
page that looks like an RSS feed (a collection of snippets of other web
pages). These rules were adjusted manually as the authors examined the text to
be deleted and extracted.

Although RefinedWeb used UT1 blocklist141414https://dsi.ut-
capitole.fr/blacklists/ for removing harmful content, its coverage for
Japanese pages may be insufficient. In order to remove web pages that may
contain harmful expressions (3), we manually created a list of NG expressions.
If the percentage of characters that match NG expressions is greater than
0.05, we exclude the text from the corpus.

This quality filtering reduced the size of the corpus to 646,238,066 pages
(1,202,868,044,631 characters). Examining the text before and after the
quality filtering, we observe that web pages that may be unuseful for training
LLMs (e.g., e-commerce sites) have disappeared.

###  3.3 Deduplication

Because Common Crawl visits and crawls the same website multiple times, the
archive includes web pages that are identical or similar because of minor
modifications or reproductions. Lee et al. (2022) reported that deduplication,
removing duplicated text from the corpus, not only reduces memorization of
LLMs but also improves pre-training efficiency.

Therefore, we performed deduplication using MinHash (Broder, 1997) in a
similar manner to Lee et al. (2022). When using MinHash for deduplication, it
is common to create $r$ buckets, where each bucket is a concatenation of $b$
MinHash values, compare $r$ pairs of buckets of two documents, recognize the
two documents as a duplicate if any of bucket pairs are identical. In this
study, we adopted a setting where $b=20,r=40$ so that a pair of documents with
a Jackard coefficient151515In this study, features were constructed with
$5$-grams of characters. of $0.9$ can be approximately detected as a duplicate
with a probability of $92.5$%. When a pair of documents was recognized as a
duplicate, we kept the one crawled recently and removed the older one. In the
snapshot used in this experiment, the non-duplicate rate of web pages
collected between March and June 2023 ranged from 77.8 to 87.9 percent, while
the non-duplicate rate of web pages collected before February 2023 dropped to
less than 40 percent, and the rate of web pages collected around 2020 was less
than 20 percent. This deduplication process reduced the corpus size to
185,766,463 pages (353,191,872,805 characters).

###  3.4 Filtering by hostnames

Even after the quality filtering in Section 3.2, we found some irrelevant
content in the corpus. Therefore, we created a block list consisting of
hostnames that met the following criteria.

  1. 1.

Included in the UT1 blocklist.

  2. 2.

Percentage of pages containing the name of a dating site (greater than 0.001).

  3. 3.

Percentage of pages containing NG expressions (greater than 0.005).

  4. 4.

*wikipedia.org

  5. 5.

*.5ch.net

Although the UT1 blocklist includes some Japanese websites, we introduced
Criteria 2 and 3 to improve the coverage. The authors manually adjusted the
thresholds for the percentage161616We placed emphasis on recall at the expense
of precision to reduce harmful behaviors of trained models. This design
results in including a lot of non-harmful websites in the block list.. We
applied Criterion 4 because we planned to use Wikipedia text extracted from
the dump. This filtering process resulted in a corpus of 173,350,375 pages
(312,674,522,606 characters).

###  3.5 Cleaning extracted text

Sections 3.1 to 3.4 process text at the document level, i.e., filter out
irrelevant documents without altering the text within a document. In this
study, to avoid an unexpected side effect, we minimally edit the text:
punctuation normalization (replacing Western-style punctuations with Japanese-
style ones) and footer trimming (removing footers that were left by
Trafilatura). Refer to Appendix A.3 for the details. The process does not
delete web pages, but the number of characters decreased from 312,674,522,606
to 312,093,428,689.

##  4 Experiments

###  4.1 Models and training data

In order to evaluate the usefulness of the presented corpus, we conducted
continual pre-training of popular LLMs that excel in English171717It may be
straightforward to evaluate the corpus by training LLMs from scratch, but we
chose continual pre-training because the broader goal of this effort was to
build high-performance LLMs that excel in Japanese under the limited amount of
computing budget. We can assess the quality of the corpus by checking
performance gains from the base models and also comparing the performance with
other models.. We used Llama 2 7B, 13B, 70B, Mistral 7B v0.1, and Mixtral 8x7B
Instruct v0.1 as base models and performed continual pre-training of these
base models on the presented corpus to examine whether the corpus improves
their Japanese knowledge and skill. The training data for continual pre-
training was a mixture of the presented corpus, Japanese Wikipedia,
RefinedWeb, and arXiv component in The Pile181818We expected that the
RefinedWeb and arXiv would help maintain the base LLMs’ English knowledge and
skill during continual pre-training. (Gao et al., 2020). Specifically, we
prepared approximately 104.9 BT of training data, assuming a sequence length
of 4,096 tokens and a batch size of 1,024 with 25,000 steps for continuous
pre-training. The ratio of Japanese to English tokens was set to 9:1, with 5%
of the training data being the English text from RefinedWeb, 5% being The
Pile’s arXiv paper text (English), and the remaining 90% being Japanese text.
The breakdown of the Japanese text was about 1.6 BT of Japanese Wikipedia, and
the presented Japanese corpus occupied the rest. We also used AlgebraicStack
(Azerbayev et al., 2024) for Mistral’s continual pre-training. For Mixtral, we
used both AlgebraicStack and The Vault (Manh et al., 2023)191919We completed
the continual pre-training experiments on Llama 2 well before we began with
Mistral 7B. By then, our objective had expanded to enhance logical inference
capabilities in Mistral and Mixtral, leading us to incorporate AlgebraicStack
and The Vault.. We used a different ratio (72:28) of Japanese to English
tokens for continual pre-training on Mixtral 8x7B Instruct202020We changed
this ratio as an attempt to improve the performance in English..

We did not change the architecture of base LLMs; the embedding sizes of
tokens, hidden layers, feed-forward layers, the number of attention heads, and
the number of layers were unchanged from the base models in continual pre-
training. Before continual pre-training, we added Japanese vocabulary to Llama
2 and Mistral 7B tokenizers. The total number of vocabulary was 43,176 for
LLMs based on Llama 2 and 42,800 for those based on Mistral 7B. We employed
AdamW (Loshchilov & Hutter, 2019), with hyperparameters set to
$\beta_{1}=0.9$, $\beta_{2}=0.95$, $\epsilon=1.0\times 10^{-8}$. We used re-
warming and re-decaying the learning rate (Ibrahim et al., 2024) with 1,000
warm-up steps. For Llama 2, the maximum learning rate was set at $1.0\times
10^{-4}$ with a decay rate of $1/30$, while for Mistral and Mixtral, it was
$2.0\times 10^{-5}$ with a decay rate of $1/10$. The batch size was 1,024. We
used $0.1$ for weight decay and 1.0 for gradient clipping. In addition, we
used Flash Attention (Dao et al., 2022) to improve computational and memory
efficiency.

In order to examine the impact of a corpus on the performance of LLMs, we
performed continual pre-training on ClueWeb22 and llm-jp-corpus
v1.0.1212121https://github.com/llm-jp/llm-jp-corpus as well as the presented
corpus. We also compared our LLMs to others with similar numbers of
parameters: CyberAgentLM2 7B (CALM2 7B, a Japanese LLM trained from scratch);
Japanese-StableLM-Base-Beta-7B (JSLMB 7B, a continual pre-training LLM on
Llama 2 7B); Youri 7B (a continual pre-training LLM on Llama 2 7B); Qwen 7B;
Nekomata 7B (a continual pre-training LLM on Qwen 7B); Japanese Stable LM Base
Gamma 7B (JSLMG 7B, a continual pre-training LLM on Mistral 7B); Qwen 14B;
KARAKURI LM 70B (Karakuri 70B, a continual pre-training LLM on Llama 2 70B);
Japanese-StableLM-Base-Beta-70B (JSLMB 70B, a continual pre-training LLM on
LLama 2 70B); and Qwen 72B. Refer to Appendix A.4 for the complete list of
URLs of the LLMs used in the experiments.

###  4.2 Evaluation Dataset

We used llm-jp-eval222222https://github.com/llm-jp/llm-jp-eval v1.0.0 and lm-
evaluation-harness232323https://github.com/Stability-AI/lm-evaluation-
harness/tree/jp-stable commit #9b42d41 as Japanese evaluation benchmarks. llm-
jp-eval is a benchmark consisting of tasks for multiple choice (MC) question
answering, open question answering (QA), reading comprehension (RC), and
natural language inference (NLI) tasks. JCommonsenseQA (Kurihara et al., 2022)
is employed for MC, JEMHopQA (Ishii et al., 2023) and NIILC (Sekine, 2003) for
QA, JSQuAD (Kurihara et al., 2022) for RC. We decided to exclude the NLI
dataset from the benchmark because the distribution of ground-truth NLI labels
is highly imbalanced, which made the evaluation unstable242424An LLM may
obtain a high score only if the model happens to predict the majority label
without understanding and solving the NLI task.. For lm-evaluation-harness, we
used Japanese subsets of XL-Sum (Hasan et al., 2021) for the automatic
summarization task and MGSM (Shi et al., 2023) for the arithmetic reasoning
task, respectively. In addition, WMT 2020 (Barrault et al., 2020) was used for
the Japanese-English and English-Japanese machine translation.

###  4.3 Results

Corpus | QA | QA | QA | RC | Sum | Ja-En | En-Ja | Math | Avg
---|---|---|---|---|---|---|---|---|---
| JCom | JEMHop | NIILC | JSQuAD | XL-Sum | WMT20 | WMT20 | MGSM |
CALM2 7B | 21.98 | 50.47 | 50.66 | 77.99 | 2.33 | 14.99 | 23.45 | 6.00 | 30.98
JSLMB 7B | 36.10 | 44.78 | 44.32 | 83.18 | 21.95 | 12.26 | 19.46 | 7.20 | 33.66
Youri 7B | 46.20 | 47.76 | 49.99 | 85.06 | 19.57 | 19.71 | 26.71 | 6.40 | 37.67
Llama 2 7B | 38.52 | 42.40 | 34.10 | 79.17 | 19.05 | 17.37 | 17.83 | 7.60 | 32.01
\+ _this study_ | 48.08 | 50.78 | 59.68 | 85.73 | 18.30 | 15.11 | 25.10 | 12.40 | 39.40
Qwen 7B | 77.12 | 42.34 | 23.76 | 85.94 | 13.71 | 18.01 | 16.89 | 21.60 | 37.42
Nekomata 7B | 74.17 | 49.28 | 50.22 | 87.07 | 16.76 | 18.15 | 26.73 | 12.40 | 41.85
JSLMG 7B | 73.64 | 46.43 | 55.68 | 89.10 | 22.93 | 15.61 | 23.90 | 16.80 | 43.01
Mistral 7B | 73.01 | 42.45 | 27.22 | 85.63 | 20.06 | 17.33 | 14.05 | 17.60 | 37.17
\+ _this study_ | 85.70 | 49.15 | 55.19 | 88.02 | 19.88 | 16.67 | 24.94 | 22.40 | 45.24
Llama 2 13B | 68.19 | 44.55 | 41.74 | 85.51 | 21.33 | 19.81 | 21.36 | 13.20 | 39.46
\+ ClueWeb22 | 76.76 | 49.29 | 56.02 | 89.55 | 20.15 | 23.32 | 29.08 | 11.60 | 44.47
\+ llm-jp | 74.71 | 50.85 | 60.34 | 90.28 | 21.91 | 17.89 | 25.89 | 18.00 | 44.98
\+ _this study_ | 78.37 | 50.63 | 64.06 | 90.07 | 21.68 | 17.71 | 27.37 | 21.60 | 46.44
Qwen 14B | 88.29 | 42.43 | 32.20 | 89.80 | 18.51 | 22.24 | 22.23 | 38.80 | 44.31
Mixtral 8x7B | 84.00 | 50.33 | 31.07 | 88.08 | 20.02 | 20.63 | 19.56 | 45.20 | 44.86
\+ _this study_ | 92.58 | 58.43 | 56.87 | 91.48 | 25.89 | 20.74 | 27.05 | 43.60 | 52.08
Karakuri 70B | 85.79 | 51.25 | 57.13 | 91.00 | 14.64 | 21.13 | 25.40 | 27.20 | 46.69
JSLMB 70B | 91.15 | 49.25 | 60.42 | 91.92 | 25.73 | 23.35 | 27.65 | 41.60 | 51.38
Llama 2 70B | 86.86 | 46.56 | 52.56 | 90.80 | 23.61 | 23.98 | 26.43 | 35.60 | 48.30
\+ _this study_ | 93.48 | 62.90 | 69.60 | 91.76 | 22.66 | 22.98 | 30.43 | 48.40 | 55.28
Qwen 72B | 92.94 | 55.66 | 45.18 | 91.59 | 21.79 | 23.56 | 25.61 | 63.20 | 52.44
Table 2: Benchmark evaluation on Japanese tasks. A horizontal line groups LLMs
with the same number of parameters. A horizontal dash line groups LLMs from
the same base model. A bold number indicates the maximum value in the LLMs
with the same number of parameters.

Table 2 reports the performance of LLMs on the Japanese benchmark datasets.
The row “+ _this study_ ” shows the performance of the LLM after we applied
continual pre-training to the base model on the presented corpus. The rows “+
_this study_ ” demonstrate that the presented corpus consistently improves the
performance of the base models via continual pre-training by 6.6–8.1 points on
average. The models built by this study outperform other models built from
scratch or continual pre-training, establishing the state-of-the-art
performance in each model size (7B, 13B, 8x7B, and 70B). This fact shows the
usefulness of the presented corpus in building LLMs that excel in Japanese.

We can compare different Japanese corpora in continual pre-training of Llama 2
13B. All corpora (ClueWeb22, llm-jp-corpus v1.0.1, and presented corpus)
improved the base model’s performance by 5.0–7.0 points, which verifies the
effectiveness of the continual pre-training. The presented corpus yielded the
best improvement (7.0 points) of all corpora; in particular, it drastically
enhances the performance of question answering (JCom, JEMHop, NIILC), reading
comprehension (JSQuAD), and arithmetic reasoning (MGSM). Although we did not
incorporate a special effort to improve these tasks, the model acquired
knowledge about Japan and the Japanese language from the presented corpus. In
contrast, the presented corpus did not improve the summarization (XL-Sum)
task. This trend is observed across other base models, except for Mixtral 8x7B
Instruct, potentially indicating that adding Japanese vocabulary may have a
detrimental effect on this task. In addition, we found that the continual pre-
training on the presented corpus improved the performance in English-Japanese
translation but degraded that in the reversed direction (Japanese-English
translation). We will explore mitigation of this phenomenon in the future,
e.g., by changing the ratio of Japanese to English tokens in the training data
and promoting language transfer using an English-Japanese parallel corpus
during continual pre-training.

##  5 Conclusion

In this paper, we built a large Japanese web corpus by extracting and refining
text from the Common Crawl archive (21 snapshots of approximately 63.4 billion
pages crawled between 2020 and 2023). This corpus consists of approximately
312.1 billion characters (approximately 173 million pages), which is the
largest of all available training corpora for Japanese LLMs. We confirmed the
usefulness of the corpus by performing continual pre-training on Llama 2 7B,
13B, 70B, Mistral 7B v0.1, and Mixtral 8x7B Instruct as base LLMs. The
experiments demonstrated consistent (6.6–8.1 points) improvements in Japanese
benchmark datasets, and established the state-of-the-art performance in each
model size (7B, 13B, 8x7B, and 70B). We also observed that the improvement on
Llama 2 13B brought from the presented corpus was the largest among those from
other existing corpora.

Future directions include efforts towards the safety of LLMs, such as reducing
harmful generations (e.g., discrimination, exclusion, toxicity,
hallucination). Currently, we only use the lists of NG expressions and
hostnames, but it is desirable to establish more robust filtering methods to
remove harmful text for pre-training Japanese LLMs. In addition, although our
study focused on the continual pre-training setting, we want to evaluate the
presented corpus by training Japanese LLMs from scratch. Although we evaluated
the LLMs in downstream tasks such as question-answering and summarization, it
is questionable whether this can measure the “general intelligence” of an LLM.
At the same time, training an LLM on a pre-training corpus requires huge
computations. Therefore, we want to explore a lightweight method for assessing
the effectiveness of pre-training corpora without building LLMs.

##  6 Limitations

We could not present an ablation study of each step in Figure 1 because of a
large amount of computational resources required to train LLMs. Instead, we
reported the performance of each step of the corpus construction procedure in
Section 3.

This study focuses on Japanese, and the construction of corpora for other
languages is outside the scope. For English, it may be a good strategy to make
use of good quality corpora that have already been developed. Although some
ideas in this paper could be helpful in building corpora for other languages,
for example, reducing processing time with rapid language detection, we
believe that it would be better for researchers from different countries to
share the task of building corpora for their own, as assessing text quality
requires the knowledge of the language.

##  7 Ethics statement

Article 30-4 of the Copyright Act in Japan permits to use of a copyrighted
work without the permission of the copyright holder as long as the use is not
a person’s purpose to personally “enjoy” or cause another person to “enjoy”
the work. This provides the justification for building the presented corpus
and LLMs. To the best of our knowledge, we have taken every measure to keep
the corpus as non-toxic and unbiased as possible, but we are unaware of any
direct ethical consequences caused by the LLMs trained on the presented
corpus.

##  8 Reproducibility statement

All models developed in this study (continual pre-training on Llama 2 7B, 13B,
70B, Mistral 7B v0.1, and Mixtral 8x7B Instruct) have already been released on
Hugging Face. The benchmark datasets used in this study are also publicly
available. Therefore, it is straightforward to reproduce our experimental
results reported in Table 2.

## Acknowledgements

This paper is based on results obtained from a project, JPNP18002,
commissioned by the New Energy and Industrial Technology Development
Organization (NEDO). In addition, the experiments of continual pre-training of
LLMs was supported by the “Support Program for Building Large Language Models”
of the AI Bridging Cloud Infrastructure (ABCI) developed and operated by the
National Institute of Advanced Industrial Science and Technology (AIST). We
used the datasets and findings released by the Japanese LLM Study Group (LLM-
jp) in the evaluation experiments. We also received suggestions on
tokenization from Tatsuya Hiraoka of Fujitsu Ltd.

## References

  * Abadji et al. (2022) Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot.  Towards a cleaner document-oriented multilingual crawled corpus.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pp. 4344–4355, 2022.  URL https://aclanthology.org/2022.lrec-1.463.
  * Azerbayev et al. (2024) Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck.  Llemma: An open language model for mathematics.  In _the Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=4WnqRR915j.
  * Barbaresi (2021) Adrien Barbaresi.  Trafilatura: A web scraping library and command-line tool for text discovery and extraction.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations_ , pp. 122–131, 2021.  URL https://aclanthology.org/2021.acl-demo.15.
  * Barrault et al. (2020) Loïc Barrault, Magdalena Biesialska, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubešić, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri.  Findings of the 2020 conference on machine translation (WMT20).  In _Proceedings of the Fifth Conference on Machine Translation_ , pp. 1–55, 2020.  URL https://aclanthology.org/2020.wmt-1.1.
  * Broder (1997) Andrei Z Broder.  On the resemblance and containment of documents.  In _Proceedings of the Compression and Complexity of Sequences 1997_ , pp. 21, 1997.
  * Buck et al. (2014) Christian Buck, Kenneth Heafield, and Bas van Ooyen.  N-gram counts and language models from the Common Crawl.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation_ , pp. 3579–3584, 2014.  URL http://www.lrec-conf.org/proceedings/lrec2014/pdf/1097_Paper.pdf.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 8440–8451, 2020.  URL https://aclanthology.org/2020.acl-main.747.
  * Dao et al. (2022) Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.  FlashAttention: Fast and memory-efficient exact attention with IO-awareness.  In _Advances in Neural Information Processing Systems_ , volume 35, pp. 16344–16359, 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/67d57c32e20fd0a7a302cb81d36e40d5-Paper-Conference.pdf.
  * Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.  QLoRA: Efficient finetuning of quantized LLMs.  In _Advances in Neural Information Processing Systems_ , volume 36, pp. 10088–10115, 2023.  URL https://proceedings.neurips.cc/paper_files/paper/2023/file/1feb87871436031bdc0f2beaa62a049b-Paper-Conference.pdf.
  * Feng et al. (2023) Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov.  From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 11737–11762, 2023.  URL https://aclanthology.org/2023.acl-long.656.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy.  The Pile: An 800GB dataset of diverse text for language modeling.  arXiv:2101.00027, 2020.
  * Hasan et al. (2021) Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar.  XL-sum: Large-scale multilingual abstractive summarization for 44 languages.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pp. 4693–4703, 2021.  URL https://aclanthology.org/2021.findings-acl.413.
  * Ibrahim et al. (2024) Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats L. Richter, Quentin Anthony, Timothée Lesort, Eugene Belilovsky, and Irina Rish.  Simple and scalable strategies to continually pre-train large language models.  arXiv:2403.08763, 2024.
  * Ishii et al. (2023) Ai Ishii, Naoya Inoue, and Satoshi Sekine.  Construction of a Japanese multi-hop QA dataset for QA systems capable of explaining the rationale.  In _the 29th Annual Meeting of Japanese Association for Natural Language Processing (NLP2023)_ , pp. 2088–2093, 2023.  URL https://www.anlp.jp/proceedings/annual_meeting/2023/pdf_dir/Q8-14.pdf.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mistral 7B.  arXiv:2310.06825, 2023.
  * Jiang et al. (2024) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mixtral of experts.  arXiv:2401.04088, 2024.
  * Kirchenbauer et al. (2023) John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein.  A watermark for large language models.  In _Proceedings of the 40th International Conference on Machine Learning_ , volume 202, pp. 17061–17084, 2023.  URL https://proceedings.mlr.press/v202/kirchenbauer23a.html.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.  Large language models are zero-shot reasoners.  In _Advances in Neural Information Processing Systems_ , volume 35, pp. 22199–22213, 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/8bb0d291acd4acf06ef112099c16f326-Paper-Conference.pdf.
  * Kurihara et al. (2022) Kentaro Kurihara, Daisuke Kawahara, and Tomohide Shibata.  JGLUE: Japanese general language understanding evaluation.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pp. 2957–2966, 2022.  URL https://aclanthology.org/2022.lrec-1.317.
  * Ladhak et al. (2023) Faisal Ladhak, Esin Durmus, Mirac Suzgun, Tianyi Zhang, Dan Jurafsky, Kathleen McKeown, and Tatsunori Hashimoto.  When do pre-training biases propagate to downstream tasks? a case study in text summarization.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pp. 3206–3219, 2023.  URL https://aclanthology.org/2023.eacl-main.234.
  * Laurençon et al. (2023) Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, et al.  The BigScience ROOTS corpus: A 1.6TB composite multilingual dataset.  arXiv:2303.03915, 2023.
  * Lee et al. (2022) Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini.  Deduplicating training data makes language models better.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 8424–8445, 2022.  URL https://aclanthology.org/2022.acl-long.577.
  * Loshchilov & Hutter (2019) Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=Bkg6RiCqY7.
  * Manh et al. (2023) Dung Nguyen Manh, Nam Le Hai, Anh T. V. Dau, Anh Minh Nguyen, Khanh Nghiem, Jin Guo, and Nghi D. Q. Bui.  The vault: A comprehensive multilingual dataset for advancing code understanding and generation.  In _Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)_ , pp. 219–244, 2023.  URL https://aclanthology.org/2023.nlposs-1.25.
  * Mitchell et al. (2023) Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn.  DetectGPT: Zero-shot machine-generated text detection using probability curvature.  In _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pp. 24950–24962, 2023.  URL https://proceedings.mlr.press/v202/mitchell23a.html.
  * Overwijk et al. (2022) Arnold Overwijk, Chenyan Xiong, Xiao Liu, Cameron VandenBerg, and Jamie Callan.  ClueWeb22: 10 billion web documents with visual and semantic information.  arXiv:2211.15848, 2022.
  * Penedo et al. (2023) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay.  The RefinedWeb dataset for Falcon LLM: Outperforming curated corpora with web data, and web data only.  arXiv:2306.01116, 2023.
  * Rae et al. (2021) Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving.  Scaling language models: Methods, analysis & insights from training Gopher.  arXiv:2112.11446, 2021.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67, 2020.  URL http://jmlr.org/papers/v21/20-074.html.
  * Rawte et al. (2023) Vipula Rawte, Swagata Chakraborty, Agnibh Pathak, Anubhav Sarkar, S.M Towhidul Islam Tonmoy, Aman Chadha, Amit Sheth, and Amitava Das.  The troubling emergence of hallucination in large language models - an extensive definition, quantification, and prescriptive remediations.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pp. 2541–2573, 2023.  URL https://aclanthology.org/2023.emnlp-main.155.
  * Sekine (2003) Satoshi Sekine.  Development of a question answering system for encyclopedias.  _the 9th Annual Meeting of Japanese Association for Natural Language Processing (NLP2003)_ , pp. 637–640, 2003.  URL https://www.anlp.jp/proceedings/annual_meeting/2003/pdf_dir/C7-6.pdf.
  * Shi et al. (2023) Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei.  Language models are multilingual chain-of-thought reasoners.  In _the Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=fR3wGCk-IXp.
  * Shi et al. (2024) Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer.  Detecting pretraining data from large language models.  In _the Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=zWqr3MQuNs.
  * Soldaini et al. (2023) Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Khyathi Chandu, Jennifer Dumas, Li Lucy, Xinxi Lyu, Ian Magnusson, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Pete Walsh, Hannaneh Hajishirzi, Noah A. Smith, Luke Zettlemoyer, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo.  Dolma: An open corpus of 3 trillion tokens for language model pretraining research.  Technical report, Allen Institute for AI, 2023.  URL https://allenai.github.io/dolma/docs/assets/dolma-datasheet-v0.1.pdf.
  * Suzgun et al. (2023) Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei.  Challenging BIG-bench tasks and whether chain-of-thought can solve them.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pp. 13003–13051, 2023.  URL https://aclanthology.org/2023.findings-acl.824.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models.  arXiv:2307.09288, 2023.
  * Trivedi et al. (2023) Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal.  Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 10014–10037, 2023.  URL https://aclanthology.org/2023.acl-long.557.
  * Wan et al. (2023) Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein.  Poisoning language models during instruction tuning.  In _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pp. 35413–35425, 2023.  URL https://proceedings.mlr.press/v202/wan23b.html.
  * Wang et al. (2023a) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.  Self-consistency improves chain of thought reasoning in language models.  In _the Eleventh International Conference on Learning Representations_ , 2023a.  URL https://openreview.net/forum?id=1PL1NIMMrw.
  * Wang et al. (2023b) Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi.  How far can camels go? Exploring the state of instruction tuning on open resources.  In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2023b.  URL https://openreview.net/forum?id=w4zZNC4ZaV.
  * Wang et al. (2023c) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi.  Self-instruct: Aligning language models with self-generated instructions.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 13484–13508, 2023c.  URL https://aclanthology.org/2023.acl-long.754.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou.  Chain-of-thought prompting elicits reasoning in large language models.  In _Advances in Neural Information Processing Systems_ , volume 35, pp. 24824–24837, 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf.
  * Wenzek et al. (2020) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave.  CCNet: Extracting high quality monolingual datasets from web crawl data.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pp. 4003–4012, 2020.  URL https://aclanthology.org/2020.lrec-1.494.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 483–498, 2021.  URL https://aclanthology.org/2021.naacl-main.41.
  * Yao et al. (2023) Yuanshun Yao, Xiaojun Xu, and Yang Liu.  Large language model unlearning.  In _Socially Responsible Language Modelling Research_ , 2023.  URL https://openreview.net/forum?id=wKe6jE065x.
  * Zhang et al. (2023) Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A. Smith.  How language model hallucinations can snowball.  arXiv:2305.13534, 2023.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.  Judging LLM-as-a-judge with MT-bench and chatbot arena.  In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2023.  URL https://openreview.net/forum?id=uccHPGDlao.

##  Appendix A Appendix

###  A.1 Precise language detection

Following the pipeline of RefinedWeb (Penedo et al., 2023), we built a
language detector with a linear discriminator using character $n$-grams as
features. We used multilingual Wikipedia texts as the training data for the
linear discriminator, and the feature space was constructed from the training
data with character unigrams, bi-grams, and trigrams that satisfy one of the
following criteria.

  1. 1.

Within the top 400,000 occurrences in the training data for all languages.

  2. 2.

Within the top 400,000 occurrences in the training data of Japanese.

  3. 3.

Within the top 100,000 occurrences in Chinese training data.

  4. 4.

Within the top 10,000 occurrences in the training data for each language.

If only Criterion (1) was used, there would be a possibility that features
related to Japanese characters would not be obtained sufficiently. Therefore,
Criterion (2) aimed to obtain features specific to Japanese. Criterion (3)
made it easier to distinguish between Japanese and Chinese, which share
Chinese characters. We aimed to stabilize the detection results for texts in
languages with insufficient training data using Criterion (4). The total
number of distinct features was 821,484.

For training the linear discriminator, we used the dump of Wikipedia
CirrusSearch252525https://dumps.wikimedia.org/other/cirrussearch/ (as of June
12, 2023). In training the language detector, we reduced the amount of
training data to 1/20th of the dump because the total amount was too large for
training the discriminator. We sampled 100,000 instances in the remaining data
for development and evaluation sets. We trained an L2-regularized L2-loss
Support Vector Machine on the training data using
`LIBLINEAR`262626https://www.csie.ntu.edu.tw/~cjlin/liblinear/ 2.46. The
regularization coefficient was set to $C=10$ empirically by the search on the
development set. We confirmed that the performance of this classifier was
quite good: 0.996 F1 score on the test set and 0.989 F1 score on whatlang-
corpora272727https://github.com/whatlang/whatlang-corpora.

###  A.2 Rules for removing repetitions

The removal of documents that contain many repetitions aims to filter out
documents that do not seem to have useful information and to prevent the
behavior of LLMs from repeatedly generating the same words. Specifically, we
follow the rules of Rae et al. (2021) and remove a document when the
percentage of any of the following exceeds the threshold (shown in
parentheses).

  1. 1.

Number of lines duplicated in other lines / total number of all lines (0.30)

  2. 2.

Number of paragraphs that are duplicates of other paragraphs / total number of
paragraphs (0.30)

  3. 3.

Number of characters that appear in other lines / total number of all
characters (0.20)

  4. 4.

Number of characters in paragraphs that appear in other paragraphs / total
number of characters (0.20)

  5. 5.

Number of occurrences of the most frequent 2-gram / total number of
occurrences of all 2-grams (0.20)

  6. 6.

Number of occurrences of the most frequent 3-gram / total number of
occurrences of all 3-grams (0.18)

  7. 7.

Number of occurrences of the most frequent 4-gram / total number of
occurrences of all 4-grams (0.16)

  8. 8.

Total number of occurrences of 5-grams appearing more than once / Total number
of occurrences of all 5-grams (0.15)

  9. 9.

Total number of occurrences of 6-grams appearing more than once / Total number
of occurrences of all 6-grams (0.14)

  10. 10.

Total number of occurrences of 7-grams appearing more than once / Total number
of occurrences of all 7-grams (0.13)

  11. 11.

Total number of occurrences of 8-grams appearing more than once / Total number
of occurrences of all 8-grams (0.12)

  12. 12.

Total number of occurrences of 9-grams appearing more than once / Total number
of occurrences of all 9-grams (0.11)

  13. 13.

Total number of occurrences of 10-grams appearing more than once / Total
number of occurrences of all 10-grams (0.10)

###  A.3 Clearning the text

In order to normalize the punctuation in the corpus to “、” and “。”, we
replaced “，” with “、” and “．” with “。”, respectively, based on the following
rule.

  1. 1.

Check the number of occurrences the symbols “、” and “，” appear in the
document. If “，” appears more often than “、”, replace “，” with “、”. However,
“，” preceding an alphanumeric character is excluded from the replacement with
“、”.

  2. 2.

Check the number of occurrences the symbols “。” and “．” appear in the
document. If “．” appears more often than “。”, replace “．” with “。”. However,
“。” preceding an alphanumeric character is excluded from the replacement with
“．”.

This punctuation normalization process replaced “．” with “。” in 290,318
documents (0.17%) and “，” with “、” in 1,107,319 documents (0.64%).

Although Trafilatura used for text extraction removes the navigation and
footer text of web pages, we sometimes see footer text that have not be
removed by Trafilatura. Therefore, expressions such as ”Trackback list for
this article”, ”All rights reserved”, and ”Click” in the last three lines of
text were removed from the text if they occupy more than 30% of the text in
character. This footer removal process removed footers at the end of
12,617,787 documents (7.3%) of the pages.

###  A.4 List of LLMs used in the experiments

#### Base models

  * •

Llama 2 7B: https://huggingface.co/meta-llama/Llama-2-7b-hf

  * •

Llama 2 13B: https://huggingface.co/meta-llama/Llama-2-13b-hf

  * •

Llama 2 70B: https://huggingface.co/meta-llama/Llama-2-70b-hf

  * •

Mistral 7B v0.1: https://huggingface.co/mistralai/Mistral-7B-v0.1

  * •

Mixtral 8x7B Instruct v0.1:
https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1

#### Other models for comparison

  * •

CyberAgentLM2 7B: https://huggingface.co/cyberagent/calm2-7b

  * •

Japanese-StableLM-Base-Beta-7B:
https://huggingface.co/stabilityai/japanese-stablelm-base-beta-7b

  * •

Rinna Youri 7B: https://huggingface.co/rinna/youri-7b

  * •

Qwen 7B: https://huggingface.co/Qwen/Qwen-7B

  * •

Rinna Nekomata 7B: https://huggingface.co/rinna/nekomata-7b

  * •

Japanese Stable LM Base Gamma 7B:
https://huggingface.co/stabilityai/japanese-stablelm-base-gamma-7b

  * •

Qwen 14B: https://huggingface.co/Qwen/Qwen-14B

  * •

KARAKURI LM 70B:
https://huggingface.co/karakuri-ai/karakuri-lm-70b-v0.1

  * •

Japanese-StableLM-Base-Beta-70B:
https://huggingface.co/stabilityai/japanese-stablelm-base-beta-70b

  * •

Qwen 72B: https://huggingface.co/Qwen/Qwen-72B


11institutetext: Ivan Dimitrijević 22institutetext: Faculty of Mathematics,
University of Belgrade, Studentski trg 16, p.o. box 550, 11000 Belgrade,
Serbia 22email<EMAIL_ADDRESS>33institutetext: Branko
Dragovich 44institutetext: Institute of Physics, University of Belgrade,
Pregrevica 118, 11080 Belgrade, Serbia 44email<EMAIL_ADDRESS>55institutetext: Mathematical Institute of the Serbian Academy of Sciences and
Arts, Kneza Mihaila 36, 11000 Belgrade, Serbia 66institutetext: Zoran Rakić
77institutetext: Faculty of Mathematics, University of Belgrade, Studentski
trg 16, p.o. box 550, 11000 Belgrade, Serbia 77email:
<EMAIL_ADDRESS>88institutetext: Jelena Stanković 99institutetext:
Faculty of Education, University of Belgrade, Kraljice Natalije 43, Belgrade,
Serbia 99email<EMAIL_ADDRESS>
# On a nonlocal de Sitter gravity

Ivan Dimitrijević  Branko Dragovich  Zoran Rakić and Jelena Stanković

###### Abstract

In this paper, we briefly review highlights of nonlocal de Sitter gravity
based on the nonlocal term $\sqrt{R-2\Lambda}\ \mathcal{F}(\Box)\
\sqrt{R-2\Lambda}$ in the Einstein-Hilbert action without matter sector. This
nonlocal de Sitter gravity model has several exact cosmological FLRW solutions
and one of these solutions contains some effects that are usually assigned to
dark matter and dark energy. There are also some other interesting and
promising properties of this kind of gravity nonlocality. We also review some
anisotropic cosmological solutions, and mention the corresponding nonlocal
Schwarzschild-de Sitter metric.

##  1 Introduction

The Standard Model of Cosmology (SMC), or $\Lambda$CDM model, assumes that
general relativity (GR) is valid and applicable not only in the Solar system
but also at the galactic and cosmological scale. Also there is almost common
view that at the current cosmic time the universe approximately consists of 68
% of dark energy (DE), 27 % of dark matter (DM) and only 5 % of well known
visible (ordinary) matter, see Planck2018 . According to $\Lambda$CDM point of
view, there exists a cold dark matter which is responsible for observational
dynamics inside galaxies and their clusters, while dark energy in the form of
the cosmological constant $\Lambda$ acts as a repulsive force and causes
accelerated expansion of the universe. However, despite many efforts to
confirm existence of DM and DE either in the sky or in the laboratory
experiments, they are still not discovered and remain hypothetical.

In addition to problems with DE and DM, general relativity has its own
problems, like singularity of the black holes solutions as well as singularity
at the beginning of the universe, and it means that GR should be modified in
the vicinity of these singularities. It is also well known that GR is
nonrenormalizable theory from the quantum point of view. Although GR is one of
the most beautiful and successful physical theories, which has given several
significant confirmed predictions, it is nevertheless reasonable to doubt in
its validity in description and understanding of all astrophysical and
cosmological gravitational phenomena. Also, there is no a priory reason that
GR should be appropriate gravity theory at all spacetime scales. Keeping all
this in mind, it follows that general relativity is not a final theory of
gravitation and that its extension is desirable.

Since, there is still no physical principle that could indicate in which
direction we should search for right extension of GR, there are many
approaches to its modification, see nojiri ; clifton ; nojiri1 ; capozziello ;
dragovich0  as some reviews. Despite many attempts, there is not yet generally
accepted modification of general relativity. One of the current and attractive
approaches is nonlocal modified gravity, see, e.g. biswas1 ; biswas3 ; biswas4
; biswas5 ; deser ; dimitrijevic13 . In a nonlocal modification, the Einstein-
Hilbert action is extended by a term that contains all higher order degrees of
d’Alembert-Beltrami operator
$\Box=\nabla_{\mu}\nabla^{\mu}=\frac{1}{\sqrt{-g}}\partial_{\mu}(\sqrt{-g}g^{\mu\nu}\partial_{\nu})$
usually in the form of an analytic expression
$F(\Box)=\sum_{n=0}^{+\infty}f_{n}\ \Box^{n},$ see koshelev2 ; koshelev3 ;
eliz ; koivisto ; capozziello2009 ; dimitrijevic10 . There are also nonlocal
gravity models which contain some degrees of $\Box^{-1}$, e.g. see references
maggiore ; barvinsky2012 ; dimitrijevic13 .

One interesting class of nonlocal de Sitter models has the form

$\displaystyle S=\frac{1}{16\pi G}\int d^{4}x\ \sqrt{-g}\
\big{(}R-2\Lambda+P(R)\ \mathcal{F}(\Box)\ Q(R)\big{)},$ (1)

where $\Lambda$ is the cosmological constant, $P(R)$ and $Q(R)$ are some
differentiable functions of the scalar curvature $R$, and $\mathcal{F}(\Box)$
is an analytic function of $\Box$, see dimitrijevic1 ; dimitrijevic2 ;
dimitrijevic3 ; dimitrijevic4 ; dimitrijevic5 ; dimitrijevic6 ; dimitrijevic7
; dimitrijevic8 ; dimitrijevic9 ; dimitrijevic10 ; dimitrijevic11 ;
dimitrijevic12  and references therein. To better explore pure nonlocal
effects, it is intentionally omitted term with matter in (1).

This paper is a brief overview of highlights of nonlocal de Sitter gravity
model (1), where

$\displaystyle
P(R)=Q(R)=\sqrt{R-2\Lambda},\quad\mathcal{F}(\Box)=\sum_{n=1}^{+\infty}\big{(}f_{n}\Box^{n}+f_{-n}\Box^{-n}\big{)}.$
(2)

In particular, we will present several relevant exact vacuum cosmological
solutions and some aspects of the corresponding Schwarzshild-de Sitter metric.
The first step in finding some exact cosmological solutions is solving the
equation $\Box\sqrt{R-2\Lambda}=q\sqrt{R-2\Lambda},$ where
$q=\eta\Lambda\quad(\eta\in\mathbb{R})$ is an eigenvalue and
$\sqrt{R-2\Lambda}$ is an eigenfunction of the operator $\Box.$ One of these
solutions mimics effects that are usually assigned to dark matter and dark
energy. Some other solutions are examples of the nonsingular bounce ones in
flat, closed and open universe. There are also singular and cyclic solutions.
All these cosmological solutions are a result of nonlocality and do not exist
in the local de Sitter case.

##  2 Nonlocal de Sitter gravity: $\sqrt{dS}$ model

###  2.1 Action

Our nonlocal de Sitter gravity model (introduced in dimitrijevic6 ) is given
by the action

$\displaystyle S=\frac{1}{16\pi G}\int d^{4}x\ \sqrt{-g}\ \sqrt{R-2\Lambda}\
F(\Box)\ \sqrt{R-2\Lambda},$ (3)

where $F(\Box)$ is the following formal expansion in terms of the d’Alemberian
$\Box$:

$\displaystyle
F(\Box)=1+\mathcal{F}(\Box)=1+\mathcal{F}_{+}(\Box)+\mathcal{F}_{-}(\Box),$
(4) $\displaystyle\mathcal{F}_{+}(\Box)=\sum_{n=1}^{+\infty}f_{n}\ \Box^{n},\
\mathcal{F}_{-}(\Box)=\sum_{n=1}^{+\infty}f_{-n}\ \Box^{-n}.$ (5)

When $F(\Box)=1$, i.e. $\mathcal{F}(\Box)=0$, then model (3) becomes local de
Sitter one and coincides with Einstein-Hilbert action with cosmological
constant $\Lambda$:

$\displaystyle S_{0}=\frac{1}{16\pi G}\int d^{4}x\sqrt{-g}\
\sqrt{R-2\Lambda}\sqrt{R-2\Lambda}$ (6) $\displaystyle=\frac{1}{16\pi G}\int
d^{4}x\sqrt{-g}\ (R-2\Lambda).$ (7)

It is worth pointing out that action (3) can be obtained in a very simple and
natural way from action (6) by embedding nonlocal operator (4) within product
$\sqrt{R-2\Lambda}\ \sqrt{R-2\Lambda}.$ By this way, $R$ and $\Lambda$ enter
with the same form into nonlocal version as they are in the local one, and
nonlocal operator $F(\Box)$ is dimensionless. In order to differentiate (3)
from other non-local de Sitter models, we will often denote model (3) as
$\sqrt{dS}.$

###  2.2 Equations of Motion

The equations of motion (EoM) for model (1), when $Q(R)=P(R)$, are given by
(for more details, see dimitrijevic9 ; dimitrijevic13 ):

$\displaystyle G_{\mu\nu}+\Lambda
g_{\mu\nu}-\frac{g_{\mu\nu}}{2}P(R)\mathcal{F}(\Box)P(R)+R_{\mu\nu}W-K_{\mu\nu}W+\frac{1}{2}\Omega_{\mu\nu}=0,$
(8) $\displaystyle W=2P^{\prime}(R)\ \mathcal{F}(\Box)\ P(R),\quad
K_{\mu\nu}=\nabla_{\mu}\nabla_{\nu}-g_{\mu\nu}\Box,$ (9)
$\displaystyle\Omega_{\mu\nu}=\sum_{n=1}^{+\infty}f_{n}\sum_{\ell=0}^{n-1}S_{\mu\nu}(\Box^{\ell}P,\Box^{n-1-\ell}P)-\sum_{n=1}^{+\infty}f_{-n}\sum_{\ell=0}^{n-1}S_{\mu\nu}(\Box^{-\ell-1}P,\Box^{-n+\ell}P),$
(10)

where $S_{\mu\nu}(A,B)$ is defined as

$\displaystyle S_{\mu\nu}(A,B)=g_{\mu\nu}\big{(}\nabla^{\alpha}A\
\nabla_{\alpha}B+A\Box B\big{)}-2\nabla_{\mu}A\ \nabla_{\nu}B.$ (11)

If $P(R)$ is an eigenfunction of the corresponding d’Alembert-Beltrami
operator $\Box$, and consequently also of its inverse $\Box^{-1}$, i.e. holds,
for $q\neq 0,$

$\displaystyle\Box
P(R)=qP(R),\;\Box^{-1}P(R)=q^{-1}P(R),\;\mathcal{F}(\Box)P(R)=\mathcal{F}(q)P(R),$
(12)

then

$\displaystyle
W=2\mathcal{F}(q)P^{\prime}P,\quad\mathcal{F}(q)=\sum_{n=1}^{+\infty}f_{n}\
q^{n}+\sum_{n=1}^{+\infty}f_{-n}\ q^{-n},$ (13) $\displaystyle
S_{\mu\nu}(\Box^{\ell}P,\Box^{n-1-\ell}P)=q^{n-1}S_{\mu\nu}(P,P),$ (14)
$\displaystyle
S_{\mu\nu}(\Box^{-\ell-1}P,\Box^{-n+\ell}P)=q^{-n-1}S_{\mu\nu}(P,P),$ (15)
$\displaystyle S_{\mu\nu}(P,P)=g_{\mu\nu}\big{(}\nabla^{\alpha}P\
\nabla_{\alpha}P+P\Box P\big{)}-2\nabla_{\mu}P\ \nabla_{\nu}P,$ (16)
$\displaystyle\Omega_{\mu\nu}=\mathcal{F}^{\prime}(q)S_{\mu\nu}(P,P),\quad\mathcal{F}^{\prime}(q)=\sum_{n=1}^{+\infty}n\
f_{n}\ q^{n-1}-\sum_{n=1}^{+\infty}n\ f_{-n}\ q^{-n-1}$ (17)

and we get

$\displaystyle\begin{array}[]{l}\displaystyle{G_{\mu\nu}+\Lambda
g_{\mu\nu}-\frac{g_{\mu\nu}}{2}\mathcal{F}(q)P^{2}+2\mathcal{F}(q)R_{\mu\nu}PP^{\prime}-2\mathcal{F}(q)K_{\mu\nu}PP^{\prime}}\\\\[10.0pt]
\hskip
48.36967pt\displaystyle{+\frac{1}{2}\mathcal{F}^{\prime}(q)S_{\mu\nu}(P,P)=0.}\end{array}$
(20)

The last equation can be rewritten to

$\displaystyle\begin{array}[]{l}\displaystyle{\left(G_{\mu\nu}+\Lambda
g_{\mu\nu}\right)\left(1+2\mathcal{F}(q)PP^{\prime}\right)+\mathcal{F}(q)g_{\mu\nu}\left(-\frac{1}{2}P^{2}+PP^{\prime}(R-2\Lambda)\right)}\\\\[7.0pt]
\hskip
76.82243pt\displaystyle{-2\mathcal{F}(q)K_{\mu\nu}PP^{\prime}+\frac{1}{2}\mathcal{F}^{\prime}(q)S_{\mu\nu}(P,P)=0.}\end{array}$
(23)

If $P(R)=\sqrt{R-2\Lambda}$, then $P(R)P^{\prime}(R)=\frac{1}{2}$ and

$\Box\sqrt{R-2\Lambda}=q\,\sqrt{R-2\Lambda}=\eta\,\Lambda\sqrt{R-2\Lambda},\qquad\eta\,\Lambda\neq
0,$ (24)

where $q=\eta\,\Lambda$ and $q^{-1}=\eta^{-1}\Lambda^{-1}$ ($\eta$ –
dimensionless) follows from dimensionality of equalities (24). Since
$P(R)=\sqrt{R-2\Lambda}$, EoM (23) simplify to

$\displaystyle\left(G_{\mu\nu}+\Lambda
g_{\mu\nu}\right)\left(1+\mathcal{F}(q)\right)+\frac{1}{2}\mathcal{F}^{\prime}(q)S_{\mu\nu}(\sqrt{R-2\Lambda},\sqrt{R-2\Lambda})=0.$
(25)

It is clear that EoM (25) are satisfied if

$\displaystyle\mathcal{F}(q)=-1\qquad\qquad\text{and}\qquad\qquad\mathcal{F}^{\prime}(q)=0.$
(26)

It is worth pointing out that not only nonlocal de Sitter model (3) is very
simple and natural but also such are corresponding EoM (25) with respect to
all other models and their EoM that can be derived from (1) with $\Lambda\neq
0$.

Let us remark that nonlocal operator $\mathcal{F}(\Box)$, which satisfies
conditions (26) in model (3), can be taken in the rather general form
dimitrijevic13

$\displaystyle\mathcal{F}(\Box)=\alpha e\
\frac{\Box}{q}\exp{\Big{(}-\frac{\Box}{q}\Big{)}}+\beta e\
\frac{q}{\Box}\exp{\Big{(}-\frac{q}{\Box}\Big{)}},\quad\alpha+\beta=-1,\quad
q=\eta\Lambda.$ (27)

##  3 Cosmological Solutions

###  3.1 Cosmological Solutions in Homogenous and Isotropic Space

At the cosmological scale the universe is homogeneous and isotropic with the
Friedmann-Lema$\mathrm{\hat{i}}$tre-Robertson-Walker (FLRW) metric,
$(c=1,\,k=0,\pm 1)$,

$\displaystyle
ds^{2}=-dt^{2}+a^{2}(t)\left(\frac{dr^{2}}{1-kr^{2}}+r^{2}d\theta^{2}+r^{2}\sin^{2}\theta
d\phi^{2}\right).$ (28)

For the FLRW metric (28) we have the following expressions for scalar
curvature $R$ and operator $\Box$:

$\displaystyle R(t)$
$\displaystyle=6\Big{(}\frac{\ddot{a}}{a}+\big{(}\frac{\dot{a}}{a}\big{)}^{2}+\frac{k}{a^{2}}\Big{)},$
(29) $\displaystyle\Box R$ $\displaystyle=-\frac{\partial^{2}}{\partial
t^{2}}{R}-3H\frac{\partial}{\partial t}{R},$ (30)

where $H=\displaystyle{\frac{1}{a}\frac{da}{dt}\,\equiv\frac{\dot{a}}{a}}$ is
the Hubble parameter.

In the sequel of this subsection, we present some exact cosmological solutions
with $\Lambda\neq 0.$ For some details, see dimitrijevic6 ; dimitrijevic10 ;
dimitrijevic13  and references therein.

#### Solutions of the form $a(t)=A\ t^{n}\ e^{\gamma t^{2}},\ (k=0)$

In the case of next eight flat spaces (k=0), see dimitrijevic13  .

There are two solutions of the form

$\displaystyle a(t)=A\ t^{n}\ e^{\gamma t^{2}},\quad k=0,$ (31)

where $n$ and $\gamma$ are some definite real constants.

The eigenvalue problem

$\Box\sqrt{R-2\Lambda}=q\sqrt{R-2\Lambda},\qquad q=\eta\Lambda\neq 0$ (32)

is satisfied in the following two cases:

$\displaystyle 1.$ $\displaystyle\quad
n=\frac{2}{3},\qquad\gamma=\frac{\Lambda}{14},\qquad q=-\frac{3}{7}\Lambda,$
(33) $\displaystyle 2.$ $\displaystyle\quad
n=0,\qquad\gamma=\frac{1}{6}\Lambda,\qquad q=-\Lambda.$ (34)

Using (33) and (34), we obtain the following two solutions in flat space:

$\displaystyle a_{1}(t)=A\ t^{\frac{2}{3}}\ e^{\frac{\Lambda}{14}t^{2}},\quad
k=0,\quad\mathcal{F}(-\frac{3}{7}\Lambda)=-1,\ \
\mathcal{F}^{\prime}(-\frac{3}{7}\Lambda)=0,$ (35) $\displaystyle a_{2}(t)=A\
e^{\frac{\Lambda}{6}t^{2}},\quad k=0,\quad\mathcal{F}(-\Lambda)=-1,\ \
\mathcal{F}^{\prime}(-\Lambda)=0.$ (36)

#### Solutions of the form $a(t)=(\alpha\ e^{\lambda t}+\beta\ e^{-\lambda
t})^{\gamma}$, ($k=0$)

We have the following two special solutions:

$\displaystyle a_{3}(t)=A\
\cosh^{\frac{2}{3}}{\big{(}\sqrt{\frac{3}{8}\Lambda}\ t\big{)}},\quad
k=0,\quad\mathcal{F}\big{(}\frac{3}{8}\Lambda\big{)}=-1,\
\mathcal{F}^{\prime}\big{(}\frac{3}{8}\Lambda\big{)}=0,$ (37) $\displaystyle
a_{4}(t)=A\ \sinh^{\frac{2}{3}}{\big{(}\sqrt{\frac{3}{8}\Lambda}\
t\big{)}},\quad k=0,\quad\mathcal{F}\big{(}\frac{3}{8}\Lambda\big{)}=-1,\
\mathcal{F}^{\prime}\big{(}\frac{3}{8}\Lambda\big{)}=0.$ (38)

#### Solutions of the form $a(t)=(\alpha\ \sin{\lambda t}+\beta\ \cos{\lambda
t})^{\gamma}$, ($k=0$)

In this case, we obtained the following four solutions:

$\displaystyle a_{5}(t)=A\ \Big{(}1+\sin{\big{(}\sqrt{-\frac{3}{2}\Lambda}\
t\big{)}}\Big{)}^{\frac{1}{3}},\
k=0,\quad\mathcal{F}\big{(}\frac{3}{8}\Lambda\big{)}=-1,\
\mathcal{F}^{\prime}\big{(}\frac{3}{8}\Lambda\big{)}=0,$ (39) $\displaystyle
a_{6}(t)=A\ \Big{(}1-\sin{\big{(}\sqrt{-\frac{3}{2}\Lambda}\
t\big{)}}\Big{)}^{\frac{1}{3}},\
k=0,\quad\mathcal{F}\big{(}\frac{3}{8}\Lambda\big{)}=-1,\
\mathcal{F}^{\prime}\big{(}\frac{3}{8}\Lambda\big{)}=0,$ (40) $\displaystyle
a_{7}(t)=A\ \sin^{\frac{2}{3}}{\big{(}\sqrt{-\frac{3}{8}\Lambda}\
t\big{)}},\quad k=0,\quad\mathcal{F}\big{(}\frac{3}{8}\Lambda\big{)}=-1,\
\mathcal{F}^{\prime}\big{(}\frac{3}{8}\Lambda\big{)}=0,$ (41) $\displaystyle
a_{8}(t)=A\ \cos^{\frac{2}{3}}{\big{(}\sqrt{-\frac{3}{8}\Lambda}\
t\big{)}},\quad k=0,\quad\mathcal{F}\big{(}\frac{3}{8}\Lambda\big{)}=-1,\
\mathcal{F}^{\prime}\big{(}\frac{3}{8}\Lambda\big{)}=0.$ (42)

We have three type of vacuum solutions in the closed and open FLRW space
presented in the next two subsubsections.

#### Cosmological solution of the form $a(t)=A\
e^{\pm\sqrt{\frac{\Lambda}{6}}t}$, ($k=\pm 1$)

In the paper dimitrijevic10 , we presented the following exact solution:

$\displaystyle a_{9}(t)=A\ e^{\pm\sqrt{\frac{\Lambda}{6}}t},\quad k=\pm
1,\quad\mathcal{F}(\frac{1}{3}\Lambda)=-1,\ \
\mathcal{F}^{\prime}(\frac{1}{3}\Lambda)=0,\quad\Lambda>0.$ (43)

Note that this solution is different with respect to the de Sitter one.

#### Solutions of the form $a(t)=(\alpha\ e^{\lambda t}+\beta\ e^{-\lambda
t})^{\gamma}$, ($k=\pm 1$)

When $\alpha\neq 0,\ \beta\neq 0,\ R\neq 2\Lambda,\ q\neq 0$ and $k\neq 0$ we
obtain

$\gamma=\frac{1}{2},\qquad
q=\frac{1}{3}\Lambda,\qquad\lambda=\pm\sqrt{\frac{2}{3}\Lambda},\qquad k\neq
0.$ (44)

The corresponding cosmological solutions are:

$\displaystyle a_{10}(t)=A\
\cosh^{\frac{1}{2}}{\big{(}\sqrt{\frac{2}{3}\Lambda}\ t\big{)}},\quad k=\pm
1,\quad\mathcal{F}\big{(}\frac{1}{3}\Lambda\big{)}=-1,\
\mathcal{F}^{\prime}\big{(}\frac{1}{3}\Lambda\big{)}=0,$ (45) $\displaystyle
a_{11}(t)=A\ \sinh^{\frac{1}{2}}{\big{(}\sqrt{\frac{2}{3}\Lambda}\
t\big{)}},\quad k=\pm 1,\quad\mathcal{F}\big{(}\frac{1}{3}\Lambda\big{)}=-1,\
\mathcal{F}^{\prime}\big{(}\frac{1}{3}\Lambda\big{)}=0.$ (46)

###  3.2 Cosmological Solutions in Anisotropic Space

We are going now to present some solutions for the Bianchi type I anisotropic
metric in the form (for details, we refer to dimitrijevic14 )

$\mathrm{d}s^{2}=-\mathrm{d}t^{2}+a(t)^{2}\left(e^{2\beta_{1}(t)}\mathrm{d}x^{2}+e^{2\beta_{2}(t)}\mathrm{d}y^{2}+e^{2\beta_{3}(t)}\mathrm{d}z^{2}\right)$
(47)

with condition

$\beta_{1}(t)+\beta_{2}(t)+\beta_{3}(t)=0.$ (48)

Let us introduce $\sigma(t)$ as follows:

$\sigma(t)^{2}=\dot{\beta}_{1}(t)^{2}+\dot{\beta}_{2}(t)^{2}+\dot{\beta}_{3}(t)^{2}.$
(49)

One can obtain the following expressions

$\displaystyle R$ $\displaystyle=R_{FLRW}+\sigma^{2},$ (50) $\displaystyle\Box
u(t)$ $\displaystyle=\Box_{FLRW}u(t),$ (51)

where index $FLRW$ denotes quantities corresponding to the $FLRW$ metric with
scale factor $a(t)$ and $k=0$.

Solving the eigenvalue problem

$\displaystyle\Box\ \sqrt{R-2\Lambda}=q\ \sqrt{R-2\Lambda},$ (52)

we obtain several solutions for scale factor $a(t)$ together with function
$\sigma^{2}(t)$, with the corresponding conditions $\mathcal{F}(q)=-1$ and
$\mathcal{F}^{\prime}(q)=0$. For flat space $(k=0)$ and constant $\sigma^{2}$
we have:

$\displaystyle a_{1}(t)=A\ t^{2/3}e^{\frac{\Lambda}{14}(1-\eta)t^{2}},\quad
q=-\frac{3}{7}\Lambda(1-\eta),\quad\sigma^{2}=2\Lambda\eta,$ (53)
$\displaystyle
a_{2}(t)=A\cosh^{\frac{2}{3}}(\sqrt{\frac{3\Lambda}{8}}(1-\eta)\;t),\quad
q=\frac{3\Lambda}{8}(1-\eta)^{2},\quad\sigma^{2}=2\Lambda\eta(2-\eta),$ (54)
$\displaystyle
a_{3}(t)=A\sinh^{\frac{2}{3}}(\sqrt{\frac{3\Lambda}{8}}(1-\eta)\;t),\quad
q=\frac{3\Lambda}{8}(1-\eta)^{2},\quad\sigma^{2}=2\Lambda\eta(2-\eta),$ (55)
$\displaystyle
a_{4}(t)=A\cos^{\frac{2}{3}}(\sqrt{-\frac{3\Lambda}{8}}(1-\eta)\;t),\quad
q=\frac{3\Lambda}{8}(1-\eta)^{2},\quad\sigma^{2}=2\Lambda\eta(2-\eta),$ (56)
$\displaystyle
a_{5}(t)=A\sin^{\frac{2}{3}}(\sqrt{-\frac{3\Lambda}{8}}(1-\eta)\;t),\quad
q=\frac{3\Lambda}{8}(1-\eta)^{2},\quad\sigma^{2}=2\Lambda\eta(2-\eta).$ (57)

Note that the above anisotropic solutions (53) – (57) tend to the
corresponding homogeneous and isotropic ones presented in the previous
subsection, when $\sigma^{2}\to 0\ (\eta\to 0).$

##  4 Concluding Remarks

From the Friedmann equations, it is useful to introduce effective energy
density $\bar{\rho}$ and pressure $\bar{p}$:

$\bar{\rho}(t)=\frac{3}{8\pi
G}\Big{(}\frac{\dot{a}^{2}+k}{a^{2}}-\frac{\Lambda}{3}\Big{)},\quad\bar{p}(t)=\frac{1}{8\pi
G}\Big{(}\Lambda-2\frac{\ddot{a}}{a}-\frac{\dot{a}^{2}+k}{a^{2}}\Big{)},$ (58)

and the equation of state

$\bar{p}(t)=\bar{w}(t)\,\bar{\rho}(t),$ (59)

where $\bar{w}(t)$ is the corresponding effective state parameter. Using (58)
and (59), one can compute $\bar{\rho}(t)$ and $\bar{p}(t)$ for each of the
above cosmological solutions $a(t)$. In this way, it is shown in
dimitrijevic10 ; dimitrijevic13  that solution $a(t)=A\ t^{\frac{2}{3}}\
e^{\frac{\Lambda}{14}t^{2}}$ mimics dark energy and dark matter.

Recently, we started to investigate Schwarzschild-de Sitter spacetime in
$\sqrt{dS}$ nonlocal gravity with metric

$\displaystyle ds^{2}=-A(r)\ dt^{2}+A^{-1}(r)\ dr^{2}+r^{2}\
d\theta^{2}+r^{2}\ \sin^{2}{\theta}\ d\varphi^{2}\ .$ (60)

The corresponding eigenvalue problem

$\displaystyle\Box\sqrt{R(r)-2\Lambda}=\frac{1}{r^{2}}\frac{\partial}{\partial
r}\big{[}r^{2}A(r)\frac{\partial}{\partial
r}\sqrt{R(r)-2\Lambda}\big{]}=q\,\sqrt{R(r)-2\Lambda}$ (61)

is rather nonlinear in $A(r)$, since unknown function $A(r)$ is contained also
in the scalar curvature $R(r)$, i.e
$R(r)=\frac{1}{r^{2}}\frac{\partial^{2}}{\partial
r^{2}}\big{[}r^{2}\big{(}1-A(r)\big{)}\big{]}$. We found an approximative
solution of (61), while finding exact solution remains a challenge, see
dimitrijevic15 .

It is worth noting that action (3) of nonlocal $\sqrt{dS}$ gravity can be
transformed to

$\displaystyle S=\frac{1}{16\pi G}\int[R-2\Lambda+(R-4\Lambda)\
\mathcal{F}(\Box)\ (R-4\Lambda)]\ \sqrt{-g}\ d^{4}x$ (62)

when $|R|\ll|2\Lambda|$, for details, see dimitrijevic12 . Nonlocal gravity
model (62) contains cosmological solution
$a(t)=A\sqrt{t}e^{\frac{\Lambda}{4}t^{2}}$ dimitrijevic11  which mimics an
interplay between radiation $\sqrt{t}$ and dark energy
$e^{\frac{\Lambda}{4}t^{2}}$. It also has some vacuum solutions that change
topology when local de Sitter gravity extends by this nonlocal one
dimitrijevic12 .

It is also worth noting that employment of nonlocal operator in the form of
exponential function comes from string theory, in particular $p$-adic string
theory dragovich1 . Nonlocality in the matter sector is also considered, see,
e.g. dragovich2 ; dragovich3 ; arefeva2 ; koshelev2011 .

Ending, we can conclude that so far obtained results in $\sqrt{dS}$ nonlocal
gravity are encouraging. We plan to explore other aspects of $\sqrt{dS}$, in
particular inflation, gravitational waves and inclusion of matter sector.

## Acknowledgments

This research was partially funded by the Ministry of Education, Science and
Technological Developments of the Republic of Serbia: grant number
451-03-47/2023-01/ 200104 with University of Belgrade, Faculty of Mathematics,
and grant number 451-03-1/2023-01/4 with Faculty of Education, University of
Belgrade. It is also partially supported by the COST Action: CA21136 –
Addressing observational tensions in cosmology with systematics and
fundamental physics (CosmoVerse). I.D. is thankful to prof. V. Dobrev for
hospitality during the conference LT- 15 in Varna.

## References

  * (1) N. Aghanim, et al., _Planck 2018 results. VI. Cosmological parameters_ , _Planck 118 collaboration_ A&A 641 (2020) A6 [arXiv:1807.06209 [astro-ph.CO]].
  * (2) S. Nojiri and S. D. Odintsov, _Unified cosmic history in modified gravity: from $F(R)$ theory to Lorentz non-invariant models_, Phys. Rep. 505 (2011) 59–144 [arXiv:1011.0544v4 [gr-qc]].
  * (3) T. Clifton, P. G. Ferreira, A. Padilla and C. Skordis, _Modified gravity and cosmology_ , Phys. Rep. 513 (2012) 1 [arXiv:1106.2476v2 [astro-ph.CO]].
  * (4) S. Nojiri, S. D. Odintsov and V. K. Oikonomou, _Modified gravity theories on a nutshell: Inflation, bounce and late-time evolution_ , Phys. Rep. 692 (2017) 1–104 [arXiv:1705.11098 [gr-qc]].
  * (5) S. Capozziello and F. Bajardi, _Nonlocal gravity cosmology: An overview_ , Int. J. Mod. Phys. D 31 (2022) 2230009 [arXiv:2201.04512 [gr-qc]].
  * (6) B. Dragovich, _On Nonlocal modified gravity and cosmology_ , Springer Proc. Mathematics $\&$ Statistics 111 (2014) 251–-262.
  * (7) T. Biswas, A. Mazumdar and W. Siegel, _Bouncing universes in string-inspired gravity_ , JCAP 0603 (2006) 009 [arXiv:hep-th/0508194].
  * (8) T. Biswas, E. Gerwick, T. Koivisto and A. Mazumdar, _Towards singularity and ghost free theories of gravity_ , Phys. Rev. Lett. 108 (2012) 031101 [arXiv:1110.5249v2 [gr-qc]].
  * (9) T. Biswas, A. Conroy, A. S. Koshelev and A. Mazumdar, _Generalized gost-free quadratic curvature gravity_ , Class. Quantum Grav. 31 (2014) 015022 [arXiv:1308.2319 [hep-th]].
  * (10) T. Biswas, A. S. Koshelev, A. Mazumdar and S. Yu. Vernov, _Stable bounce and inflation in non-local higher derivative cosmology_ , JCAP 08 (2012) 024 [arXiv:1206.6374v2 [astro-ph.CO]].
  * (11) S. Deser and R. Woodard, _Nonlocal cosmology_ , Phys. Rev. Lett. 99 (2007) 111301 [arXiv:0706.2151 [astro-ph]].
  * (12) I. Dimitrijevic, B. Dragovich, Z. Rakic and J. Stankovic, _Nonlocal de Sitter gravity and its exact cosmological solutions_ , JHEP 12 (2022) 054 [arXiv:2206.13515v1 [gr-qc]].
  * (13) L. Modesto and L. Rachwal, _Super-renormalizable and finite gravitational theories_ , Nucl. Phys. B 889 (2014) 228 [arXiv:1407.8036 [hep-th]].
  * (14) A. S. Koshelev, L. Modesto, L. Rachwal and A. A. Starobinsky, _Occurrence of exact $R^{2}$ inflation in non-local UV-complete gravity_, JHEP 2016 (2016) 67 [arXiv:1604.03127 [hep-th]].
  * (15) L. Buoninfante, A. S. Koshelev, G. Lambiase and A. Mazumdar, _Classical properties of non-local, ghost- and singularity-free gravity_ , JCAP 09 (2018) 034 [arXiv:1802.00399 [gr-qc]].
  * (16) E. Elizalde, E. O. Pozdeeva and S. Yu. Vernov, _Stability of de Sitter solutions in non-local cosmological models_ , PoS QFTHEP2011 138 (2012) [arXiv:1202.0178].
  * (17) A. Conroy, T. Koivisto, A. Mazumdar and A. Teimouri, _Generalised quadratic curvature, non-local infrared modifications of gravity and Newtonian potentials_ , Class. Quantum Grav. 32 (2015) 015024 [arXiv:1406.4998v3 [hep-th]].
  * (18) S. Capozziello, E. Elizalde, Sh. Nojiri and S. D. Odintsov, _Accelerating cosmologies from non-local higher-derivative gravity_ , Phys. Lett. B. 671 (1) (2009) 193-198.
  * (19) I. Dimitrijevic, B. Dragovich, A. S. Koshelev, Z. Rakic and J. Stankovic, _Cosmological solutions of a nonlocal square-root gravity_ , Phys. Lett. B 797 (2019) 134848 [arXiv:1906.07560 [gr-qc]].
  * (20) E. Belgacem, Y. Dirian, S. Foffa and M. Maggiore, _Nonlocal gravity. Conceptual aspects and cosmological predictions_ , JCAP 03 (2018) 002 [arXiv:1712.07066 [hep-th]].
  * (21) A. O. Barvinsky, _Dark energy and dark matter from nonlocal ghost-free gravity theory_ , Phys. Lett. B 710 (2012) 12.
  * (22) I. Dimitrijevic, B. Dragovich, J. Grujic and Z. Rakic, _On modified gravity_ , Springer Proc. Mathematics $\&$ Statistics 36 (2013) 251–259 [arXiv:1202.2352 [hep-th]].
  * (23) I. Dimitrijevic, B. Dragovich, J. Grujic and Z. Rakic, _New cosmological solutions in nonlocal modified gravity_ , Rom. J. Phys. 58 (5-6) (2013) 550–559 [arXiv:1302.2794 [gr-qc]].
  * (24) I. Dimitrijevic, B. Dragovich, J. Grujic and Z. Rakic, _A new model of nonlocal modified gravity_ , Publications de l’Institut Mathematique 94 (108) (2013) 187–196.
  * (25) I. Dimitrijevic, B. Dragovich, J. Grujic and Z. Rakic, _Some power-law cosmological solutions in nonlocal modified gravity_ , Springer Proc. Mathematics $\&$ Statistics 111 (2014) 241–250.
  * (26) I. Dimitrijevic, B. Dragovich, J. Grujic, A. S. Koshelev and Z. Rakic, _Cosmology of non-local $f(R)$ gravity_, Filomat 33 (2019) 1163 [arXiv:1509.04254v2 [hep-th]].
  * (27) I. Dimitrijevic, B. Dragovich, J. Stankovic, A. S. Koshelev and Z. Rakic, _On nonlocal modified gravity and its cosmological solutions_ , Springer Proc. Mathematics $\&$ Statistics 191 (2016) 35–51 [arXiv:1701.02090 [hep-th]].
  * (28) I. Dimitrijevic, B. Dragovich, J. Grujic and Z. Rakic, _Some cosmological solutions of a nonlocal modified gravity_ , Filomat 29 (2015) 619 [arXiv:1508.05583 [hep-th]].
  * (29) I. Dimitrijevic, _Cosmological solutions in modified gravity with monomial nonlocality_ , Appl. Math. Comput. 285 (2016) 195.
  * (30) I. Dimitrijevic, B. Dragovich, Z. Rakic and J. Stankovic, _Variations of infinite derivative modified gravity_ , Springer Proc. Mathematics $\&$ Statistics 263 (2018) 91.
  * (31) I. Dimitrijevic, B. Dragovich, A. S. Koshelev, Z. Rakic and J. Stankovic, _Some cosmological solutions of a new nonlocal gravity model_ , Symmetry 2020 12 (2020) 917 [arXiv:2006.16041 [gr-qc]].
  * (32) I. Dimitrijevic, B. Dragovich, Z. Rakic and J. Stankovic, _New cosmological solutions of a nonlocal gravity model_ , Symmetry 2022 14 (2022) 3 [arXiv:2112.06312 [gr-qc]].
  * (33) I. Dimitrijevic, _Some exact anisotropic cosmological solutions of a simple nonlocal de Sitter gravity_ , to be published in Intern. J. Mod. Phys. A (2023).
  * (34) I. Dimitrijevic, B. Dragovich, Z. Rakic and J. Stankovic, _On the Schwarzschild-de Sitter metric of nonlocal de Sitter gravity_ , to be published in Filomat, arXiv:2212.13896 [gr-qc], (2023).
  * (35) B. Dragovich, A. Yu. Khrennikov, S. V. Kozyrev, I. V. Volovich and E. I. Zelenov, _$p$ -Adic mathematical physics: the first 30 years_, $p$-Adic Numb. Ultrametric Anal. Appl. 9 (2) (2017) 87–121 [arXiv:1705.04758 [math-ph]].
  * (36) _$p$ ‐Adic and adelic cosmology: $p$‐adic origin of dark energy and dark matter_, AIP Conference Proceedings 826 (1), 25-42 (2006).
  * (37) B. Dragovich, _A $p$-Adic matter in a closed universe_, Symmetry 2022 14 (2022) 73 [arXiv:2201.02200 [hep-th]].
  * (38) I. Ya. Aref’eva and I. V. Volovich, _Cosmological daemon_ , JHEP 2011 (2011) 102 [arXiv:1103.0273v2 [hep-th]].
  * (39) A. Koshelev and S. Yu. Vernov, _Analysis of scalar perturbations in cosmological models with a non-local scalar field_ , Class. Quantum Grav. 28 (2011) 085019.