title: MMRA: A Benchmark for Multi-granularity Multi-image Relational Association

authors: S Wu, K Zhu, Y Bai, Y Liang, Y Li, H Wu, J Liu, R Liu… - arXiv preprint arXiv …, 2024

snippet: Given the remarkable success that large visual language models (LVLMs) have achieved in image perception tasks, the endeavor to make LVMLs perceive the world like humans is drawing increasing attention. Current multi-modal benchmarks …


title: Mobile Generative AI: Opportunities and Challenges

authors: Y Zhang, J Zhang, S Yue, W Lu, J Ren, X Shen - IEEE Wireless Communications, 2024

snippet: Recently, generative artificial intelligence (GenAI) has gained significant interest on a global scale, particularly with the explosion of some killer GenAl applications, like ChatGPT. However, due to the excessively large sizes of generative models, most …


title: MobileVLM: A Fast, Reproducible and Strong Vision Language Assistant for Mobile Devices

authors: X Chu, L Qiao, X Lin, S Xu, Y Yang, Y Hu, F Wei… - arXiv preprint arXiv …, 2023

snippet: We present MobileVLM, a competent multimodal vision language model (MMVLM) targeted to run on mobile devices. It is an amalgamation of a myriad of architectural designs and techniques that are mobile-oriented, which comprises a set of language …


title: Model Internals-based Answer Attribution for Trustworthy Retrieval-Augmented Generation

snippet: Ensuring the verifiability of model answers is a fundamental challenge for retrieval-augmented generation (RAG) in the question answering (QA) domain. Recently, self-citation prompting was proposed to make large language models (LLMs) generate citations …


title: Modeling Caption Diversity in Contrastive Vision-Language Pretraining

authors: S Lavoie, P Kirichenko, M Ibrahim, M Assran… - arXiv preprint arXiv …, 2024

snippet: There are a thousand ways to caption an image. Contrastive Language Pretraining (CLIP) on the other hand, works by mapping an image and its caption to a single vector -- limiting how well CLIP-like models can represent the diverse ways to describe an …


title: Modeling the Sacred: Considerations when Using Considerations when Using Religious Texts in Natural Language Processing

authors: B Hutchinson - arXiv preprint arXiv:2404.14740, 2024

snippet: This position paper concerns the use of religious texts in Natural Language Processing (NLP), which is of special interest to the Ethics of NLP. Religious texts are expressions of culturally important values, and machine learned models have a …


title: Modelling Multimodal Integration in Human Concept Processing with Vision-and-Language Models

authors: A Bavaresco, MH Kloots, S Pezzelle, R Fernández - arXiv preprint arXiv:2407.17914, 2024

snippet: Representations from deep neural networks (DNNs) have proven remarkably predictive of neural activity involved in both visual and linguistic processing. Despite these successes, most studies to date concern unimodal DNNs, encoding either …


title: Modelling Visual Semantics via Image Captioning to extract Enhanced Multi-Level Cross-Modal Semantic Incongruity Representation with Attention for Multimodal …

authors: S Aggarwal, A Pandey, DK Vishwakarma - arXiv preprint arXiv:2408.02595, 2024

snippet: … With over 2 terabytes of cleaned CommonCrawl data [45], the masked language model [44] has already been pre-trained on over a hundred languages, including Hindi. We chose this particular variant of [46] for our investigation since the majority …


title: Mono-lingual text reuse detection for the Urdu language at lexical level

authors: A Noreen, I Muneer, RMA Nawab - Engineering Applications of Artificial Intelligence, 2024

snippet: Text reuse is the process of creating new texts from pre-existing ones. In recent years, Urdu Text Reuse Detection (U-TRD) has garnered the attention of researchers due to the ready availability of digital text all over the internet, which can be copied or …


title: mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus

authors: M Futeral, A Zebaze, PO Suarez, J Abadji, R Lacroix… - arXiv preprint arXiv …, 2024

snippet: … This motivated us to collect and release the first large-scale multilingual and multimodal document dataset derived from Common Crawl.Our dataset, multimodal OSCAR (mOSCAR), follows the OSCAR initiative [Ortiz Suárez et al., 2019, Abadji et …


title: MoVA: Adapting Mixture of Vision Experts to Multimodal Context

authors: Z Zong, B Ma, D Shen, G Song, H Shao, D Jiang, H Li… - arXiv preprint arXiv …, 2024

snippet: As the key component in multimodal large language models (MLLMs), the ability of the visual encoder greatly affects MLLM's understanding on diverse image content. Although some large-scale pretrained vision encoders such as vision encoders in …


title: mPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models

authors: P Lin, C Hu, Z Zhang, AFT Martins, H Schütze - Findings of the Association for …, 2024

snippet: Recent multilingual pretrained language models (mPLMs) have been shown to encode strong language-specific signals, which are not explicitly provided during pretraining. It remains an open question whether it is feasible to employ mPLMs to …


title: mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding

authors: A Hu, H Xu, J Ye, M Yan, L Zhang, B Zhang, C Li… - arXiv preprint arXiv …, 2024

snippet: Structure information is critical for understanding the semantics of text-rich images, such as documents, tables, and charts. Existing Multimodal Large Language Models (MLLMs) for Visual Document Understanding are equipped with text recognition …


title: MQM-Chat: Multidimensional Quality Metrics for Chat Translation

authors: Y Li, J Suzuki, M Morishita, K Abe, K Inui - arXiv preprint arXiv:2408.16390, 2024

snippet: The complexities of chats pose significant challenges for machine translation models. Recognizing the need for a precise evaluation metric to address the issues of chat translation, this study introduces Multidimensional Quality Metrics for Chat …


title: MS MARCO Web Search: a Large-scale Information-rich Web Dataset with Millions of Real Click Labels

authors: Q Chen, X Geng, C Rosset, C Buractaon, J Lu, T Shen… - Companion Proceedings of …, 2024

snippet: Recent breakthroughs in large models have highlighted the critical significance of data scale, labels and modals. In this paper, we introduce MS MARCO Web Search, the first large-scale information-rich web dataset, featuring millions of real clicked …


title: Multi Class Depression Detection Through Tweets using Artificial Intelligence

authors: MO Nusrat, W Shahzad, SA Jamal - arXiv preprint arXiv:2404.13104, 2024

snippet: … Typically, GloVe embeddings were pretrained on extensive corpora like Wikipedia or Common Crawl and fine-tuned for specific tasks. 300 LSTM units were used with a dropout rate of 0.4. For depression classification, the dense layer was …


title: Multi-class hate speech detection in the Norwegian language using FAST-RNN and multilingual fine-tuned transformers

authors: E Hashmi, SY Yayilgan - Complex & Intelligent Systems, 2024

snippet: … FastText, a word representation tool developed by Facebook’s research division, offers both unsupervised and supervised modes and features a comprehensive database of 2 million words from Common Crawl, each represented by 300 …


title: Multi-Head Mixture-of-Experts

authors: X Wu, S Huang, W Wang, F Wei - arXiv preprint arXiv:2404.15045, 2024

snippet: Sparse Mixtures of Experts (SMoE) scales model capacity without significant increases in training and inference costs, but exhibits the following two issues: (1) Low expert activation, where only a small subset of experts are activated for …


title: Multi-News+: Cost-efficient Dataset Cleansing via LLM-based Data Annotation

authors: J Choi, J Yun, K Jin, YB Kim - arXiv preprint arXiv:2404.09682, 2024

snippet: … an analysis of undesirable content in the common crawl corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2 …


title: Multi-Reference Benchmarks for Russian Grammatical Error Correction

authors: FP Gomez, A Rozovskaya - Proceedings of the 18th Conference of the European …, 2024

snippet: This paper presents multi-reference benchmarks for the Grammatical Error Correction (GEC) of Russian, based on two existing single-reference datasets, for a total of 7,444 learner sentences from a variety of first language backgrounds. Each …


title: Multi-Stream Keypoint Attention Network for Sign Language Recognition and Translation

authors: M Guan, Y Wang, G Ma, J Liu, M Sun - arXiv preprint arXiv:2405.05672, 2024

snippet: Sign language serves as a non-vocal means of communication, transmitting information and significance through gestures, facial expressions, and bodily movements. The majority of current approaches for sign language recognition (SLR) …


title: Multi-Task Contrastive Learning for 8192-Token Bilingual Text Embeddings

authors: I Mohr, M Krimmel, S Sturua, MK Akram, A Koukounas… - arXiv preprint arXiv …, 2024

snippet: … Seeking further diversity and volume, we utilize Common Crawl data to create two types of pairs: one from web page titles and their contents, and another by mining questionanswer pairs from FAQ and support-related pages. Additionally, we pair …


title: Multidimensional Text Feature Analysis: Unveiling the Veil of Automatically Generated Text

snippet: … Given the widespread use of transformer-based pre-trained models in various downstream tasks and the availability of multilingual datasets in official task data, XLM-RoBERTa[19] is a multilingual version of Roberta that has been pre-trained on …


title: Multilingual Blending: LLM Safety Alignment Evaluation with Language Mixture

authors: J Song, Y Huang, Z Zhou, L Ma - arXiv preprint arXiv:2407.07342, 2024

snippet: … We consider most state-of-the-art LLMs to have trained on these 55 source languages, as these languages are enclosed in the CommonCrawl corpus. All multilingual translations are conducted using Google Translate API3. All codes for …


title: Multilingual Detection of Cyberbullying in Mixed Urdu, Roman Urdu, and English Social Media Conversations

snippet: … The pre-training data is carefully selected from Common Crawl, a vast collection of web pages, to ensure that it’s both high-quality and diverse. This diverse training data allows m-BERT to learn language-agnostic representations, making it easier to …


title: Multilingual Diversity Improves Vision-Language Representations

authors: T Nguyen, M Wallingford, S Santy, WC Ma, S Oh… - arXiv preprint arXiv …, 2024

snippet: … Data We experiment with the medium pool of the DataComp benchmark [14], which consists of 128M image-text pairs randomly sampled from Common Crawl dumps between 2014 and 2022, and deduplicated. Unlike other heavily filtered …


title: Multilingual E5 Text Embeddings: A Technical Report

authors: L Wang, N Yang, X Huang, L Yang, R Majumder, F Wei - arXiv preprint arXiv …, 2024

snippet: This technical report presents the training methodology and evaluation results of the open-source multilingual E5 text embedding models, released in mid-2023. Three embedding models of different sizes (small / base / large) are provided, offering a …


title: Multilingual Fake News Detection: A Study on Various Models and Training

authors: R Chalehchaleh, R Farahbakhsh, N Crespi - … Systems and Applications: Proceedings of the …

snippet: … XLM-RoBERTa model is pre-trained on 2.5 TB of filtered CommonCrawl data containing 100 languages it was introduced in [8]. LASER (Language-Agnostic SEntence Representations)[3] is trained to capture the semantic meaning of sentences …


title: Multilingual Hate Speech Detection: Comparison of Transfer Learning Methods to Classify German, Italian, and Spanish Posts

authors: J Fillies, MP Hoffmann, A Paschke - 2023 IEEE International Conference on Big Data …, 2023

snippet: … This datasetincludes Wikipedia dumps for all supported languages anda substantial 2.5 terabytes of Common Crawl Data collectedfrom the Internet. The ”RoBERTa” component in the model’sname indicates the adherence to the training procedure …


title: Multilingual Large Language Models and Curse of Multilinguality

authors: D Gurgurov, T Bäumel, T Anikina - arXiv preprint arXiv:2406.10602, 2024

snippet: … Common Crawl The Common Crawl corpus is a vast collection of web data accumulated over more than 10 years through web crawling. It includes raw web page data, metadata extracts, and text extracts. This noncurated dataset includes …


title: Multilingual Model Fine-tuning for Sentiment Analysis

authors: ML Elrefai, MI Khalil, HM Abbas - … on Intelligent Computing and Information Systems …, 2023

snippet: … A sizable multilingual language model was created using 2.5 TB of filtered CommonCrawl data. One of the outstanding XLMRoBERTa in low-resource languages, Low-resource languages are languages with limited available data for …


title: Multilingual Models in Neural Machine Translation

snippet: … XGLM [19] uses the decoder-only model architecture similar to that of GPT-3 [6], and was pre-trained on CC100-XL that covers 68 Common Crawl (CC) snapshots and 134 languages. Experiment results show that XGLM (7.5B) outperforms GPT-3 (6.7B) …


title: Multilingual Speech and Language Analysis for the Assessment of Mild Cognitive Impairment: Outcomes from the Taukadial Challenge

authors: PA Perez-Toro, T Arias-Vergara, P Klumpp, T Weise…

snippet: … Our analysis utilized the BERTBase and RoBERTa-Base models, trained on Wikicorpus data spanning 102 languages and CommonCrawl data filtered from 100 languages [16], respectively. For both models, the final layer, comprising 768 units …


title: MultiLS: A Multi-task Lexical Simplification Framework

authors: K North, T Ranasinghe, M Shardlow, M Zampieri - arXiv preprint arXiv:2402.14972, 2024

snippet: Lexical Simplification (LS) automatically replaces difficult to read words for easier alternatives while preserving a sentence's original meaning. LS is a precursor to Text Simplification with the aim of improving text accessibility to various target …


title: Multimodal Large Language Models for Bioimage Analysis

authors: S Zhang, G Dai, T Huang, J Chen - arXiv preprint arXiv:2407.19778, 2024

snippet: Rapid advancements in imaging techniques and analytical methods over the past decade have revolutionized our ability to comprehensively probe the biological world at multiple scales, pinpointing the type, quantity, location, and even temporal …


title: Multimodal Learning for Visual Question Answering using World Knowledge

snippet: Navigating the frontier of the Visual Turing Test, this research delves into multimodal learning to bridge the gap between visual perception and linguistic interpretation, a foundational challenge in artificial intelligence. It scrutinizes the integration of visual …


title: Multimodal predictive model for analyzing news data

snippet: The aim of this thesis is to create a predictive tool utilising a multimodal machine-learning model for the combined textual and visual data from a dataset of news articles to predict the political orientation of news articles. A thorough exploration of multimodal …


title: Multiscale cascaded domain-based approach for Arabic fake reviews detection in e-commerce platforms

authors: N Qandos, G Hamad, M Alharbi, S Alturki, W Alharbi… - Journal of King Saud …, 2024

snippet: Fake reviews in e-commerce can lead to customer deception and financial losses. Despite the importance of fake reviews detection, studies for Arabic language are scarce due to the lack of comprehensive datasets. This study addresses this gap by …


title: Multivariate Log-based Anomaly Detection for Distributed Database

authors: L Zhang, T Jia, M Jia, Y Li, Y Yang, Z Wu - arXiv preprint arXiv:2406.07976, 2024

snippet: … Following this, MultiLog utilizes pre-trained word vectors based on the Common Crawl Corpus with the FastText algorithm[45], which is adept at capturing the inherent relationships among words in natural language. This means each word in a …


title: Musical Word Embedding for Music Tagging and Retrieval

authors: SH Doh, J Lee, D Jeong, J Nam - arXiv preprint arXiv:2404.13569, 2024

snippet: … We used two publicly available embeddings trained on Common Crawl, a large-scale general word corpus, with the GloVe and skipgram … It demonstrates that word embeddings trained with a general word corpus, such as Common Crawl and Wiki …


title: Myanmar XNLI: Building a Dataset and Exploring Low-resource Approaches to Natural Language Inference with Myanmar

snippet: Despite dramatic recent progress in NLP, it is still a major challenge to apply Large Language Models (LLM) to low-resource languages. This is made visible in benchmarks such as Cross-Lingual Natural Language Inference (XNLI), a key task …


title: Named Entity Recognition in Italian Lung Cancer Clinical Reports using Transformers

authors: D Paolo, A Bria, C Greco, M Russano, S Ramella… - 2023 IEEE International …, 2023

snippet: The widespread adoption of electronic health records (EHRs) offers a valuable opportunity to support clinical research by containing crucial patient information, including diagnoses, symptoms, medications, lab tests, and more. Despite the …


title: NATALITY AT RISK? RAISING DOUBTS ON THE EDUCATIONAL IMPORTANCE OF ChatGPT

snippet: Digital tools are part of our daily life. Thus, they also enter educational contexts. With reference to Hannah Arendt’s writings this paper explores if, and to what extent, digital tools can be considered to be helpful and desirable in education. For this …


title: Native Design Bias: Studying the Impact of English Nativeness on Language Model Performance

authors: M Reusens, P Borchert, J De Weerdt, B Baesens - arXiv preprint arXiv:2406.17385, 2024

snippet: Large Language Models (LLMs) excel at providing information acquired during pretraining on large-scale corpora and following instructions through user prompts. This study investigates whether the quality of LLM responses varies depending on …


title: Natural Language Processing Advancements: Breaking Barriers in Human-Computer Interaction

authors: JGC Ramírez - Journal of Artificial Intelligence General science (JAIGS …, 2024

snippet: Natural Language Processing (NLP) advancements have revolutionized human-computer interaction, breaking barriers and opening new frontiers in technology. NLP techniques enable machines to understand, interpret, and generate human …


title: Natural Language Processing Approaches for Monitoring Health Activities

snippet: Monitoring health activities is vital for individual and public health. It identifies trends, detects risks early, and tailors interventions. At the individual level, it informs customized treatments. At the population level, monitoring broad health patterns …


title: Natural Language Processing for Dialects of a Language: A Survey

authors: A Joshi, R Dabre, D Kanojia, Z Li, H Zhan, G Haffari… - arXiv preprint arXiv …, 2024

snippet: State-of-the-art natural language processing (NLP) models are trained on massive training corpora, and report a superlative performance on evaluation datasets. This survey delves into an important attribute of these datasets: the dialect of a language …


title: Natural Language Processing Journal

authors: MN Hoque, U Salma, MJ Uddin, MM Ahamad, S Aktar

snippet: In the sentiment analysis (SA) task, we can obtain a positive or negative-typed comment or feedback from an online user or a customer about any object, such as a movie, drama, food, and others. This user’s sentiment may positively impact various …


title: Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey

authors: DVT Le, L Bigo, M Keller, D Herremans - arXiv preprint arXiv:2402.17467, 2024

snippet: Several adaptations of Transformers models have been developed in various domains since its breakthrough in Natural Language Processing (NLP). This trend has spread into the field of Music Information Retrieval (MIR), including studies …


title: Natural Language Processing Tools for Romanian–Going Beyond a Low-Resource Language

snippet: … Oscar (Open Super-large Crawled Aggregated coRpus) [64] is a multilingual dataset obtained via filtering the common crawl corpus. It encompasses 151 languages. The Romanian sub-corpus contains over 4,5 million documents …


title: Natural Language Processing

authors: CH Focke, RM Sylvester - Statistical Methods in Epilepsy, 2024

snippet: In this chapter, we introduce modern natural language processing (NLP) libraries, techniques and their applications. This chapter focuses on deep learning methods and less on computational linguistics that require nuanced knowledge of linguistics …


title: Navigating Challenges and Technical Debt in Large Language Models Deployment

authors: A Menshawy, Z Nawaz, M Fahmy - Proceedings of the 4th Workshop on Machine …, 2024

snippet: … These models, trained on an internet scale data including common crawl, books, and articles, have acquired an extensive understanding of language patterns, nuances, and contexts. This profound comprehension allows LLMs to perform a …


title: Navigating Data Scarcity using Foundation Models: A Benchmark of Few-Shot and Zero-Shot Learning Approaches in Medical Imaging

authors: S Woerner, CF Baumgartner - arXiv preprint arXiv:2408.08058, 2024

snippet: … LAION-2B contains 2 billion image-text pairs extracted from common crawl [3] and is the English language subset of the larger LAION-5B [12] … https://commoncrawl.org. [4] Alexey Dosovitskiy et al. An Image is Worth 16x16 Words: Transformers for Image …


title: Navigating Multimodal Complexity: Advances in Model Design, Dataset Creation, and Evaluation Techniques

snippet: Ibn Sina, a philosopher of 11th-century Persia, wrote of a `Floating Man'. This man is floating through a void, without the use of his sight or touch or any of the senses which make us human. Yet as he has a human brain this man, according to Ibn Sina …


title: Navigating the ethical landscape behind ChatGPT

authors: L Peng, B Zhao - Big Data & Society, 2024

snippet: … While ChatGPT and similar text-generating AIs claim to be trained on mostly public data such as the Common Crawl archives, rendering … content in public datasets used for AI training, such as the Common Crawl, which is beyond the …


title: Navigating the Legal Landscape: Technical Implementation of Copyright Reservations for Text and Data Mining in the Era of AI Language Models

authors: L Löbling, C Handschigl, K Hofmann, J Schwedhelm - JIPITEC–Journal of Intellectual …, 2023

snippet: … However, limitations arose from the potential omission of specific pages of a website in individual monthly crawls by Common Crawl. Through manual investigation, a section containing the terms of use could be identified for 40 …


title: Ncl-uor at semeval-2024 task 8: Fine-tuning large language models for multigenerator, multidomain, and multilingual machine-generated text detection

authors: F Xiong, T Markchom, Z Zheng, S Jung, V Ojha… - Proceedings of the 18th …, 2024

snippet: SemEval-2024 Task 8 introduces the challenge of identifying machine-generated texts from diverse Large Language Models (LLMs) in various languages and domains. The task comprises three subtasks: binary classification in monolingual …


title: Near to Mid-term Risks and Opportunities of Open Source Generative AI

authors: F Eiras, A Petrov, B Vidgen, CS de Witt, F Pizzati… - arXiv preprint arXiv …, 2024

snippet: In the next few years, applications of Generative AI are expected to revolutionize a number of different areas, ranging from science & medicine to education. The potential for these seismic changes has triggered a lively debate about potential …


title: Need a Programming Exercise Generated in Your Native Language? ChatGPT's Got Your Back: Automatic Generation of Non-English Programming Exercises Using …

authors: M Jordan, K Ly, AG Soosai Raj - Proceedings of the 55th ACM Technical Symposium …, 2024

snippet: Large language models (LLMs) like ChatGPT are changing computing education and may create additional barriers to those already faced by non-native English speakers (NNES) learning computing. We investigate an opportunity for a positive …


title: Negotiating becoming: a Nietzschean critique of large language models

authors: SWS Fischer, B Boer - Ethics and Information Technology, 2024

snippet: Large language models (LLMs) structure the linguistic landscape by reflecting certain beliefs and assumptions. In this paper, we address the risk of people unthinkingly adopting and being determined by the values or worldviews embedded …


title: Nemotron-4 15B Technical Report

authors: J Parmar, S Prabhumoye, J Jennings, M Patwary… - arXiv preprint arXiv …, 2024

snippet: We introduce Nemotron-4 15B, a 15-billion-parameter large multilingual language model trained on 8 trillion text tokens. Nemotron-4 15B demonstrates strong performance when assessed on English, multilingual, and coding tasks: it …


title: Neural Approaches to Decode Semantic Similarities in Spanish Song Lyrics for Enhanced Recommendation Systems

snippet: … In this model, it was found that traditional methods of pre-training, which rely on large amounts of data from sources such as Common Crawl… The mC4 dataset is a multilingual variant of C4, containing natural text in 101 languages from the public …


title: NEURAL MACHINE TRANSLATION BETWEEN MYANMAR AND ENGLISH LANGUAGES

snippet: This study aims to present a thorough overview of cutting-edge machine translation solutions and evaluate their effectiveness in translating between Myanmar and English language pairs. Translating Myanmar poses challenges due to its unique …


title: NEURAL MODELS FOR ONTOLOGY ANNOTATIONS-NEMO

snippet: Words cannot express my gratitude to my advisors Dr. Somya D. Mohanty and Dr. Prashanti Manda for their invaluable supervision and guidance. I am extremely grateful to them for their continuous support, constructive suggestions, and …


title: Neural Network-based Language Modelling for Code-Switching in South African Languages

snippet: Language models, specifically those based on neural network architectures, have become an attractive alternative to classical statistical language models for natural language processing and speech recognition. Naturally, therefore, the field of neural …


title: Neural Networks with Model Compression

authors: B Zhang, T Wang, S Xu, D Doermann - 2024

snippet: … Firstly, the availability of large-scale datasets, such as ImageNet for computer vision or the Common Crawl dataset for natural language processing, has enabled the training of deep neural networks with millions or even billions of parameters …


title: Next word prediction for newspaper article formulation

snippet: Next word prediction describes the process of predicting the next word after every written word in the authoring of texts. It refers to machine learning techniques, which are part of the language modelling domain, a sub-domain of natural language …


title: NLP for The Greek Language: A Longer Survey

authors: K Papantoniou, Y Tzitzikas - arXiv preprint arXiv:2408.10962, 2024

snippet: English language is in the spotlight of the Natural Language Processing (NLP) community with other languages, like Greek, lagging behind in terms of offered methods, tools and resources. Due to the increasing interest in NLP, in this paper we …


title: No Filter: Cultural and Socioeconomic Diversityin Contrastive Vision-Language Models

authors: A Pouget, L Beyer, E Bugliarello, X Wang, AP Steiner… - arXiv preprint arXiv …, 2024

snippet: We study cultural and socioeconomic diversity in contrastive vision-language models (VLMs). Using a broad range of benchmark datasets and evaluation metrics, we bring to attention several important findings. First, the common filtering of training …


title: Non-governmental Governance of Trust on the Internet: WebPKI as Public Good

snippet: … The population of sites we sampled from was indexed by the non-profit foundation Common Crawl. We then employed a script to pull certificate information from our list of URLs. Slightly more than half the sample did not return certificate …


title: Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Models

snippet: This paper identifies a cultural dominance issue within large language models (LLMs) due to the predominant use of English data in model training (eg, ChatGPT). LLMs often provide inappropriate English-culture-related answers that are not relevant to …


title: Notes towards infrastructure governance for large language models

snippet: This paper draws on information infrastructures (IIs) in science and technology studies (STS), as well as on feminist STS scholarship and contemporary critical accounts of digital technologies, to build an initial mapping of the infrastructural …


title: nowhash at SemEval-2024 Task 4: Exploiting Fusion of Transformers for Detecting Persuasion Techniques in Multilingual Memes

authors: AN Chowdhury, M Ptaszynski - Proceedings of the 18th International Workshop on …, 2024

snippet: Nowadays, memes are considered one of the most prominent forms of medium to disseminate information on social media. Memes are typically constructed in multilingual settings using visuals with texts. Sometimes people use memes to …


title: NRK at SemEval-2024 Task 1: Semantic Textual Relatedness through Domain Adaptation and Ensemble Learning on BERT-based models

authors: NT Kiet, D Van Thin - Proceedings of the 18th International Workshop on …, 2024

snippet: This paper describes the system of the team NRK for Task A in the SemEval-2024 Task 1: Semantic Textual Relatedness (STR). We focus on exploring the performance of ensemble architectures based on the voting technique and different …


title: NSINA: A News Corpus for Sinhala

authors: H Hettiarachchi, D Premasiri, L Uyangodage… - arXiv preprint arXiv …, 2024

snippet: The introduction of large language models (LLMs) has advanced natural language processing (NLP), but their effectiveness is largely dependent on pre-training resources. This is especially evident in low-resource languages, such as Sinhala …


title: Numerical Claim Detection in Finance: A New Financial Dataset, Weak-Supervision Model, and Market Analysis

authors: A Shah, A Hiray, P Shah, A Banerjee, A Singh… - arXiv preprint arXiv …, 2024

snippet: In this paper, we investigate the influence of claims in analyst reports and earnings calls on financial market returns, considering them as significant quarterly events for publicly traded companies. To facilitate a comprehensive analysis, we construct a …


title: NusaBERT: Teaching IndoBERT to be Multilingual and Multicultural

authors: W Wongso, DS Setiawan, S Limcorn, A Joyoadikusumo - arXiv preprint arXiv …, 2024

snippet: … We also note that NusaBERT was pre-trained on Wikipedia and common crawl corpora, which explains its effectiveness on and closeness to NusaX and NusaTranslation source domains, but not so for NusaParagraph. Thus, due to their high linguistic and …


title: Nyonic Technical Report

authors: J Tian, R Wang, C Li, Y Zhou, J Liu, J Wang - arXiv preprint arXiv:2404.15702, 2024

snippet: … Our curated dataset is specifically designed to meet these criteria, incorporating sources such as the Common Crawl, books, Wikipedia, code, and … Our data sources encompass Common Crawl, Wiki, and various coding repositories. The final …


title: Ocean Decade Vision 2030 White Papers–Challenge 8: Create a digital representation of the ocean.

authors: JB Calewaert, PC Sierra-Correa, O McMeel… - 2024

snippet: … The Common crawl and WebImageText are examples, with communities such as Hugging Face and KAGGLE creating new communities of practice16F … 17 https://commoncrawl.org, https://github.com/google-research-datasets/wit, https://huggingface.co, https://www.kaggle.com …


title: Offensive language detection in low resource languages: A use case of Persian language

authors: M Mozafari, K Mnassri, R Farahbakhsh, N Crespi - PLOS ONE, 2024

snippet: … XLM-RoBERTa [27]: is a transformer-based multilingual masked language model pre-trained on 100 languages, including Persian, using more than two terabytes of filtered CommonCrawl data. To fine-tune XLM-RoBERTa model on our target …


title: OLMo: Accelerating the Science of Language Models

authors: D Groeneveld, I Beltagy, P Walsh, A Bhagia, R Kinney… - arXiv preprint arXiv …, 2024

snippet: … well on evaluations predominated by Common Crawl, such as C4, though different ways of postprocessing Common Crawl are best fit by … The RedPajama evaluation shows a similar pattern, perhaps as only 2 of its 7 domains are from …


title: OLMoE: Open Mixture-of-Experts Language Models

authors: N Muennighoff, L Soldaini, D Groeneveld, K Lo… - arXiv preprint arXiv …, 2024

snippet: We introduce OLMoE, a fully open, state-of-the-art language model leveraging sparse Mixture-of-Experts (MoE). OLMoE-1B-7B has 7 billion (B) parameters but uses only 1B per input token. We pretrain it on 5 trillion tokens and further adapt it to …


title: Omniview-Tuning: Boosting Viewpoint Invariance of Vision-Language Pre-training Models

authors: S Ruan, Y Dong, H Liu, Y Huang, H Su, X Wei - arXiv preprint arXiv:2404.12139, 2024

snippet: Vision-Language Pre-training (VLP) models like CLIP have achieved remarkable success in computer vision and particularly demonstrated superior robustness to distribution shifts of 2D images. However, their robustness under 3D viewpoint …


title: On humans'(explicit) intuitions about the meaning of novel words

authors: D Gatti, F Rodio, L Rinaldi, M Marelli - Cognition, 2024

snippet: Pseudowords offer a unique opportunity to investigate how humans deal with new (verbal) information. Within this framework, previous studies have shown that, at the implicit level, humans exploit systematic associations in the form-meaning interface to …


title: On Initializing Transformers with Pre-trained Embeddings

authors: HY Kim, N Balasubramanian, B Kang - arXiv preprint arXiv:2407.12514, 2024

snippet: It has become common practice now to use random initialization schemes, rather than the pre-trained embeddings, when training transformer based models from scratch. Indeed, we find that pre-trained word embeddings from GloVe, and some …


title: On Labs and Fabs: Mapping How Alliances, Acquisitions, and Antitrust are Shaping the Frontier AI Industry

authors: T Aguirre - arXiv preprint arXiv:2406.01722, 2024

snippet: As frontier AI models progress, policy proposals for safe AI development are gaining increasing attention from researchers and policymakers. This paper evaluates the present landscape of integration within the AI supply chain, emphasizing vertical …


title: On Protecting the Data Privacy of Large Language Models (LLMs): A Survey

authors: B Yan, K Li, M Xu, Y Dong, Y Zhang, Z Ren, X Cheng - arXiv preprint arXiv …, 2024

snippet: … For example, GPT-3, developed by OpenAI, was pre-trained using CommonCrawl, constituting 45TB of compressed plaintext before filtering [37]. Regarding multimodal LLMs, CLIP’s training dataset encompasses 400 million pairs of images and text …


title: On the data scarcity problem of neural-based named entity recognition

snippet: … The initial release trained on Wikipedia includes word representations for nine languages, namely Arabic, Czech, German, English, Spanish, French, Italian, Romanian and Russian, and has quickly expanded to 157 languages by training on larger …


title: On the evaluation and application of neural language models for grammatical error detection

snippet: Neural language models (NLM) have become a core component in many downstream applications within the field of natural language processing, including the task of data-driven automatic grammatical error detection (GED). This thesis …


title: On the importance of Data Scale in Pretraining Arabic Language Models

authors: A Ghaddar, P Langlais, M Rezagholizadeh, B Chen - arXiv preprint arXiv:2401.07760, 2024

snippet: Pretraining monolingual language models have been proven to be vital for performance in Arabic Natural Language Processing (NLP) tasks. In this paper, we conduct a comprehensive study on the role of data in Arabic Pretrained Language …


title: On the Mathematical Relationship Between Contextual Probability and N400 Amplitude

authors: JA Michaelov, BK Bergen - Open Mind, 2024

snippet: Accounts of human language comprehension propose different mathematical relationships between the contextual probability of a word and how difficult it is to process, including linear, logarithmic, and super-logarithmic ones. However, the …


title: On the Question of Authorship in Large Language Models

authors: C Soos, L Haroutunian - KO KNOWLEDGE ORGANIZATION, 2024

snippet: … CommonCrawl exemplifies the problem of documentation debt. The dataset is an effort by The Common Crawl Foundation to “[democratize] access to web information by producing and maintaining an open repository of web crawl data” (Common …


title: On the Robustness of Neural Networks Quantization against Data Poisoning Attacks

authors: Y Lu, Y Wang, G Zhang, Y Yu - ICML 2024 Next Generation of AI Safety Workshop

snippet: … For example, practitioners may continuously collect user data online (eg, using Common Crawl1), which may be maliciously contaminated by an attacker. Such a threat is called data poisoning attacks (Biggio et al. 2012), … https://commoncrawl.org/ …


title: On the Tip of the Tongue: Analyzing Conceptual Representation in Large Language Models with Reverse-Dictionary Probe

authors: N Xu, Q Zhang, M Zhang, P Qian, X Huang - arXiv preprint arXiv:2402.14404, 2024

snippet: Probing and enhancing large language models' reasoning capacity remains a crucial open question. Here we re-purpose the reverse dictionary task as a case study to probe LLMs' capacity for conceptual inference. We use in-context learning …


title: One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support

authors: V Rasiah, R Stern, V Matoshi, M Stürmer, I Chalkidis…

snippet: Recent strides in Large Language Models (LLMs) have saturated many NLP benchmarks (even professional domain-specific ones), emphasizing the need for more challenging ones to properly assess LLM capabilities. Domain-specific and …


title: Online health search via multi-dimensional information quality assessment based on deep language models

authors: B Zhang, N Naderi, R Mishra, D Teodoro - 2024

snippet: Background Widespread misinformation in Web resources can lead to serious implications for individuals seeking health advice. Despite that, information retrieval models are often focused only on the query-document relevance dimension to rank …


title: Online Health Search Via Multidimensional Information Quality Assessment Based on Deep Language Models: Algorithm Development and Validation

authors: B Zhang, N Naderi, R Mishra, D Teodoro - JMIR AI, 2024

snippet: … In this study, we simulated online health information search scenarios with a topic set of 32 different health-related inquiries and a corpus containing 1 billion web documents from the April 2019 snapshot of Common Crawl. Using state-of-the-art …


title: OnlySportsLM: Optimizing Sports-Domain Language Models with SOTA Performance under Billion Parameters

authors: Z Chen, C Li, X Xie, P Dube - arXiv preprint arXiv:2409.00286, 2024

snippet: This paper explores the potential of a small, domain-specific language model trained exclusively on sports-related data. We investigate whether extensive training data with specially designed small model structures can overcome model size …


title: Ontology Completion with Natural Language Inference and Concept Embeddings: An Analysis

authors: N Li, T Bailleux, Z Bouraoui, S Schockaert - arXiv preprint arXiv:2403.17216, 2024

snippet: We consider the problem of finding plausible knowledge that is missing from a given ontology, as a generalisation of the well-studied taxonomy expansion task. One line of work treats this task as a Natural Language Inference (NLI) problem, thus relying …


title: Open Assistant Toolkit--version 2

authors: S Fischer, F Rossetto, C Gemmell, A Ramsay, I Mackie… - arXiv preprint arXiv …, 2024

snippet: … To source knowledge and tasks, we release a new offline pipeline to parse and augment task data from CommonCrawl. The offline pipeline supports using LLMs and additional open-source multimodal data sources to enhance tasks and make …


title: Open foundation models for Azerbaijani language

authors: J Isbarov, K Huseynova, E Mammadov, M Hajili - arXiv preprint arXiv:2407.02337, 2024

snippet: The emergence of multilingual large language models has enabled the development of language understanding and generation systems in Azerbaijani. However, most of the production-grade systems rely on cloud solutions, such as GPT-4 …


title: Open Science at the Generative AI Turn: An Exploratory Analysis of Challenges and Opportunities

snippet: Technology influences Open Science (OS) practices, because conducting science in transparent, accessible, and participatory ways requires tools/platforms for collaborative research and sharing results. Due to this direct relationship …


title: Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications

authors: Q Xie, D Li, M Xiao, Z Jiang, R Xiang, X Zhang, Z Chen… - arXiv preprint arXiv …, 2024

snippet: Large language models (LLMs) have advanced financial applications, yet they often lack sufficient financial knowledge and struggle with tasks involving multi-modal inputs like tables and time series data. To address these limitations, we introduce \textit{Open-FinLLMs} …


title: OpenAI ChatGPT and Biased Information in Higher Education

snippet: Motivated by the appearance of large language models and their sudden societal impacts—both beneficial and harmful, realized and potential—we evaluated several of them with respect to bias in its myriad forms. Bias in machine-learning models …


title: OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models

authors: F Xue, Z Zheng, Y Fu, J Ni, Z Zheng, W Zhou, Y You - arXiv preprint arXiv:2402.01739, 2024

snippet: To help the open-source community have a better understanding of Mixture-of-Experts (MoE) based large language models (LLMs), we train and release OpenMoE, a series of fully open-sourced and reproducible decoder-only MoE LLMs, ranging from …


title: OptiComm-GPT: a GPT-based versatile research assistant for optical fiber communication systems

authors: X Jiang, M Zhang, Y Song, Y Zhang, Y Wang, C Ju… - Optics Express, 2024

snippet: … Generally, LLMs are pre-trained on publicly available datasets such as BooksCorpus, Common Crawl, and Wikipedia, thereby ensuring comprehensive coverage of knowledge. Meanwhile, with the support of powerful analysis, reasoning …


title: Optimization and Evaluation in Machine Learning Challenges

snippet: To develop new machine learning methods, it is necessary to evaluate them reliably. This doctoral thesis discusses some aspects of preparing machine learning challenges and techniques for developing their solutions. The work consists of …


title: Optimization Hyper-parameter Laws for Large Language Models

authors: X Xie, K Ding, S Yan, KC Toh, T Wei - arXiv preprint arXiv:2409.04777, 2024

snippet: Large Language Models have driven significant AI advancements, yet their training is resource-intensive and highly sensitive to hyper-parameter selection. While scaling laws provide valuable guidance on model size and data requirements, they …


title: Optimization of news dissemination push mode by intelligent edge computing technology for deep learning

authors: JL DeGe, S Sang - Scientific Reports, 2024

snippet: … Realnew (https://paperswithcode.com/dataset/realnews) is a large corpus of news articles from CommonCrawl. The data is crawled from CommonCrawl, which is limited to 5000 news fields of Google News Index. The author uses the …


title: Optimization of Unsupervised Neural Machine Translation Based on Syntactic Knowledge Improvement

snippet: … Common Crawl is a multilingual aligned dataset based on web crawling. Para Crawl is a multilingual parallel corpus for large-scale web crawling. As shown in Table III, the detection accuracies of LSTM in Newstest2020, Newstest2021 …


title: Optimizing Data I/O for LLM Datasets on Remote Storage

snippet: Training large language models (LLMs) demands increasingly larger datasets for optimal performance [13]. In practice, these datasets may include hundreds of terabytes (TB) or even petabytes of data. For example, Common Crawl [3] needs …


title: Optimizing Knowledge Extraction in Large Language Models Using Dynamic Tokenization Dictionaries

snippet: Tokenization methods have long been a critical component in the performance of language models, yet traditional static approaches often fall short in capturing the dynamic nature of language. The novel concept of implementing a dynamic …


title: Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean

authors: CS Choi, Y Jeong, S Park, IH Won, HS Lim, SM Kim… - arXiv preprint arXiv …, 2024

snippet: … Llama2 is a multilingual language model trained using largescale publicly available data (including CommonCrawl, Github, Wikipedia, and ArXiv) for more than 28 languages. Thus, it possesses cross-language understanding capabilities …


title: Optimizing Large Language Models with Multi-Degree Low-Rank Approximations

snippet: … The use of datasets such as Common Crawl and Wikipedia facilitated a broad coverage of language variations, enhancing the generalizability of the results. Preprocessing steps were critical to ensure data quality and consistency, involving …


title: Optimizing Phishing Site Detection: Analyzing the Impact of Feature Quantity in Random Forest Model

authors: T Nabila, HH Nuha, SA Kristyan - 2024 International Conference on Data Science …, 2024

snippet: … The first dataset used is a dataset consisting of 5000 phishing sites and 5000 normal sites with 48 features using the Selenium Webdriver framework for extraction technique, This first dataset was collected and selected from Alexa and Common …


title: Optimizing Quality Estimation for Low-Resource Language Translations: Exploring the Role of Language Relatedness

authors: A Sindhujan, D Kanojia, C Orasan - Proceedings of the Conference New Trends in …, 2024

snippet: Evaluation of machine translation (MT) is vital to determine the effectiveness of MT systems. This paper investigates quality estimation (QE) for machine translation (MT) for low-resource Indic languages. We analyse the influence of language relatedness …


title: Optimizing social determinant of health concept extraction

snippet: Generative Artificial Intelligence has recently garnered enormous attention for a varied number of reasons, one of them being its capacity to automate complex tasks. As the potential for integrating Generative Artificial Intelligence across technology …


title: ORCID

snippet: How are Asian and Black men and women stereotyped? Research from the gendered race and stereotype content perspectives has produced mixed empirical findings. Using BERT models pre-trained on English language books, news articles …


title: Organic Data-Driven Approach for Turkish Grammatical Error Correction and LLMs

authors: A Ersoy, OT Yıldız - arXiv preprint arXiv:2405.15320, 2024

snippet: … 2020) which contains random paragraphs sampled from the CommonCrawl dataset3. Some studies put some effort into filling the gap and … 2020), a multilingual pre-trained encoder-decoder text-to-text transformer trained on a Common Crawl-based …


title: Outlier Detection in Serbian CommonCrawl Data

authors: V Kalušev, D Ćulibrk - 2024 23rd International Symposium INFOTEH …, 2024

snippet: The surge in large language models (LLMs) has greatly advanced natural language processing. However, their development is often hampered by the limited availability and quality of training datasets, particularly for underrepresented languages. Our …


title: Outsmarting Artificial Intelligence in the Classroom—Incorporating Large Language Model-Based Chatbots into Teaching

authors: J Wutzler - Issues in Accounting Education, 2024

snippet: Since the release of ChatGPT in November 2022, large language model-based chatbots have attracted much attention. Although businesses value their potential for efficiency gains, academics are concerned about their effects on learning and …


title: Overcoming the Labeled Training Data Bottleneck

authors: R Starzl - The Cyber Defense Review, 2024

snippet: This article explores the potential of large language models (LLMs) to transform dataset creation and analysis in cybersecurity. The proposed method leverages LLMs to overcome the labeled data bottleneck by generating high-quality, task-specific …


title: Overcoming the Labeled Training Data Bottleneck: A Route to Specialized AI


title: Overdetermination of businesswomen in the British press: a corpus-based approach in critical discourse analysis

authors: TNT Nguyen - ICTE Conference Proceedings, 2024

snippet: Within the strand of critical discourse analysis (CDA), research on media portrayal of individuals is prevalent, with the scope having been expanded beyond such subjects as refugees or asylum seekers, but also the breadth of research having …


title: Overview of Existing LLM Families

snippet: … data that are considered as redundant—namely the Common Crawl and Colossal Cleaned Common Crawl (C4). Researchers who created the … to Common Crawl when it came to training LLMs [30] and should be used instead of the whole …


title: Overview of the TREC 2023 NeuCLIR Track (Notebook Paper)

authors: D Lawrie, S MacAvaney, J Mayfield, P McNamee…

snippet: … For more information about extracting the text for the documents from CommonCrawl News and how the collection can be obtained, see … of the track including HC4 — a CLIR collection built over three years of Common Crawl data in …


title: Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization

authors: C Mavromatis, P Karypis, G Karypis - arXiv preprint arXiv:2404.11531, 2024

snippet: … 2019) is sa publicly available distribution of a Common Crawl snapshot. Following cBTM, we use 168B tokens of the no blocklist version (en.noblocklist) that is out of distribution to the OPT model’s pretraining corpus. S2ORC (Lo et al.…


title: PAL: Proxy-Guided Black-Box Attack on Large Language Models

authors: C Sitawarin, N Mu, D Wagner, A Araujo - arXiv preprint arXiv:2402.09674, 2024

snippet: Large Language Models (LLMs) have surged in popularity in recent months, but they have demonstrated concerning capabilities to generate harmful content when manipulated. While techniques like safety fine-tuning aim to minimize harmful use …


title: Palöri at CheckThat! 2024 Shared Task 6: GloTa-Combining GloVe Embeddings with RoBERTa for Adversarial Attack

snippet: This paper describes the submission of attack methods and results for shared task 6 at CheckThat! Lab at CLEF 2024. We present two novel attack methods to test the robustness of credibility assessment (CA) classifiers across five tasks: fact-checking …


title: Parameter-Efficient Tuning of Large Convolutional Models

authors: W Chen, Z Miao, Q Qiu - arXiv preprint arXiv:2403.00269, 2024

snippet: To address the high computational and parameter complexity associated with fine-tuning large pre-trained models, researchers have developed parameter-efficient methods, where only partial parameters are updated for downstream tasks. However, these …


title: Paraphrase Generation and Supervised Learning for Improved Automatic Short Answer Grading

authors: L Ouahrani, D Bennouar - International Journal of Artificial Intelligence in …, 2024

snippet: We consider the reference-based approach for Automatic Short Answer Grading (ASAG) that involves scoring a textual constructed student answer comparing to a teacher-provided reference answer. The reference answer does not cover the variety of student …


title: Parsability revisited and reassessed

authors: S Monakhov - Journal of Linguistics, 2023

snippet: This paper provides evidence that the inveterate way of assessing linguistic items’ degrees of analysability by calculating their derivation to base frequency ratios may obfuscate the difference between two meaning processing models: one based on …


title: Part-of-Speech Tagger for Bodo Language using Deep Learning approach

authors: D Pathak, S Narzary, S Nandi, B Som - arXiv preprint arXiv:2401.03175, 2024

snippet: Language Processing systems such as Part-of-speech tagging, Named entity recognition, Machine translation, Speech recognition, and Language modeling (LM) are well-studied in high-resource languages. Nevertheless, research on these …


title: Pathological Liars: Algorithmic Knowing in the Rhetorical Ecosystem of Wallstreetbets

authors: MH Yang, ZP Majdik - Rhetoric Society Quarterly, 2024

snippet: This essay demonstrates the value of using artificial intelligence (AI) technologies to address specific kinds of research questions in rhetoric. The essay builds on a study of a novel rhetorical object first observed by Yang on the Reddit subreddit r/wallstreetbets …


title: Pathological-Llama: an Explainable Medical Visual Question An

snippet: This thesis introduces Pathological-Llama, an explainable medical visual question answering system that integrates computer vision and natural language processing to accurately interpret medical images through a generative task approach. By …


title: Pedro L. Rodriguez Arthur Spirling Brandon M. Stewart Elisa M. Wirsching

snippet: … from Common Crawl and Wikipedia. A strength of the fastText model is that it uses subword … On inspection, we saw that Common Crawl includes many typos and rare terms (plus many … Beyond this potential for noise, Common Crawl is not …


title: Performance Analysis of Embedding Methods for Deep Learning-Based Turkish Sentiment Analysis Models

authors: A Ba Alawi, F Bozkurt - Arabian Journal for Science and Engineering, 2024

snippet: … Then, we chose the model which is specifically for Turkish that was trained using data from Common Crawl and Wikipedia. It employs position-weighted Continuous Bag of Words (CBOW) architecture. The model has a size of 300 and includes 5-character …


title: Performance Comparison of Large Language Models, GPT and Gemini on Turkish News Classification Task

snippet: … This was achieved by training a Transformer-based masked language model on 100 languages using more than two terabytes of filtered CommonCrawl data. The balance between high and low-resource languages and the impact of language …


title: Performance Evaluation of Word Embedding Algorithms

authors: S Kulshretha, L Lodha - Performance Evaluation, 2023

snippet: This study intends to explore the field of word embedding and thoroughly examine and contrast various word embedding algorithms. Words retain their semantic relationships and meaning when they are transformed into vectors using word …


title: Performance of ChatGPT on American Board of Surgery In-Training Examination Preparation Questions

authors: CG Tran, J Chang, SK Sherman, JP De Andrade - Journal of Surgical Research, 2024

snippet: … The training dataset of ChatGPT 3.5, where the artificial intelligence (AI) “knowledge” is derived, is comprised of Common Crawl (a monthly updated web archive of data), books, journals, and Wikipedia and totals over 500 billion encoded “tokens.” Many …


title: Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models

authors: Z Ankner, C Blakeney, K Sreenivasan, M Marion… - arXiv preprint arXiv …, 2024

snippet: … Secondly, while previous work only investigates pruning on datasets composed of just one domain (CommonCrawl1), we consider two … dataset, with 81.31% of its data being derived from the CommonCrawl. We find that successful pruning …


title: Personal Science Guide to AI

snippet: Imagine you have access to a zillion documents, preferably curated in some way reassures you about their quality and consistency. Wikipedia, for example, or maybe Reddit and other posts that have been sufficiently up-voted. Maybe you also have a …


title: Personality trait detection via transfer learning

authors: B Alshouha, J Serrano-Guerrero, F Chiclana… - Comput Mater Continua …, 2024

snippet: Personality recognition plays a pivotal role when developing user-centric solutions such as recommender systems or decision support systems across various domains, including education, e-commerce, or human resources. Traditional machine …


title: Perspectives of Global and Hong Kong's Media on China's Belt and Road Initiative

authors: LC Khoo, A Datta - arXiv preprint arXiv:2312.17013, 2023

snippet: This study delves into the media analysis of China's ambitious Belt and Road Initiative (BRI), which, in a polarized world, and furthermore, owing to the very polarizing nature of the initiative itself, has received both strong criticisms and …


title: PharmGPT: Domain-Specific Large Language Models for Bio-Pharmaceutical and Chemistry

authors: L Chen, W Wang, Z Bai, P Xu, Y Fang, J Fang, W Wu… - arXiv preprint arXiv …, 2024

snippet: … reinforcing inherent biases present in the source materials (eg, CommonCrawl dumps) and those introduced through the filtering processes. Such … New tools were developed for extracting text from HTML in Common Crawl WARC files …


title: PHISHING DETECTION SYSTEM THROUGH MACHINE LEARNING BASED ON URL

authors: PU Chandra, TV Teja, VN Reddy, S Agarwalla

snippet: … Legitimate site data is often gathered from Alexa's top-ranking website database or through common-crawl. Additionally, publicly available datasets like those from the UCI machine learning repository, containing over 11,000 records with 31 …


title: Phishing Detection Using Hybrid Algorithm Based on Clustering and Machine Learning

authors: L Al-Shalabi, Y Hasan Jazyah - International Journal of Computing and Digital …, 2024

snippet: Phishing is a prevalent and evolving cyber threat that continues to exploit human vulnerability to deceive individuals and organizations into revealing sensitive information. Phishing attacks encompass a range of tactics, from deceptive emails …


title: Phishing URL Detection Using Machine Learning and Deep Learning

authors: R Ferdaws, NE Majd - 2024 IEEE World AI IoT Congress (AIIoT), 2024

snippet: In the modern digital world, phishing attacks have become a persistent threat, jeopardizing the security and privacy of individuals and organizations, demanding prompt action to avoid excessive costs imposed to individuals and businesses. The …


title: Phishing Webpage Detection: Unveiling the Threat Landscape and Investigating Detection Techniques

authors: A Kulkarni, V Balachandran, T Das - IEEE Communications Surveys & Tutorials, 2024

snippet: … Common Crawl [53] is a nonprofit organization that maintains and provides access to a large-scale web crawling dataset. This dataset, known as the Common Crawl corpus, contains a comprehensive collection of web pages and their …


title: Physics-informed Machine Learning with Uncertainty Quantification

snippet: Physics Informed Machine Learning (PIML) has emerged as the forefront of research in scientific machine learning with the key motivation of systematically coupling machine learning (ML) methods with prior domain knowledge often available in the …


title: PIE: A Personalized Information Embedded model for text-based depression detection

authors: Y Wu, Z Liu, J Yuan, B Chen, H Cai, L Liu, Y Zhao… - Information Processing & …, 2024

snippet: … It undergoes pre-training on 2.5TB of filtered CommonCrawl data, encompassing 100 languages. By acquiring a comprehensive understanding of these 100 languages, the model can extract valuable features for downstream tasks. …


title: PIRB: A Comprehensive Benchmark of Polish Dense and Hybrid Text Retrieval Methods

authors: S Dadas, M Perełkiewicz, R Poświata - arXiv preprint arXiv:2402.13350, 2024

snippet: We present Polish Information Retrieval Benchmark (PIRB), a comprehensive evaluation framework encompassing 41 text information retrieval tasks for Polish. The benchmark incorporates existing datasets as well as 10 new, previously …


title: Pitfalls of Conversational LLMs on News Debiasing

authors: IB Schlicht, D Altiok, M Taouk, L Flek - arXiv preprint arXiv:2404.06488, 2024

snippet: This paper addresses debiasing in news editing and evaluates the effectiveness of conversational Large Language Models in this task. We designed an evaluation checklist tailored to news editors' perspectives, obtained generated texts from three …


title: PLAID SHIRTTT for Large-Scale Streaming Dense Retrieval

authors: D Lawrie, E Kayi, E Yang, J Mayfield, DW Oard - arXiv preprint arXiv:2405.00975, 2024

snippet: … With ten million documents from the news subset of CommonCrawl, NeuCLIR is presently the largest available test collection for MLIR; It is tiny though relative to ClueWeb, and thus has relatively small shards. The NeuCLIR MLIR task is to …


title: Please be polite to your peers: a multi-task model for assessing the tone and objectivity of critiques of peer review comments

authors: PK Bharti, M Agarwal, A Ekbal - Scientometrics, 2024

snippet: The peer-review process plays a pivotal role in maintaining the quality and credibility of scientific publications. However, in recent times, there has been an increase in unhelpful and overly critical reviews, which can be detrimental to the …


title: Politehnica University of Bucharest, Splaiul Independentei Nr. 313, Cector 6, Bucharest, Romania radu. stefan@ live. de, radu. stefan3@ student. upt. ro

authors: R Stefan, G Carutasu, M Mocan - The 17th International Conference Interdisciplinarity in …

snippet: This paper aims to investigate the ethical considerations surrounding the implementation and usage of large language models (LLMs). LLMs have shown tremendous advance in natural language processing and have the potential to …


title: Polysemy through the lens of psycholinguistic variables: a dataset and an evaluation of static and contextualized language models

authors: A Bruera, F Zamani, M Poesio - Proceedings of the 13th Joint Conference on Lexical …, 2024

snippet: … This version was trained on a combination of Common Crawl and Wikipedia and has 300dimensional vectors. We extract word vectors for all nouns and verbs and create a phrase-specific representation for each noun as described for countpmi. …


title: Position: Data Authenticity, Consent, & Provenance for AI are all broken: what will it take to fix them?

authors: S Longpre, R Mahari, N Obeng-Marnu, W Brannon…

snippet: … Common Crawl. For pretraining text data, models routinely rely on the Common Crawl.For instance, sixty percent of GPT-3 training data is … Common Crawl is accurate with wide coverage, but not detailed. Hugging Face can be inaccurate with …


title: Position: Mission Critical–Satellite Data is a Distinct Modality in Machine Learning

authors: E Rolf, K Klemmer, C Robinson, H Kerner - Forty-first International Conference on Machine …

snippet: Satellite data has the potential to inspire a seismic shift for machine learning---one in which we rethink existing practices designed for traditional data modalities. As machine learning for satellite data (SatML) gains traction for its real-world impact …


title: Position: Near to Mid-term Risks and Opportunities of Open-Source Generative AI

authors: F Eiras, A Petrov, B Vidgen, CS de Witt, F Pizzati… - Forty-first International Conference …

snippet: In the next few years, applications of Generative AI are expected to revolutionize a number of different areas, ranging from science & medicine to education. The potential for these seismic changes has triggered a lively debate about potential …


title: Position: Will we run out of data? Limits of LLM scaling based on human-generated data

authors: P Villalobos, A Ho, J Sevilla, T Besiroglu, L Heim… - Forty-first International Conference …

snippet: … Assuming that Common Crawl is a representative sample of the indexed web,11 we can use it to estimate the average amount of plain text bytes per web page. This number has increased over time, from around 6100 bytes in 2013 to about 8200 …


title: Post-processing Techniques for Word Embedding

snippet: Word embedding has been a significant breakthrough in natural language processing (NLP). Although word representation has improved remarkably and resulted in better performance in downstream NLP applications, interpretability of …


title: Posthumanist AI

snippet: … In their now canonical paper on “Stochastic Parrots”, Bender, Gebru, McMillan-Major, and Mitchell note that "it is easy to imagine that very large datasets, such as Common Crawl “…petabytes of data collected over 8 years of web crawling” a …


title: Practical Unlearning for Large Language Models

authors: C Gao, L Wang, C Weng, X Wang, Q Zhu - arXiv preprint arXiv:2407.10223, 2024

snippet: While LLMs have demonstrated impressive performance across various domains and tasks, their security issues have become increasingly severe. Machine unlearning (MU) has emerged as a promising solution to address these issues by …


title: Pre-Trained Acoustic-and-Textual Modeling for End-To-End Speech-To-Text Translation

authors: W Zhang, H Zhang, C Liu, Z Ye, X Zhou, C Lin, L Dai - ICASSP 2024-2024 IEEE …, 2024

snippet: … The monolingual data are mainly from News Commentary, News crawl and Common Crawl. We apply language identification [23] to retain sentences predicted as desired language, remove sentences longer than 250 tokens and with a source/target …


title: Pre-trained language models in medicine: A survey

authors: X Luo, Z Deng, B Yang, MY Luo - Artificial Intelligence in Medicine, 2024

snippet: With the rapid progress in Natural Language Processing (NLP), Pre-trained Language Models (PLM) such as BERT, BioBERT, and ChatGPT have shown great potential in various medical NLP tasks. This paper surveys the cutting-edge …


title: Pre-trained Large Language Models for Financial Sentiment Analysis

authors: W Luo, D Gong - arXiv preprint arXiv:2401.05215, 2024

snippet: Financial sentiment analysis refers to classifying financial text contents into sentiment categories (eg positive, negative, and neutral). In this paper, we focus on the classification of financial news title, which is a challenging task due to a lack of …


title: Pre-training Small Base LMs with Fewer Tokens

authors: S Sanyal, S Sanghavi, AG Dimakis - arXiv preprint arXiv:2404.08634, 2024

snippet: We study the effectiveness of a simple approach to develop a small base language model (LM) starting from an existing large base LM: first inherit a few transformer blocks from the larger LM, and then train this smaller model on a very small subset (0.1\%) …


title: Predicting fraud in MD&A sections using deep learning

authors: S Velloor Sivasubramanian, D Skillicorn - Journal of Business Analytics, 2024

snippet: Conventional data analytic techniques have been successfully applied to detecting fraud in the Management’s Discussion and Analysis sections of company filings mandated by the SEC. Here, we investigate whether fraud detection can be …


title: Predicting semantic category of answers for question answering systems using transformers: a transfer learning approach

authors: S CM, J Prakash, VS Alaparthi - Multimedia Tools and Applications, 2024

snippet: … The model uses a data set of web extracted text that has been created with the help of the Common Crawl and preprocessed to remove short, duplicate, and non-English content, resulting in a corpus called "Colossal Clean Crawled Corpus" (C4). An …


title: PREDICTING TARGETS WITH SEC 8-K FILINGS

snippet: There are large financial benefits in predicting acquisitions, but so far no adequate method has been found to that end. In this research textual data from 8-K Filings is used to improve the predictive performance of three classification models. 8-Ks are …


title: Prediction of Author's Profile basing on Fine-Tuning BERT model

authors: B Bsir, N Khoufi, M Zrigui - Informatica, 2024

snippet: … Indeed, XLMRoBERTa is the multilingual variant of RoBERTa trained with a multilingual MLM on one hundred languages, with more than two terabytes of filtered Common Crawl data.XLM-RoBERTa showed its superiority over BERT by its …


title: Preemptive Answer" Attacks" on Chain-of-Thought Reasoning

authors: R Xu, Z Qi, W Xu - arXiv preprint arXiv:2405.20902, 2024

snippet: Large language models (LLMs) showcase impressive reasoning capabilities when coupled with Chain-of-Thought (CoT) prompting. However, the robustness of this approach warrants further investigation. In this paper, we introduce a novel scenario …


title: Preprocessing Techniques for Clustering Arabic Text: Challenges and Future Directions.

authors: T Almutairi, S Saifuddin, R Alotaibi, S Sarhan, S Nassif - International Journal of …, 2024

snippet: Arabic is a complex language for text analysis because of its orthographic features, rich synonyms, and semantic style. Thus, Arabic text must be prepared more carefully in the preprocessing stage for the analyzer to improve the quality of the …


title: Preserving Privacy in Large Language Models: A Survey on Current Threats and Solutions

authors: M Miranda, ES Ruzzetti, A Santilli, FM Zanzotto… - arXiv preprint arXiv …, 2024

snippet: … In particular, in their experiments the language model generation is conditioned to some prompts that the attacker predefined: to maximize the probability of success of the attack, they choose prompts that are likely to be in the training data, like …


title: Pretraining and Updating Language-and Domain-specific Large Language Model: A Case Study in Japanese Business Domain

authors: K Takahashi, T Omi, K Arima, T Ishigaki - arXiv preprint arXiv:2404.08262, 2024

snippet: … , mC4, and Common Crawl. These datasets contribute to general knowledge, which is essential for diverse natural language processing tasks. … the proportion of clean domain-specific data, we aim to mitigate the impact of noisy data sources such …


title: Pretraining Data and Tokenizer for Indic LLM

authors: R Kumar, S Kakde, D Rajput, D Ibrahim, R Nahata… - arXiv preprint arXiv …, 2024

snippet: … Additionally, we perform deduplication on Common Crawl data to address the redundancy present in 70% of the crawled web pages. This study focuses on developing high-quality data, optimizing tokenization for our multilingual dataset for …


title: PRIVACY HARMS IN THE AI AGE

snippet: These problems run deeper than false accusations. Many AI models have trained on datasets that contain personal information, and every AI model is capable of regurgitating any data it has previously processed. In 2023, researchers …


title: Privacy in Federated Learning

snippet: The rise of Artificial Intelligence technology has raised concerns about the potential compromise of privacy due to the handling of personal data. Private AI prevents cybercrimes and falsehoods and protects human freedom and trust. While …


title: Probability-Aware Word-Confusion-Network-To-Text Alignment Approach for Intent Classification

authors: E Villatoro-Tello, S Madikeri, B Sharma, D Khalil… - ICASSP 2024-2024 IEEE …, 2024

snippet: … The Language Model (LM) was trained with 34M utterances from publicly available English datasets including People’s Speech, Fisher, Switchboard, AMI, Wikitext103, and subsets of Common Crawl and Reddit datasets. The model was …


title: Probing and enhancing the reliance of Transformer models on poetic information

snippet: Transformer models have achieved remarkable success in the widest variety of domains, spanning not just a multitude of tasks within natural language processing, but also those in computer vision, speech, and reinforcement learning. The key to …


title: Probing Critical Learning Dynamics of PLMs for Hate Speech Detection

authors: S Masud, MA Khan, V Goyal, MS Akhtar, T Chakraborty - arXiv preprint arXiv …, 2024

snippet: … Each variant is trained on a training corpus from Wikipedia, and Common-Crawl is curated and updated before the date associated with … that are pretrained on regular and latest Internet snapshots obtained via Common Crawl and Wikipedia …


title: Probing Large Language Models for Scalar Adjective Lexical Semantics and Scalar Diversity Pragmatics

authors: F Lin, D Altshuler, JB Pierrehumbert - arXiv preprint arXiv:2404.03301, 2024

snippet: … 2018) trained on 600B Common Crawl data as a baseline. For the baseline, we add the representations of strongest and weakest adjectives on s as ⃗s, then also rank cos(⃗a, ⃗s) for each a to calculate MRR. Here, we only report the results for the …


title: PRODIS-a speech database and a phoneme-based language model for the study of predictability effects in Polish

authors: Z Malisz, J Foremski, M Kul - arXiv preprint arXiv:2404.10112, 2024

snippet: We present a speech database and a phoneme-level language model of Polish. The database and model are designed for the analysis of prosodic and discourse factors and their impact on acoustic parameters in interaction with predictability effects. The …


title: Prometheus' Digital Fire: The Civic Responsibilities of Artificial Intelligence

snippet: [Prometheus], the noble son of Iapetus outwitted [Zeus] and stole the far-seen gleam of unwearying fire in a hollow fennel stalk. And Zeus who thunders on high was stung in spirit, and his dear heart was angered when he saw amongst men the far-seen …


title: Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars

authors: Z Wu, X Lin, Z Dai, W Hu, Y Shu, SK Ng, P Jaillet… - arXiv preprint arXiv …, 2024

snippet: … [2], it is highly likely that GPT-3 has been trained on common benchmark datasets potentially contained in Common Crawl due to data contamination. Hence, further providing high-quality in-context exemplars could hardly improve the performance of …


title: Prompting Change: ChatGPT's Impact on Digital Humanities Pedagogy–A Case Study in Art History

authors: Q Guo - International Journal of Humanities and Arts …, 2024

snippet: This article explores the transformative impact of ChatGPT, a Generative Pre-trained Transformers model, on humanities methodologies and its integration into a digital humanities (DH) course. The advent of ChatGPT enhances the capacity of people …


title: Prompting Encoder Models for Zero-Shot Classification: A Cross-Domain Study in Italian

authors: S Auriemma, M Miliani, M Madeddu, A Bondielli… - arXiv preprint arXiv …, 2024

snippet: … We may suppose that this is because UmBERTo was trained on Common-Crawl, which also contains legal and administrative texts in its Italian section. Very high results are obtained by UmBERTo for PER entities, reaching 0.83 in our zero-shot …


title: Prospect of large language models and natural language processing for lung cancer diagnosis: A systematic review

authors: A Garg, S Gupta, S Vats, P Handa, N Goel - Expert Systems

snippet: Lung cancer, a leading cause of global mortality, demands a combat for its effective prevention, early diagnosis, and advanced treatment methods. Traditional diagnostic methods face limitations in accuracy and efficiency, necessitating …


title: Protein-Protein Interaction Prediction

snippet: Proteins are essential components of cellular processes, playing pivotal roles in various biological functions. Their interactions with other proteins are fundamental for maintaining cellular functionality, a phenomenon referred to as protein-protein …


title: PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models

authors: H Tan, Z Guo, Z Shi, L Xu, Z Liu, X Li, Y Wang, L Shang… - arXiv preprint arXiv …, 2024

snippet: … Nevertheless, scaling the context window during the pretraining phase remains a daunting task as the computational demands surge quadratically with the length of the attention span, and a majority of texts within standard corpora, like Common …


title: Pruning Large Language Models with Semi-Structural Adaptive Sparse Training

authors: W Huang, G Jian, Y Hu, J Zhu, J Chen - arXiv preprint arXiv:2407.20584, 2024

snippet: Transformer-based Large Language Models (LLMs) have demonstrated remarkable success across various challenging tasks. However, the deployment of LLMs is hindered by their substantial parameter count and memory consumption. Recently …


title: PSILENCE: A Pseudonymization Tool for International Law

authors: LA Cabrera-Diego, A Gheewala - Proceedings of the Workshop on Computational …, 2024

snippet: Since the announcement of the GDPR, the pseudonymization of legal documents has become a high-priority task in many legal organizations. This means that for making public a document, it is necessary to redact the identity of certain entities …


title: PSQE: Personalized Semantic Query Expansion for user-centric query disambiguation

snippet: … Furthermore, they enable explainable intelligent systems, in that the results of querying the embedding for similar terms relate directly to the training set, rather than webscale collections such as Common Crawl. To our knowledge, our work is …


title: PSQS: Parallel Semantic Querying Service for Self-describing File Formats

authors: C Niu, W Zhang, S Byna, Y Chen - 2023 IEEE International Conference on Big Data …, 2023

snippet: … Typically, theseword representations are learned from a training dataset suchas the Common Crawl web corpus (840 billion tokens, 2.2million words) or Wikipedia 2014 (6 billion tokens, 400,000words). However, this assumption does not always …


title: Public perception of the Chinese president's visit to Saudi Arabia and the China–Arab Summit: sentiment analysis of Arabic tweets

authors: AAM Hassan - Social Network Analysis and Mining, 2024

snippet: In recent years, China and Saudi Arabia have had frequent exchanges in the political and economic fields, and public opinion evaluation is an essential aspect of evaluating the two countries’ current relations. This paper analyzes tweets in Arabic …


title: PUBLIC-PRIVATE PARTNERSHIPS IN HEALTH SECTOR INNOVATION: LESSONS FROM AROUND THE WORLD

authors: CC Ebulue, OR Ebulue, CS Ekesiobi - International Medical Science Research …, 2024

snippet: Public-Private Partnerships (PPPs) have emerged as a crucial mechanism for fostering innovation in the health sector globally. This review encapsulates the lessons learned from diverse PPP models worldwide, highlighting their significance …


title: PyLoomer-The Automated Weaving of Python Text

authors: A Sahani, RB Trivedi, T Singh, SR Goyal - 2024 2nd International Conference on …, 2024

snippet: The introduction of attention mechanisms has evolved NLP methodologies, giving rise to Large Language Models (LLMs) that excel in various sequence-to-sequence tasks like translation, question-answering, summarization, intent, and entity …


title: Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients

authors: Z Zhang, A Jaiswal, L Yin, S Liu, J Zhao, Y Tian… - arXiv preprint arXiv …, 2024

snippet: … The C4 dataset is a massive collection of Common Crawl’s web crawl corpus, meticulously filtered and cleaned to ensure highquality language modeling and training. It is widely used for pre-training large language models due to its diverse …


title: Quality Does Matter: A Detailed Look at the Quality and Utility of Web-Mined Parallel Corpora

authors: S Ranathunga, N de Silva, M Velayuthan, A Fernando… - arXiv preprint arXiv …, 2024

snippet: … 2020) is a dataset created using 68 snapshots of CommonCrawl10. Document … of CommonCrawl. It contains around 4.5 billion parallel sentences across 576 language pairs. In building CCMatrix, it was assumed the aligned sentence could …


title: Quantifying Gender Bias in Arabic Pre-trained Language Models

authors: W Alrajhi, H Al-Khalifa, AM Al-Salman - IEEE Access, 2024

snippet: The current renaissance in the development of Arabic Pre-trained Language models (APLMs) has yielded significant advancement across many fields. Nevertheless, no study has explored the dimensions of gender bias in these models. It is argued that …


title: Quantifying Geospatial in the Common Crawl Corpus

authors: I Ilyankou, M Wang, J Haworth, S Cavazzi - arXiv e-prints, 2024

snippet: … Common Crawl corpus. However, the geospatial content within CC remains largely unexplored, impacting our understanding of LLMs' spatial reasoning. This paper investigates the prevalence of geospatial data in recent Common Crawl …


title: Quantifying semantic relatedness across base verbs and derivatives: English

snippet: Most authors on derivational semantics agree that the meaning of complex words builds on components contributed by the base, components contributed by the word formation process, as well as contextual knowledge components. There is far less …


title: Quantifying task-related gaze

authors: K Walter, M Freeman, P Bex - Attention, Perception, & Psychophysics, 2024

snippet: … For this study, we used the pretrained word vector Common Crawl (840B tokens, 2.2M vocab, cased, 300d vectors), which can be obtained at https://nlp.stanford.edu/projects/glove/. To utilize GloVe with the LabelMe images, we used it in conjunction with LASS …


title: Quantifying the Importance of Data Alignment in Downstream Model Performance

authors: K Chawla, A Sahai, M DePavia, S Sundar, B Miranda - 2024

snippet: Contrary to the conventional emphasis on dataset size, we explore the role of data alignment---an often overlooked aspect of data quality---in training capable Large Language Models (LLMs). To do so, we use the Task2Vec-based alignment …


title: Quantitative Analysis of the Relationship Between Optimal Learning Rate and Batch Size Scaling in Large Language Models

authors: R Schneider, H Baumgartner, D Wohlgemuth

snippet: … Publicly available large-scale text datasets were utilized, including Common Crawl, Wikipedia, and BooksCorpus, which provided diverse and extensive language data. These datasets were chosen for their rich linguistic content and their …


title: Query Obfuscation for Information Retrieval through Differential Privacy

snippet: Protecting the privacy of a user querying an Information Retrieval (IR) system is of utmost importance. The problem is exacerbated when the IR system is not cooperative in satisfying the user’s privacy requirements. To address this …


title: Query of CC: Unearthing Large Scale Domain-Specific Knowledge from Public Corpora

authors: Z Fei, Y Shao, L Li, Z Zeng, H Yan, X Qiu, D Lin - arXiv preprint arXiv:2401.14624, 2024

snippet: … become increasingly rich, including the Pile, RedPajama, and Common Crawl. To ensure the diversity of data sources for retrieval, we constructed our retrieval corpus based on Common CrawlCommon Crawl is an open-source web crawler project …


title: Query Performance Prediction using Relevance Judgments Generated by Large Language Models

authors: C Meng, N Arabzadeh, A Askari, M Aliannejadi… - arXiv preprint arXiv …, 2024

snippet: Query performance prediction (QPP) aims to estimate the retrieval quality of a search system for a query without human relevance judgments. Previous QPP methods typically return a single scalar value and do not require the predicted …


title: Query Spelling Correction using Pre-trained Word Embeddings

snippet: … The GloVe word embeddings were trained on multiple dumps of Wikipedia, as well as on a Common Crawl corpus from 2014. Vector dimensions larger than 300 were empirically found to provide diminishing returns on semantic and syntactic …


title: QuRating: Selecting High-Quality Data for Training Language Models

authors: A Wettig, A Gupta, S Malik, D Chen - arXiv preprint arXiv:2402.09739, 2024

snippet: … We visualize the distributions for the cluster in CommonCrawl and C4 in Figure 7 in the appendix. They are similarly encouraging, eg, the clusters associated with cells, protein, gene and energy, climate, species are rated highly on required …


title: R2GQA: Retriever-Reader-Generator Question Answering System to Support Students Understanding Legal Regulations in Higher Education

authors: PTP Do, DND Cao, KQ Tran, K Van Nguyen - arXiv preprint arXiv:2409.02840, 2024

snippet: In this article, we propose the R2GQA system, a Retriever-Reader-Generator Question Answering system, consisting of three main components: Document Retriever, Machine Reader, and Answer Generator. The Retriever module employs …


title: RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors

authors: L Dugan, A Hwang, F Trhlik, JM Ludan, A Zhu, H Xu… - arXiv preprint arXiv …, 2024

snippet: … The model was allegedly trained with a variety of data including the Common Crawl (fil- … Released on July 18th 2023, LLaMA 2 is the successor to the original LLaMA model which was trained on webpages from CommonCrawl, multilingual …


title: Ranking Generated Answers: On the Agreement of Retrieval Models with Humans on Consumer Health Questions

authors: S Heineking, J Probst, D Steinbach, M Potthast… - arXiv preprint arXiv …, 2024

snippet: … We therefore obtained only the original Web documents from CommonCrawl, discarded those containing fewer than 50 characters in the HTML body, and extracted plain text using the Resiliparse library.We were able to restore 6,692 Web …


title: Rapidly Developing High-quality Instruction Data and Evaluation Benchmark for Large Language Models with Minimal Human Effort: A Case Study on Japanese

authors: Y Sun, Z Wan, N Ueda, S Yahata, F Cheng, C Chu… - arXiv preprint arXiv …, 2024

snippet: The creation of instruction data and evaluation benchmarks for serving Large language models often involves enormous human annotation. This issue becomes particularly pronounced when rapidly developing such resources for a non-English …


title: Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learning

authors: J Hejna, C Bhateja, Y Jian, K Pertsch, D Sadigh - arXiv preprint arXiv:2408.14037, 2024

snippet: … Many breakthroughs in machine learning can be attributed to “Internet-scale” datasets, from the development of vision models like CLIP [1] to recent advancements in transformer-based language modeling powered by the Common …


title: Real-Time Intelligent Detection System Designs for Web Phishing Attacks

snippet: … First, they collected URLs using Common Crawl and extracted their domain names to build their vocabulary. Then, they combine this list of vocabulary with common English names. Next, they use the regular expression pattern [a-zA-Z]+|[0-9]+ …


title: Real-time phishing detection using deep learning methods by extensions

authors: DM Linh, HD Hung, HM Chau, QS Vu, TN Tran - International Journal of Electrical …, 2024

snippet: … [2] used two algorithms, random forest (RF) and long short-term memory (LSTM), to evaluate the PhishTank and Common Crawl datasets, achieving accuracies of 93.5% and 98.7%, respectively. Similarly, Kumar et al. [3] evaluated using the …


title: Real-World Robot Applications of Foundation Models: A Review

authors: K Kawaharazuka, T Matsushima, A Gambardella… - arXiv preprint arXiv …, 2024

snippet: … In terms of the data, the datasets for training LLMs are Internet-scale; for example, the pre-training of GPT-3 [47] uses CommonCrawl covering Internet data from years 2016 to 2019, which amounts to 45TB of compressed plain text and 410 billion …


title: Reasoning for fact verification using language models

snippet: In response to the proliferation of misinformation on social media platforms, this thesis introduces the Triple-R framework (Retriever, Ranker, Reasoner) to enhance fact-checking by leveraging the Web for evidence retrieval and generating …


title: Recent Advancements and Challenges of Turkic Central Asian Language Processing

authors: Y Veitsman - arXiv preprint arXiv:2407.05006, 2024

snippet: Research in the NLP sphere of the Turkic counterparts of Central Asian languages, namely Kazakh, Uzbek, Kyrgyz, and Turkmen, comes with the typical challenges of low-resource languages, like data scarcity and a general lack of linguistic resources …


title: Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectives

authors: DH Hagos, R Battle, DB Rawat - arXiv preprint arXiv:2407.14962, 2024

snippet: The emergence of Generative Artificial Intelligence (AI) and Large Language Models (LLMs) has marked a new era of Natural Language Processing (NLP), introducing unprecedented capabilities that are revolutionizing various domains. This paper …


title: Recent Advances in Large Language Models for Healthcare

authors: K Nassiri, MA Akhloufi - BioMedInformatics, 2024

snippet: Recent advances in the field of large language models (LLMs) underline their high potential for applications in a variety of sectors. Their use in healthcare, in particular, holds out promising prospects for improving medical practices. As we highlight in …


title: Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark

authors: H Cao - arXiv preprint arXiv:2406.01607, 2024

snippet: … data: (post, comment) pairs from Reddit3, (question, upvoted answer) pairs from Stackexchange4, (entity name + section title, passage) pairs from English Wikipedia, (title, abstract) and citation pairs from Scientific papers [53], and (title, passage) pairs …


title: Recognizing Textual Inference in Mongolian Bar Exam Questions

authors: G Khaltarkhuu, B Batjargal, A Maeda - Applied Sciences, 2024

snippet: This paper examines how to apply deep learning techniques to Mongolian bar exam questions. Several approaches that utilize eight different fine-tuned transformer models were demonstrated for recognizing textual inference in Mongolian bar exam …


title: Recording First-person Experiences to Build a New Type of Foundation Model

authors: D Barcari, D Gamez, A Grig - arXiv preprint arXiv:2408.02680, 2024

snippet: … These models are trained on large amounts of text and image data scraped from the Internet - often accessed through the Common Crawl repository. Researchers and technology companies are starting to realize that this data source could become …


title: RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent

authors: H Xu, W Zhang, Z Wang, F Xiao, R Zheng, Y Feng, Z Ba… - arXiv preprint arXiv …, 2024

snippet: Recently, advanced Large Language Models (LLMs) such as GPT-4 have been integrated into many real-world applications like Code Copilot. These applications have significantly expanded the attack surface of LLMs, exposing them to a variety of …


title: Referring expression segmentation: from conventional to generalized

snippet: In recent years, many remarkable achievements have been made in the field of deep machine learning in various data modalities, such as image processing and natural language comprehension. Based on the good performance of deep neural …


title: RegMix: Data Mixture as Regression for Language Model Pre-training

authors: Q Liu, X Zheng, N Muennighoff, G Zeng, L Dou, T Pang… - arXiv preprint arXiv …, 2024

snippet: … is taken from the C4 dataset [47] and supposed to share a similar distribution as Pile-CC since they are all derived from the CommonCrawl corpus. Since CommonCrawl is a collection of diverse domains, we would expect the correlation …


title: Reliability Estimation of News Media Sources: Birds of a Feather Flock Together

authors: S Burdisso, D Sánchez-Cortés, E Villatoro-Tello…

snippet: Evaluating the reliability of news sources is a routine task for journalists and organizations committed to acquiring and disseminating accurate information. Recent research has shown that predicting sources’ reliability represents an …


title: Reliability of large language models in managing odontogenic sinusitis clinical scenarios: a preliminary multidisciplinary evaluation

authors: AM Saibene, F Allevi, C Calvo-Henriquez, A Maniaci… - European Archives of Oto …, 2024

snippet: Purpose This study aimed to evaluate the utility of large language model (LLM) artificial intelligence tools, Chat Generative Pre-Trained Transformer (ChatGPT) versions 3.5 and 4, in managing complex otolaryngological clinical scenarios …


title: ReLU $^ 2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs

authors: Z Zhang, Y Song, G Yu, X Han, Y Lin, C Xiao, C Song… - arXiv preprint arXiv …, 2024

snippet: … For the corpus used for calculating the sparsity statistics, we a mixture of texts from Wikipedia, Pile, and Common Crawl. The total number of tokens is about 217. Since we need to store the activation values of all neurons, we only use a part of the …


title: Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling

authors: P Maini, S Seto, H Bai, D Grangier, Y Zhang, N Jaitly - arXiv preprint arXiv …, 2024

snippet: … There are different versions of CommonCrawl data and our selection of C4 for pretraining is driven by driven by its size and quality. … The dataset is also derived from the CommonCrawl, however has a more stringent filtering process. Our …


title: Representation Bias of Adolescents in AI: A Bilingual, Bicultural Study

authors: R Wolfe, A Dangol, B Howe, A Hiniker - arXiv preprint arXiv:2408.01961, 2024

snippet: Popular and news media often portray teenagers with sensationalism, as both a risk to society and at risk from society. As AI begins to absorb some of the epistemic functions of traditional media, we study how teenagers in two countries speaking two …


title: Representation learning for computational sociopragmatics

snippet: Natural Language Processing (NLP) emerges as a critical solution for analyzing, manipulating, and understanding human language automatically and computationally, enabling the processing of vast amounts of language data swiftly …


title: Representation Learning for Generative Models with Applications to Healthcare, Astronautics, and Aviation

snippet: … Large Language Models (LLMs) like BERT are typically pre-trained on extensive open web datasets, such as the English Wikipedia [50], BookCorpus [180], and Common Crawl [126]. These datasets enable LLMs to develop a robust …


title: Representation Learning for Hierarchical Classification of Entity Titles

authors: E Chistova - Proceedings of the 6th International Conference on …, 2023

snippet: We present a method for effective title encoding for hierarchical classification in a large taxonomy. The method enables taxonomy-aware encoding in pre-trained text encoders, such as fastText and BERT, which are additionally fine-tuned for the …


title: Representing Online Handwriting for Recognition in Large Vision-Language Models

authors: A Fadeeva, P Schlattner, A Maksai, M Collier… - arXiv preprint arXiv …, 2024

snippet: The adoption of tablets with touchscreens and styluses is increasing, and a key feature is converting handwriting to text, enabling search, indexing, and AI assistance. Meanwhile, vision-language models (VLMs) are now the go-to solution …


title: Representing relational knowledge with language models

snippet: Relational knowledge is the ability to recognize the relationship between instances, and it has an important role in human understanding a concept or commonsense reasoning. We, humans, structure our knowledge by understanding individual …


title: Research and Application of Large Model-Based Intelligent Customer Service System

authors: Y Xi - International Journal of Emerging Technologies and …, 2024

snippet: With the rapid development of artificial intelligence technology, intelligent customer service systems have been widely used. This paper addresses the limitations of traditional intelligent customer service systems, such as limited language …


title: Research on cross-lingual multi-label patent classification based on pre-trained model

authors: Y Lu, L Chen, X Tong, Y Peng, H Zhu - Scientometrics, 2024

snippet: Patent classification is an important part of the patent examination and management process. Using efficient and accurate automatic patent classification can significantly improve patent retrieval performance. Current monolingual patent classification …


title: Research Scholar (JRF), Department of Philosophy, The University of Burdwan, Golapbug

snippet: In these days, technology advances very rapidly. Today, we are at the doorstep of constructing a super-intelligent machine capable of mimicking almost all human activities. Initially, we took recourse to science and technology to transcend our …


title: ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models

authors: J Baek, SK Jauhar, S Cucerzan, SJ Hwang - arXiv preprint arXiv:2404.07738, 2024

snippet: Scientific Research, vital for improving human life, is hindered by its inherent complexity, slow pace, and the need for specialized experts. To enhance its productivity, we propose a ResearchAgent, a large language model-powered …


title: Rethinking Higher Education Teaching and Assessment In-Line with AI Innovations: A Systematic Review and Meta-Analysis

authors: JN Lyanda, SO Owidi, AM Simiyu - African Journal of Empirical Research, 2024

snippet: With the rapid advancement of artificial intelligence (AI) technologies, higher education institutions are increasingly exploring innovative ways to rethink teaching and assessment practices. This research paper examines the implications of AI on …


title: Retrieval Augmented Generation via Context Compression Techniques for Large Language Models

snippet: Natural language processing has seen lots of improvements, yet optimizing large-scale models to efficiently handle vast amounts of contextual data remains a critical challenge. The novel approach presented integrates advanced context compression …


title: Retrieval-Based Grammatical Error Correction

snippet: … a cleaned version of the common crawl to produce nearly 200 million ungrammatical sentences with a realistic distribution of errors. … CWEB consists of randomly selected texts from the first 18 dumps of the Common Crawl, which include …


title: Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models

authors: J Parmar, S Satheesh, M Patwary, M Shoeybi… - arXiv preprint arXiv …, 2024

snippet: … In our English corpus, the Web Crawl data is sourced from Common Crawl (CC) snapshots while the remaining categories are comprised of high-quality sets. For instance, the miscellaneous category consists of BigScience ROOTS (Lachaux et al.…


title: Revisión de modelos de lenguaje y aplicación a la automatización de tareas de oficina en empresas industriales

snippet: This project focuses on the evaluation and application of Large Language Models (LLMs) in natural language processing (NLP). The technical properties, advantages, and disadvantages of LLMs are analysed, highlighting their wide use in machine …