title: How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites

authors: Z Chen, W Wang, H Tian, S Ye, Z Gao, E Cui, W Tong… - arXiv preprint arXiv …, 2024

snippet: In this report, we introduce InternVL 1.5, an open-source multimodal large language model (MLLM) to bridge the capability gap between open-source and proprietary commercial models in multimodal understanding. We introduce three simple …


title: How Gender Interacts with Political Values: A Case Study on Czech BERT Models

authors: AA Ali, J Libovický - arXiv preprint arXiv:2403.13514, 2024

snippet: Neural language models, which reach state-of-the-art results on most natural language processing tasks, are trained on large text corpora that inevitably contain value-burdened content and often capture undesirable biases, which the models …


title: How Large Corpora Sizes Influence the Distribution of Low Frequency Text n-grams

authors: JF Silva, JC Cunha - Pacific-Asia Conference on Knowledge Discovery and …, 2024

snippet: The prediction of the numbers of distinct word n-grams and their frequency distributions in text corpora is important in domains like information processing and language modelling. With big data corpora, there is an increased application …


title: How Lexical is Bilingual Lexicon Induction?

authors: H Kohli, H Feng, N Dronen, C McCarter, S Moeini… - arXiv preprint arXiv …, 2024

snippet: … derived from Common Crawl and Wikipedia. The plot visualizes the Spearman’s Rank correlation of term frequencies between each of the source (by row) and target (by column) language pairs in the 5k vocabularies in the XLING corpus derived from …


title: How many news websites block AI crawlers?

snippet: In this factsheet we describe the proportion of news websites in ten countries that block AI (artificial intelligence) crawlers. We find that, (i) by the end 2023, 48% of the most widely used news websites across ten countries were blocking OpenAI’s (ChatGPT) …


title: How Much are LLMs Contaminated? A Comprehensive Survey and the LLMSanitize Library

authors: M Ravaut, B Ding, F Jiao, H Chen, X Li, R Zhao, C Qin… - arXiv preprint arXiv …, 2024

snippet: … This includes policies and protocols for data privacy, consent, and use that help prevent the incorporation of contaminated data from unethical sources and the contamination of widely used pre-training data sources (eg, CommonCrawl) …


title: How much semantic information is available in large language model tokens?

snippet: … We first regressed MMLU score on the percentage of the Common Crawl that a language makes up and whether it uses the Latin alphabet… is a significant predictor when added to the model that includes Common Crawl percentage and …


title: How to Train Data-Efficient LLMs

authors: N Sachdeva, B Coleman, WC Kang, J Ni, L Hong… - arXiv preprint arXiv …, 2024

snippet: The training of large language models (LLMs) is expensive. In this paper, we study data-efficient approaches for pre-training LLMs, ie, techniques that aim to optimize the Pareto frontier of model quality and training resource/data consumption. We …


title: HPLT's First Release of Data and Models

authors: N Arefyev, M Aulamo, P Chen, O de Gibert, B Haddow… - 2024

snippet: … Datasets For the first release, we processed 1.85 petabytes of the Internet Archive and CommonCrawl to create monolingual and parallel corpora. We release them under the permissive CC0 licence1 through our project website2, OPUS3, and Hugging …


title: Human-Centered Interaction in Virtual Worlds: A New Era of Generative Artificial Intelligence and Metaverse

authors: Y Wang, L Wang, KL Siau - International Journal of Human–Computer Interaction, 2024

snippet: The metaverse has emerged as an exciting new paradigm for human-computer interaction (HCI) and virtual collaboration. This paper presents a comprehensive review of the metaverse to address the gap in the existing literature where there is a …


title: Human-Centered Natural Language Processing for Countering Misinformation

snippet: As curbing the spread of online misinformation has proven to be challenging, we look to artificial intelligence (AI) and natural language technology for helping individuals and society counter and limit it. Despite current advances, state-of-the-art …


title: Humkinar: Construction of a Large Scale Web Repository and Information System for Low Resource Urdu Language

snippet: … first Common Crawl Corpus (CCC) filter leverages the open-source Common Crawl repository to filter Urdu language webpages. Common Crawl… From August 2018 onward releases, Common Crawl also provides the meta information …


title: Hybrid Bilingual Question-Answering System with Human-Level Naturalness Text-to-Speech Extension

snippet: This project introduces a robust bilingual question-answering and natural Text-to-Speech (TTS) system, primarily designed to support both Vietnamese and English languages. Leveraging pre-trained Large Language Models (LLMs), the system enables …


title: Hybrid semantic models for building smart and robust home robots

snippet: The creation of home robots that can aid human beings in daily mundane chores has been a long-standing goal of robotics research. Some common indoor tasks that service robots can help with include retrieving objects from different locations …


title: Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers

authors: S Hwang, A Lahoti, T Dao, A Gu - arXiv preprint arXiv:2407.09941, 2024

snippet: A wide array of sequence models are built on a framework modeled after Transformers, comprising alternating sequence mixer and channel mixer layers. This paper studies a unifying matrix mixer view of sequence mixers that can be …


title: HYPE: Hyperbolic Entailment Filtering for Underspecified Images and Texts

authors: W Kim, S Chun, T Kim, D Han, S Yun - arXiv preprint arXiv:2404.17507, 2024

snippet: In an era where the volume of data drives the effectiveness of self-supervised learning, the specificity and clarity of data semantics play a crucial role in model training. Addressing this, we introduce HYPerbolic Entailment filtering (HYPE), a …


title: Hypertext Entity Extraction in Webpage

authors: Y Yang, T Liu, B Shao, H Zhao, L Shou, M Gong… - arXiv preprint arXiv …, 2024

snippet: … Common Crawl5 is a widely used large-scale dataset for IE. However, they all overlook rich hypertext features and contain too much noise. Moreover, their annotations are often determined by rules or automated tools, lacking accurate manual annotations. …


title: I'm In! A Comparative Analysis of Business Pitch Competition

authors: E Barron, I Bernatovic, LE Ruiz - DIANA INTERNATIONAL RESEARCH …, 2023

snippet: Methods Based on a perceived theoretical gap in research, entrepreneurship scholars have called for increased research to understand how institutions (including governments and academia), influence the construction of intersectional …


title: Identification of patients' smoking status using an explainable AI approach: a Danish electronic health records case study

authors: A Ebrahimi, MBH Henriksen, CL Brasen, O Hilberg… - BMC Medical Research …, 2024

snippet: Smoking is a critical risk factor responsible for over eight million annual deaths worldwide. It is essential to obtain information on smoking habits to advance research and implement preventive measures such as screening of high-risk …


title: Identification of Trolling Memes in Kannada and Tulu-Under-resourced Dravidian Languages

snippet: … Trained on a diverse dataset comprising Common Crawl and Wikipedia content across 157 languages, fastText employs Continuous Bag-of-Words (CBOW) with position weights, generating embeddings in a low-dimensional space of size 300 [44] …


title: Identifying the sources of ideological bias in GPT models through linguistic variation in output

authors: C Walker, JC Timoneda - arXiv preprint arXiv:2409.06043, 2024

snippet: Extant work shows that generative AI models such as GPT-3.5 and 4 perpetuate social stereotypes and biases. One concerning but less explored source of bias is ideology. Do GPT models take ideological stances on politically sensitive topics? In …


title: iDRAMA-rumble-2024: A Dataset of Podcasts from Rumble Spanning 2020 to 2022

authors: U Balci, J Patel, B Balci, J Blackburn - 2024

snippet: Rumble has emerged as a prominent platform hosting controversial figures facing restrictions on YouTube. Despite this, the academic community’s engagement with Rumble has been minimal. To help researchers address this gap, we introduce a …


title: If You Don't Understand It, Don't Use It: Eliminating Trojans with Filters Between Layers

authors: A Hernandez - arXiv preprint arXiv:2407.06411, 2024

snippet: … In this model an LLM is pre-trained on a large web corpus such as Common Crawl, the model server tunes the model and inserts it into a (business) mission-critical system, and the model user uses this system. This system can be a chat-bot, code-completion …


title: IGOT: Information Gain Optimized Tokenizer on Domain Adaptive Pretraining

authors: D Feng, Y Zhang, Z Xu - arXiv preprint arXiv:2405.09857, 2024

snippet: Pretrained Large Language Models (LLM) such as ChatGPT, Claude, etc. have demonstrated strong capabilities in various fields of natural language generation. However, there are still many problems when using LLM in specialized domain-specific …


title: Impact of COVID-19 Pandemic on Social Determinants of Health Issues of Marginalized Black and Asian Communities: A Social Media Analysis Empowered by …

authors: C Whitfield, Y Liu, M Anwar - Journal of Racial and Ethnic Health Disparities, 2024

snippet: Purpose This study aims to understand the impact of the COVID-19 pandemic on social determinants of health (SDOH) of marginalized racial/ethnic US population groups, specifically African Americans and Asians, by leveraging natural language …


title: Improved methodology for longitudinal Web analytics using Common Crawl

authors: HS Thompson - 16th ACM Web Science Conference 2024, 2024

snippet: Common Crawl is a multi-petabyte longitudinal dataset containing over 100 billion web pages which is widely used as a source of language data for sequence model training and in web science research. Each of its constituent archives is on the order …


title: Improving and Understanding Clarifying Question Generation in Conversational Search

authors: D Ortega, S Söhnel, NT Vu - International Conference on Text, Speech, and …, 2024

snippet: Conversational information-seeking systems (CISs), such as chatbots and virtual personal assistants, encounter difficulty when processing ambiguous user requests (URs) and generate an accurate response, especially when multiple search results match …


title: Improving Arithmetical Reasoning of Language Models

snippet: This work focuses on improving language models for solving math word problems. First, we clean and transform existing datasets to create Calc-X, a standardized collection of 300,000 math problems with step-by-step solutions. We demonstrate …


title: Improving Bengali and Hindi Large Language Models

authors: A Shahriar, D Barbosa - Proceedings of the 2024 Joint International Conference …, 2024

snippet: Despite being widely spoken worldwide, Bengali and Hindi are low-resource languages. The state-of-the-art in modeling such languages uses BERT and the Wordpiece tokenizer. We observed that the Wordpiece tokenizer often breaks words …


title: Improving Cross-lingual Representation for Semantic Retrieval with Code-switching

authors: M Maimaiti, Y Zheng, J Zhang, F Huang, Y Zhang… - arXiv preprint arXiv …, 2024

snippet: Semantic Retrieval (SR) has become an indispensable part of the FAQ system in the task-oriented question-answering (QA) dialogue scenario. The demands for a cross-lingual smart-customer-service system for an e-commerce platform or some particular …


title: Improving Parameter-Efficient Cross-Lingual Transfer for Low-Resource Languages

snippet: … The success of the XLM-R can largely be attributed to the quality and size of its training corpora the model is pretrained on more than two terabytes of the filtered Common Crawl data. Notable differences in comparison with mBERT (beyond much …


title: Improving Pretraining Data Using Perplexity Correlations

authors: T Thrush, C Potts, T Hashimoto - arXiv preprint arXiv:2409.05816, 2024

snippet: Quality pretraining data is often seen as the key to high-performance language models. However, progress in understanding pretraining data has been slow due to the costly pretraining runs required for data selection experiments. We present a …


title: Improving the Consistency in Cross-Lingual Cross-Modal Retrieval with 1-to-K Contrastive Learning

authors: Z Nie, R Zhang, Z Feng, H Huang, X Liu - arXiv preprint arXiv:2406.18254, 2024

snippet: … For instance, XLM-R, CCR𝑘 ’s cross-lingual encoder, is trained on the 2.5TB CommonCrawl Corpus encompassing 100 languages. Discrepancies in data sizes between high-resource and low-resource languages within this corpus, like the …


title: INCLURE: a Dataset and Toolkit for Inclusive French Translation

authors: P Lerner, C Grouin - The 17th Workshop on Building and Using …, 2024

snippet: … Indeed, this corpus mostly contains transcripts of political speeches, whose oral style differs from the text typically found in OSCAR/CommonCrawl. Exceptions are six examples used to illustrate the use of the inclusive neutralization process …


title: Incorporating Geo-Diverse Knowledge into Prompting for Increased Geographical Robustness in Object Recognition

authors: K Buettner, S Malakouti, XL Li, A Kovashka - arXiv preprint arXiv:2401.01482, 2024

snippet: Existing object recognition models have been shown to lack robustness in diverse geographical scenarios due to significant domain shifts in design and context. Class representations need to be adapted to more accurately reflect an object concept …


title: Incorporating Human Judgment in AI-Assisted Content Development: The HEAT Heuristic

authors: G Verhulsdonck, J Weible, DM Stambler, T Howard… - Technical Communication, 2024

snippet: Purpose: As technical and professional communicators (TPCs) use AI to develop content, inaccuracies due to AI limitations are introduced; it is vital TPCs evaluate AI-generated content to improve accuracy and human-centeredness. In this article, we present a …


title: IndiBias: A Benchmark Dataset to Measure Social Biases in Language Models for Indian Context

authors: NR Sahoo, PP Kulkarni, N Asad, A Ahmad, T Goyal… - arXiv preprint arXiv …, 2024

snippet: The pervasive influence of social biases in language data has sparked the need for benchmark datasets that capture and evaluate these biases in Large Language Models (LLMs). Existing efforts predominantly focus on English language and the …


title: INDIC QA BENCHMARK: A Multilingual Benchmark to Evaluate Question Answering capability of LLMs for Indic Languages

authors: AK Singh, R Murthy, J Sen, G Ramakrishnan - arXiv preprint arXiv:2407.13522, 2024

snippet: … To address this, we sampled numerous Wikipedia and Common Crawl pages, focusing on paragraphs rich in cultural nuances and domain diversity. Using these paragraphs, we employed large language models (LLMs) to generate QA pairs, thus …


title: IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages

authors: MSUR Khan, P Mehta, A Sankar, U Kumaravelan… - arXiv preprint arXiv …, 2024

snippet: Despite the considerable advancements in English LLMs, the progress in building comparable models for other languages has been hindered due to the scarcity of tailored resources. Our work aims to bridge this divide by introducing an expansive …


title: Indiscriminate Data Poisoning Attacks on Pre-trained Feature Extractors

authors: Y Lu, MYR Yang, G Kamath, Y Yu - arXiv preprint arXiv:2402.12626, 2024

snippet: … To collect such large-scale datasets, practitioners usually extract the desired data by crawling on the internet (eg, web pages using Common Crawl1). However, using outsourced data raises an imminent security risk (Kumar et al. 2020; Nelson et al …


title: Individual-vs. Multiple-Objective Strategies for Targeted Sentiment Analysis in Finances Using the Spanish MTSA 2023 Corpus

authors: R Pan, JA García-Díaz, R Valencia-García - Electronics, 2024

snippet: … complexity; (vi) RoBERTuito [22], a pretrained language model for user-generated content in Spanish, trained following RoBERTa guidelines on 500 million tweets; and (vii) XLM-RoBERTa [17], a multilingual version of RoBERTa, trained with data …


title: IndoBerea: Evolving Semantic Search in Theological Context

authors: FVP Samosir, S Mendrofa - 2023 Eighth International Conference on Informatics …, 2023

snippet: This paper presents IndoBerea, a semantic search model pre-trained on an Indonesian Bible dataset and based on SentenceTransformers and IndoBERT. It aims to enhance theological research by providing contextually relevant verses in …


title: INDUS: Effective and Efficient Language Models for Scientific Applications

authors: B Bhattacharjee, A Trivedi, M Muraoka… - arXiv preprint arXiv …, 2024

snippet: Large language models (LLMs) trained on general domain corpora showed remarkable results on natural language processing (NLP) tasks. However, previous research demonstrated LLMs trained using domain-focused corpora perform better …


title: Inference-Based No-Learning Approach on Pre-Trained BERT Model Retrieval

authors: HL Pham, R Mibayashi, T Yamamoto, MP Kato… - 2024 IEEE International …, 2024

snippet: In recent years, the practice of leveraging pre-trained machine learning models for specific tasks has gained traction. Instead of training models from the ground up, it is now common to fine-tune existing pre-trained models. However, users have the …


title: Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks

authors: N Yax, PY Oudeyer, S Palminteri - arXiv preprint arXiv:2404.04671, 2024

snippet: This paper introduces PhyloLM, a method applying phylogenetic algorithms to Large Language Models to explore their finetuning relationships, and predict their performance characteristics. By leveraging the phylogenetic distance metric, we …


title: Information Retrieval Chatbot on Military Policies and Standards

authors: C Gunasekara, A Sharafeldin, M Triff, Z Kabir…

snippet: In the Canadian Armed Forces (CAF), navigating through extensive policies and standards can be a challenging task. To address the need for streamlined access to these vital documents, this paper explores the usage of artificial intelligence (AI) and …


title: Information Retrieval with Dense and Sparse Representations

snippet: … However, they use 1TB of the training data from Common Crawl dumps while our model only use 115MB of the Wikipedia data for pretraining. We put their scores in Table 2.2 for reference. In SimCSE, the authors propose to use MLM as an auxiliary …


title: Initial exploration into sarcasm and irony through machine translation

authors: ZL Chia, M Ptaszynski, M Karpinska, J Eronen, F Masui - Natural Language …, 2024

snippet: In this paper, we investigate sarcasm and irony as seen through a novel perspective of machine translation. We employ various techniques for translation, comparing both manually and automatically translated datasets of irony and sarcasm. We first …


title: InjectBench: An Indirect Prompt Injection Benchmarking Framework

snippet: … Many datasets such as CommonCrawl cover a much wider distribution of the types of website online compared to the few datasets we have selected. However, we have chosen to use the following datasets to better understand and emphasize …


title: Innovative Use of Self-Attention-Based Ensemble Deep Learning for Suicide Risk Detection in Social Media Posts

authors: HS Choi, J Yang - Applied Sciences, 2024

snippet: … Specifically, we employed the extensive corpus variant, known as “Common Crawl (840 B tokens, 2.2 M vocab, cased, 300 d vectors)”, which encompasses an impressive 840 billion tokens, a vocabulary of 2.2 million cased terms, and provides …


title: Inside the Black Box: Detecting Data Leakage in Pre-trained Language Encoders

authors: Y Xin, Z Li, N Yu, D Chen, M Fritz, M Backes, Y Zhang - arXiv preprint arXiv …, 2024

snippet: Despite being prevalent in the general field of Natural Language Processing (NLP), pre-trained language models inherently carry privacy and copyright concerns due to their nature of training on large-scale web-scraped data. In this paper, we pioneer a …


title: Inspecting and Measuring Fairness of unlabeled Image Datasets

authors: R Görge, M Mock, M Akila - 2024 IEEE 40th International Conference on Data …, 2024

snippet: Bias in training data can lead to algorithmic unfairness in machine learning tasks. Therefore, a general requirement for trustworthy AI is that data should be representative and free of bias. There are several approaches to measure fairness …


title: Instruction-tuned Language Models are Better Knowledge Learners

authors: Z Jiang, Z Sun, W Shi, P Rodriguez, C Zhou, G Neubig… - arXiv preprint arXiv …, 2024

snippet: … However, its scope is limited to Wikipedia, which restricts the trained models’ adaptability to other sources like web pages from Common Crawl or scientific documents from arXiv. We focus on eliciting factual knowledge with instruction-tuning …


title: Integrating Generative AI Literacy into the Information Retrieval Course at a university in Canada: towards critical evaluation of online search results

snippet: … • Trained on information (billions of words) on the open web prior to 2021 • Among other open sources, dataset comes from Common Crawl (crawls the web) and Wikipedia • Length of answers limited - between 500 and 700 words, “leaving …


title: Intelligent Pharmacy

snippet: The modern language generation model ChatGPT, created by Open Artificial Intelligence (AI), is recognised for its capacity to comprehend context and produce pertinent content. This model is built on the transformer architecture, which enables …


title: Intelligent Phishing Website Detection Model Powered by Deep Learning Techniques

authors: U Chetachi, O Henry, OA Agbugba - Asian Journal of Research in Computer …, 2024

snippet: … Approximately one million authentic and fraudulent URLs were employed in the dataset gathered from PhishTank and Common Crawl. Over 10,000 images and one million URLs were used in training for the CNN classifier and LSTM to create the …


title: Intercity relationships between 293 Chinese cities quantified based on toponym co-occurrence

authors: W Tongjing, Z Yin, Z Bao, E Meijers - Cybergeo: European Journal of Geography, 2024

snippet: … 14The primary dataset of this study was obtained from the Common Crawl, a web archive that has periodically crawled the Internet since … We used the entire Common Crawl text archive from April 2019 for processing and conducting …


title: International Journal of Cognitive Computing in Engineering

snippet: … MC4 dataset contains 101 language variants drawn from the Common Crawl web scrape. MT5 did not deviates much from the original T5 model and attempted to extend the capacities of the T5 to multilingual settings. T5 is a pre-trained language …


title: Internet-scale Topic Modeling using Large Language Models

snippet: … This proportion is estimated from the Common Crawl dataset’s statistics [28]. In short, the model should have an excellent understanding of English, an adequate understanding of languages such as German and Spanish, and be satisfactory at …


title: InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning

authors: H Ying, S Zhang, L Li, Z Zhou, Y Shao, Z Fei, Y Ma… - arXiv preprint arXiv …, 2024

snippet: The math abilities of large language models can represent their abstract reasoning ability. In this paper, we introduce and open-source our math reasoning LLMs InternLM-Math which is continue pre-trained from InternLM2. We unify chain-of-thought …


title: InternLM2 Technical Report

authors: Z Cai, M Cao, H Chen, K Chen, K Chen, X Chen… - arXiv preprint arXiv …, 2024

snippet: … Our web page data mainly comes from Common Crawl1. Firstly, we need to decompress the original Warc format files and use Trafilatura (… The entire filtering process eliminates a large proportion of Web page data (Common Crawl) and Patent …


title: Interpretable Tensor Fusion

authors: S Varshneya, A Ledent, P Liznerski, A Balinskyy… - arXiv preprint arXiv …, 2024

snippet: Conventional machine learning methods are predominantly designed to predict outcomes based on a single data type. However, practical applications may encompass data of diverse types, such as text, images, and audio. We introduce …


title: Interpreting Themes from Educational Stories

authors: Y Zhang, FA González, T Solorio - arXiv preprint arXiv:2404.05250, 2024

snippet: Reading comprehension continues to be a crucial research focus in the NLP community. Recent advances in Machine Reading Comprehension (MRC) have mostly centered on literal comprehension, referring to the surface-level …


title: Interrupted time series analysis of clickbait on worldwide news websites, 2016-2023

authors: A McCutcheon, C Brogly - arXiv preprint arXiv:2408.06376, 2024

snippet: … This research provides an analysis of clickbait at the scale of the web on primarily English-language news sites from the Common Crawl. In this analysis, 3 of 5 selected newsgenerating events produced significant terms that suggest major …


title: Intersectional Male-Centric and White-Centric Biases in Collective Concepts

authors: AH Bailey, A Williams, A Poddar, A Cimpian - 2024

snippet: … Here we used NLP tools (specifically, static word embeddings trained on the Common Crawl corpus) to investigate collective understanding of three fundamental concepts: PERSON, WOMAN, and MAN.These three concepts organize much of …


title: Introducing the Djinni Recruitment Dataset: A Corpus of Anonymized CVs and Job Postings

authors: N Drushchak, M Romanyshyn - Proceedings of the Third Ukrainian Natural …, 2024

snippet: This paper introduces the Djinni Recruitment Dataset, a large-scale open-source corpus of candidate profiles and job descriptions. With over 150,000 jobs and 230,000 candidates, the dataset includes samples in English and Ukrainian, thereby …


title: Invasion of ChatGPT and LLMs in Review and Writing Spaces: A Revolution or Radical

authors: A Hadap, V Khatri - Scientific Publishing Ecosystem

snippet: Though AI flourished before 1970s but in the last five years, the invasion of artificial intelligence (AI) tools, such as ChatGPT and Large Language Model (LLMs), has received much attention in scientific research writing. These AI tools assist the …


title: Investigating Complex Answer Attribution Approaches with Large Language Models

snippet: This master’s thesis explores the attribution of answers in complex question-answering scenarios utilizing large language models (LLMs). The research aims to assess and enhance the traceability of answers back to their source documents, a critical aspect …


title: Investigating Conversational Agents to Support Secondary School Computer Science Exploratory Search

snippet: Online educational resources (eg, curricula, tutorials, documentation, Q&A sites) increasingly serve as key sources for secondary school students learning Computer Science Principles (CSP). A big obstacle to using these resources is finding …


title: Investigating Gender Bias in Turkish Language Models

authors: O Caglidil, M Ostendorff, G Rehm - arXiv preprint arXiv:2404.11726, 2024

snippet: … an analysis of undesirable content in the common crawl corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2 …


title: Investigating the Classification of radicalisation phases of Dutch social media messages upon physical threats

snippet: The increase of social media platforms such as Twitter and Telegram has transformed communication, enhanced free speech, but also facilitated the spread of harmful content. This rise in online threats against Dutch politicians highlights the …


title: Investigating the pre-training bias in low-resource abstractive summarization

authors: D Chernyshev, B Dobrov - IEEE Access, 2024

snippet: Recent advances in low-resource abstractive summarization were largely made through the adoption of specialized pre-training, pseudo-summarization, that integrates the content selection knowledge through various centrality-based …


title: IPA Transcription of Bengali Texts

authors: K Fatema, FD Haider, NF Turpa, T Azmal, S Ahmed… - arXiv preprint arXiv …, 2024

snippet: The International Phonetic Alphabet (IPA) serves to systematize phonemes in language, enabling precise textual representation of pronunciation. In Bengali phonology and phonetics, ongoing scholarly deliberations persist concerning the …


title: IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models

authors: DI Adelani, J Ojo, IA Azime, JY Zhuang, JO Alabi, X He… - arXiv preprint arXiv …, 2024

snippet: Despite the widespread adoption of Large language models (LLMs), their remarkable capabilities remain limited to a few high-resource languages. Additionally, many low-resource languages (eg African languages) are often …


title: Is AI changing learning and assessment as we know it? Evidence from a ChatGPT experiment and a conceptual framework

authors: O Kolade, A Owoseni, A Egbetokun - Heliyon, 2024

snippet: ChatGPT, a state-of-the-art chatbot built upon Open AI's generative pre-trained transformer, has generated a major public interest and caused quite a stir in the higher education sector, where reactions have ranged from excitement to …


title: Is ChatGPT taking over the language classroom? How language ideologies of large language models impact teaching and learning

authors: M Lau - Working papers in Applied Linguistics and Linguistics …, 2024

snippet: … Thus, using datasets such as the Common Crawl can lead to the over-representation of English and the perspectives of English-speaking content … This means that LLMs that use Common Crawl for training, such as ChatGPT, may perform with …


title: Is Google Getting Worse? A Longitudinal Investigation of SEO Spam in Search Engines

authors: J Bevendorff, M Wiegmann, M Potthast, B Stein

snippet: Many users of web search engines have been complaining in recent years about the supposedly decreasing quality of search results. This is often attributed to an increasing amount of search-engineoptimized but low-quality content. Evidence for …


title: Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Models

authors: C Liu, W Zhang, Y Zhao, AT Luu, L Bing - arXiv preprint arXiv:2403.10258, 2024

snippet: … We categorize languages larger than 1% frequency in Common Crawl2 as high-resource languages (ie, de, ru, fr, zh, es, ja, it and vi), and the rest as low-resource languages. We exclude English since we want to evaluate the efficient prompting strategy for …


title: It is Time to Develop an Auditing Framework to Promote Value Aware Chatbots

authors: Y Wang, L Singh - arXiv preprint arXiv:2409.01539, 2024

snippet: … According to OpenAI, 60% of the training data come from Common Crawl, a large data set consisting of web pages, extracted metadata and text since 2008. Another 22% of data are from WebText2, containing all Reddit posts until December 2017 …


title: It's About Time: Incorporating Temporality in Retrieval Augmented Language Models

authors: A Gade, J Jetcheva - arXiv preprint arXiv:2401.13222, 2024

snippet: … Additionally both models are fine-tuned jointly as a Fusion-in-Decoder [3] on common-crawl data using Wikipedia as a document index. The authors showed that Atlas performs well out-of-box and can be adapted to different tasks such as question-answering …


title: IUST at ClimateActivism 2024: Towards Optimal Stance Detection: A Systematic Study of Architectural Choices and Data Cleaning Techniques

authors: G Mahmoudi, S Eetemadi - Proceedings of the 7th Workshop on Challenges and …, 2024

snippet: This work presents a systematic search of various model architecture configurations and data cleaning methods. The study evaluates the impact of data cleaning methods on the obtained results. Additionally, we demonstrate that a combination of …


title: Jack of All Trades, Master of Some, a Multi-Purpose Transformer Agent

authors: Q Gallouédec, E Beeching, C Romac, E Dellandréa - arXiv preprint arXiv:2402.09844, 2024

snippet: The search for a general model that can operate seamlessly across multiple domains remains a key goal in machine learning research. The prevailing methodology in Reinforcement Learning (RL) typically limits models to a single task …


title: Japanese Pointer Network based Entity Linker for Wikidata

authors: Y Sawamura, T Morita, S Egami, T Ugai, K Fukuda - 2023

snippet: … Pre-trained fastText is a model trained on Common Crawl and Wikipedia and provides pre-computed vectors for a word. The length of the POS Tags was changed from 36 to 18. The other five embeddings, namely Entity Embedding, Text Match …


title: JetMoE: Reaching Llama2 Performance with 0.1 M Dollars

authors: Y Shen, Z Guo, T Cai, Z Qin - arXiv preprint arXiv:2404.07413, 2024

snippet: Large Language Models (LLMs) have achieved remarkable results, but their increasing resource demand has become a major obstacle to the development of powerful and accessible super-human intelligence. This report introduces JetMoE-8B …


title: Jina CLIP: Your CLIP Model Is Also Your Text Retriever

authors: A Koukounas, G Mastrapas, M Günther, B Wang… - arXiv preprint arXiv …, 2024

authors: H Xiao, G Mastrapas, B Wang - Multi-modal Foundation Model meets Embodied AI …

snippet: Contrastive Language-Image Pretraining (CLIP) is widely used to train models to align images and texts in a common embedding space by mapping them to fixed-sized vectors. These models are key to multimodal information retrieval and related tasks …


title: JiuZhang3. 0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models

authors: K Zhou, B Zhang, J Wang, Z Chen, WX Zhao, J Sha… - arXiv preprint arXiv …, 2024

snippet: Mathematical reasoning is an important capability of large language models~(LLMs) for real-world applications. To enhance this capability, existing work either collects large-scale math-related texts for pre-training, or relies on stronger LLMs (\eg GPT-4) …


title: JLBert: Japanese Light BERT for Cross-Domain Short Text Classification

authors: C Kayal, S Chattopadhyay, A Gupta, S Abrol, A Gugol - Proceedings of the 2024 Joint …, 2024

snippet: … Second, there are limited models that have been trained on diverse datasets beyond Wikipedia and Common Crawl, limiting experimentation with other forms of text data. Moreover, with the English language being the focus of research interest …


title: Journal of Business Academy

snippet: This study examines in detail the potential of artificial intelligence (AI)-assisted ChatGPT in the field of accounting education and the opportunities and challenges that this utilisation may bring. ChatGPT can benefit by reducing error rates and …


title: Juru: Legal Brazilian Large Language Model from Reputable Sources

authors: RM Junior, R Pires, R Romero, R Nogueira - arXiv preprint arXiv:2403.18140, 2024

snippet: The high computational cost associated with pretraining large language models limits their research. Two strategies have emerged to address this issue: domain specialization and pretraining with high-quality data. To explore these strategies, we …


title: Just Rewrite It Again: A Post-Processing Method for Enhanced Semantic Similarity and Privacy Preservation of Differentially Private Rewritten Text

authors: S Meisenbacher, F Matthes - arXiv preprint arXiv:2405.19831, 2024

snippet: The study of Differential Privacy (DP) in Natural Language Processing often views the task of text privatization as a $\textit{rewriting}$ task, in which sensitive input texts are rewritten to hide explicit or implicit private information. In order to evaluate the …


title: KazQAD: Kazakh Open-Domain Question Answering Dataset

authors: R Yeshpanov, P Efimov, L Boytsov, A Shalkarbayuli… - arXiv preprint arXiv …, 2024

snippet: … Kaz-RoBERTa6 is a monolingual model trained on a collection of Kazakh texts from Common Crawl, the Leipzig Corpora Collection, the … XLM-R was pre-trained on a larger multilingual corpus derived from Common Crawl, in which Kazakh is …


title: KazSAnDRA: Kazakh Sentiment Analysis Dataset of Reviews and Attitudes

authors: R Yeshpanov, HA Varol - arXiv preprint arXiv:2403.19335, 2024

snippet: This paper presents KazSAnDRA, a dataset developed for Kazakh sentiment analysis that is the first and largest publicly available dataset of its kind. KazSAnDRA comprises an extensive collection of 180,064 reviews obtained from various sources …


title: KDPII: A New Korean Dialogic Dataset for the Deidentification of Personally Identifiable Information

authors: L Fei, Y Kang, S Park, Y Jang, J Lee, H Kim - IEEE Access, 2024

snippet: … Using this framework, they examined the "Common Crawl's web crawl corpus," a frequently utilized resource for training LMs, to determine whether it contains high-risk personal information. They also compared various detection methods, including …


title: Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI

authors: N Pangakis, S Wolken - arXiv preprint arXiv:2409.09467, 2024

snippet: … Because LLMs are trained on huge volumes of text data, using digital resources such as the Common Crawl, they are able to “memorize” publicly available information during the training process. This issue is compounded due to the fact …


title: KEPA-CRF: Knowledge expansion prototypical amortized conditional random field for few-shot event detection

authors: R Wu, L Yu, S Tian, J Long, T Zhou, B Wang - Journal of Intelligent & Fuzzy Systems

snippet: Event Detection (ED) has long struggled with the ambiguous definition of event categories, making it challenging to accurately classify events. Previous endeavors aimed to tackle this problem by constructing prototypes for specific event categories …


title: Kernel Least Squares Transformations for Cross-Lingual Semantic Spaces

authors: A Mištera, T Brychcín - International Conference on Text, Speech, and …, 2024

snippet: The rapid development in the field of natural language processing (NLP) and the increasing complexity of linguistic tasks demand the use of efficient and effective methods. Cross-lingual linear transformations between semantic spaces play a …


title: Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative tasks

authors: N Chirkova, V Nikoulina - arXiv preprint arXiv:2402.12279, 2024

snippet: … For Intermediate tuning we finetune models for 100k steps on the CommonCrawl data uniformly sampled across all target languages and English, with the batch size of 5k tokens and the LR chosen to optimize fluency of model generations, inspected …


title: Knowledge Induced Transformer Network for Causality Prediction

authors: T Dasgupta, M Sinha, A Naskar - Companion Proceedings of the ACM on Web …, 2024

snippet: … The CausalBank dataset3 contains 314 million pairs of cause-effect statements scraped from the Common Crawl corpus using causal lexical patterns. Baselines: Most of the earlier work on causality detection is based on identifying cause, effect …


title: Knowledge Sources

authors: M Jiang, BY Lin, S Wang, Y Xu, W Yu, C Zhu - Knowledge-augmented Methods for …, 2024

snippet: … To be more specific, the dataset includes Pile-CommonCrawl (227.12G, web crawling data), PubMed Central (90.27G, biomedical articles), Books (100.96G, a mix of fiction and nonfiction books), OpenWebText (262.77G, content from Reddit) …


title: Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models

authors: R Vemulapalli, H Pouransari, F Faghri, S Mehta… - Forty-first International Conference …

snippet: … 2023) which contains 1.28B images filtered from Common Crawl (web-crawled data). We take 10 randomly augmented crops from each image and create a gallery of 12.8B images. See Appendix B.2 for additional details about this gallery set and …


title: Knowledge-augmented Methods for Natural Language Processing

snippet: Ever since language was invented and used, humans have leveraged this powerful tool to pass down experience over generations. These kinds of knowledge summarize precious findings and inspiring ideas, which constitute the human …


title: Knowledge-Enhanced Neural Networks for Machine Reading Comprehension

snippet: Machine Reading Comprehension is a language understanding task where a system is expected to read a given passage of text and typically answer questions about it. When humans assess the task of reading comprehension, in addition to the …


title: KOMPYUTER LINGVISTIKASIDA TA'MINOT MASALASI VA SINONIM BIRLIKLARNING LINGVISTIK TA'MINOTI

authors: N SOTVOLDIYEVA - News of UzMU journal, 2024

snippet: … Lingvistik ma'lumotlar konsorsiumi (LDC) va Common Crawl[8] loyihasi kabi ochiq hamkorlik tashabbuslari lingvistik resurslarning mavjudligiga sezilarli hissa qo'shdi. Bu hamkorlik resurslar almashinuvini targ‘ib qiladi, ortiqchalikni kamaytiradi va …


title: Krishiq-BERT: A Few-Shot Setting BERT Model to Answer Agricultural-Related Questions in the Kannada Language

authors: P Ajawan, V Desai, S Kale, S Patil - Journal of The Institution of Engineers (India) …, 2024

snippet: … corpora for 11 languages from two language families obtained primarily by sourcing articles from news crawls [3], and OSCAR (Open Super-large Crawled ALMAnaCH coRpus), a huge multilingual corpus procured by language …


title: KVP10k: A Comprehensive Dataset for Key-Value Pair Extraction in Business Documents

authors: O Naparstek, R Pony, I Shapira, FA Dahood, O Azulai… - arXiv preprint arXiv …, 2024

snippet: … : extensive web data from Common Crawl and a collection of images from publicfiles.fcc.gov. From Common Crawl, we employed a systematic … A schematic describing the data acquisition process for the common crawl data is shown in Fig.2 …


title: L3i++ at SemEval-2024 Task 8: Can Fine-tuned Large Language Model Detect Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text?

authors: HTH Tran, TN Nguyen, A Doucet, S Pollak - … of the 18th International Workshop on …, 2024

snippet: … Meanwhile, XLM-R is a multilingual version of RoBERTa that was pretrained on 2.5TB of filtered CommonCrawl data containing 100 languages. These models are also suggested as the baseline methods from SemEval2024 Task 8 organizers. …


title: Labadain-30k+: A Monolingual Tetun Document-Level Audited Dataset

authors: G de Jesus, S Nunes - Proceedings of the 3rd Annual Meeting of the Special …, 2024

snippet: This paper introduces Labadain-30k+, a monolingual dataset comprising 33.6 k documents in Tetun, a low-resource language spoken in Timor-Leste. The dataset was acquired through web crawling and augmented with Wikipedia documents …


title: Label-dependent semantic associations for existing concepts

snippet: In a series of studies, we examined whether a label assigned to a cue concept (political event or its actor) modulates(a) its semantic associations (semantic effects) and (b) valence of the associations and attitude judgments (valence effects). We introduced …


title: Ladder: A Model-Agnostic Framework Boosting LLM-based Machine Translation to the Next Level

authors: Z Feng, R Chen, Y Zhang, Z Meng, Z Liu - arXiv preprint arXiv:2406.15741, 2024

snippet: General-purpose Large Language Models (LLMs) like GPT-4 have achieved remarkable advancements in machine translation (MT) by leveraging extensive web content. On the other hand, translation-specific LLMs are built by pre-training on …


title: Language as a Lens: A Hybrid Text Summarization and Sentiment Analysis Approach for Multiclass Stock Return Prediction

authors: F Balaneji - Intelligent Systems Conference, 2024

snippet: This research explores the application of text summarization and sentiment analysis techniques in the multiclass classification of hourly stock price returns, studying six companies from the Dow Jones Index between 2017 and the first quarter of 2020 …


title: Language as a Lens: A Hybrid Text Summarization and Sentiment Analysis Approach for Multiclass Stock Return

authors: F Balaneji - Intelligent Systems and Applications: Proceedings of …


title: Language Model Frame Filling for Low Resource Languages

snippet: Acceptability judgment tasks are key tools used in the understanding of the grammar of languages by linguists. A typical way to construct these tasks is to use semantic frames. We provide evidence that the fill-mask objective for language models (LMs) …


title: Language Models Fine-Tuning for Automatic Format Reconstruction of SEC Financial Filings

authors: G Lombardo, G Trimigno, M Pellegrino, S Cagnoni - IEEE Access, 2024

snippet: The analysis of financial reports is a crucial task for investors and regulators, especially the mandatory annual reports (10-K) required by the SEC (Securities and Exchange Commission) that provide crucial information about a public company in …


title: Language Models for Online Depression Detection: A Review and Benchmark Analysis on Remote Interviews

authors: R Qin, R Cook, K Yang, A Abbasi, D Dobolyi, S Seyedi… - ACM Transactions on …, 2024

snippet: … The project primarily uses data from the English CommonCrawl (67%) but also includes other public data sources including Wikipedia, GitHub, and ArXiv. LLaMA’s ability to outperform GPT-3 on the RACE Reading Comprehension benchmark …


title: Language Models on a Diet: Cost-Efficient Development of Encoders for Closely-Related Languages via Additional Pretraining

authors: N Ljubešić, V Suchomel, P Rupnik, T Kuzman… - arXiv preprint arXiv …, 2024

snippet: The world of language models is going through turbulent times, better and ever larger models are coming out at an unprecedented speed. However, we argue that, especially for the scientific community, encoder models of up to 1 billion parameters …


title: Language Models Pre-training

authors: U Kamath, K Keenan, G Somers, S Sorenson - Large Language Models: A Deep Dive …, 2024

snippet: … The Common Crawl data is the most notable publicly available web scrape. … They demonstrated that these sources improved downstream performance over models trained on less diverse corpora such as Common Crawl. Taking this idea …


title: Language models scale reliably with over-training and on downstream tasks

authors: SY Gadre, G Smyrnis, V Shankar, S Gururangan… - arXiv preprint arXiv …, 2024

snippet: Scaling laws are useful guides for developing language models, but there are still gaps between current scaling studies and how language models are ultimately trained and evaluated. For instance, scaling is usually studied in the compute-optimal …


title: Large Language Model-Based Chatbots in Higher Education

authors: D Yigci, M Eryilmaz, AK Yetisen, S Tasoglu, A Ozcan - Advanced Intelligent Systems, 2024

snippet: Large language models (LLMs) are artificial intelligence (AI) platforms capable of analyzing and mimicking natural language processing. Leveraging deep learning, LLM capabilities have been advanced significantly, giving rise to generative …


title: Large Language Models (LLMs) on Tabular Data: Predic-tion, Generation, and Understanding-A Survey

authors: X Fang, W Xu, FA Tan, J Zhang, Z Hu, Y Qi… - arXiv preprint arXiv …, 2024

snippet: Recent breakthroughs in large language modeling have facilitated rigorous exploration of their application in diverse tasks related to tabular data modeling, such as prediction, tabular data synthesis, question answering, and table …


title: Large Language Models and Rule-Based Approaches in Domain-Specific Communication

authors: D Halvoník, J Kapusta - IEEE Access, 2024

snippet: Currently, we are once again experiencing a frenzy related to artificial intelligence. Generative Pre-trained Transformers (GPT) models are highly effective at various natural language processing tasks. Different varieties of GPT models are widely …


title: Large Language Models and Sentiment Analysis in Financial Markets: A Review, Datasets and Case Study

authors: C Liu, A Arulappan, R Naha, A Mahanti… - IEEE Access, 2024

snippet: This paper comprehensively examines Large Language Models (LLMs) in sentiment analysis, specifically focusing on financial markets and exploring the correlation between news sentiment and Bitcoin prices. We systematically categorize various …


title: Large Language Models and the Disappearing Private Sphere

authors: A Cappello, M Dada, V Grigoreva, R Khan, C Stinson… - 2024

snippet: … The higher quality datasets are sampled more often during training, but the Reddit approved contents of CommonCrawl still … of CommonCrawl contains fan fiction, video game chats, conspiracy theories, pornography, junk advertising, and …


title: Large Language Models aren't all that you need

authors: KV Holla, C Kumar, A Singh - arXiv preprint arXiv:2401.00698, 2024

snippet: This paper describes the architecture and systems built towards solving the SemEval 2023 Task 2: MultiCoNER II (Multilingual Complex Named Entity Recognition) [1]. We evaluate two approaches (a) a traditional Conditional Random …


title: LARGE LANGUAGE MODELS AS AN AID FOR SOFTWARE DEVELOPMENT USING GRAPHICAL PROGRAMMING LANGUAGE.

snippet: … In pre-training, the model is trained with the Common Crawl dataset, internet books, and Python code, summing up to 400B tokens. After the model is pre-trained, the phase preference model pre-training comes, and after that, the preference model …


title: Large Language Models As Genetic Counselors: AI-DRIVEN TOOLS FOR HEALTH MANAGEMENT

snippet: This thesis, in collaboration with Arkus-AI, explores the development of an AI-driven genetic counselor designed to enhance genetic counseling services at Eiira. By utilizing advanced large language models, this project aims to improve the quality …


title: Large Language Models As Genetic Counselors: Personalized AI Driven Tool for Health Management


title: Large language models as oracles for instantiating ontologies with domain-specific knowledge

authors: G Ciatto, A Agiollo, M Magnini, A Omicini - arXiv preprint arXiv:2404.04108, 2024

snippet: … Models following the RoBERTa [30] approach were trained on (i) the same datasets as BERT, plus (ii) a sample of the CommonCrawl4 News dataset (a dump of news … 4https://commoncrawl.org 5https://mistral.ai/news/mixtral-of-experts …


title: Large Language Models for Biomedical Text Simplification: Promising But Not There Yet

authors: Z Li, S Belkadi, N Micheletti, L Han, MSG Nenadic

snippet: … the common crawl corpus and filtered to keep only natural text and de-duplication processing. They extracted 750GB of clean English data to feed into the model for multi-task pretraining. Different masking strategies are integrated into the T5 model …


title: Large language models for biomedicine: foundations, opportunities, challenges, and best practices

authors: SS Sahoo, JM Plasek, H Xu, Ö Uzuner, T Cohen… - Journal of the American …, 2024

snippet: … In the pre-training phase, generic training data, such as the Common Crawl dataset or social media posts, are used for unsupervised learning.With further training on prompt-response pairs (such as the Alpaca dataset developed at Stanford23) …


title: Large Language Models for Forecasting and Anomaly Detection: A Systematic Literature Review

authors: J Su, C Jiang, X Jin, Y Qiao, T Xiao, H Ma, R Wei… - arXiv preprint arXiv …, 2024

snippet: … The training data for GPT-3 includes the Common Crawl dataset for lower quality, as well as the WebText2 dataset for higher quality, as well as the Books1, Books2, and Wikipedia datasets for higher quality [148]. GPT-3 assigns different weights to …


title: Large Language Models for Human-Machine Collaborative Particle Accelerator Tuning through Natural Language

authors: J Kaiser, A Eichler, A Lauscher - arXiv preprint arXiv:2405.08888, 2024

snippet: Autonomous tuning of particle accelerators is an active and challenging field of research with the goal of enabling novel accelerator technologies cutting-edge high-impact applications, such as physics discovery, cancer research and material sciences. A …


title: Large Language Models for identification of medical data in unstructured records

snippet: … Sources for the training data include web pages extracted by CommonCrawl, opensource GitHub repositories, Wikipedia in twenty languages, Project Gutenberg books in the public domain, LaTeX source code of scientific articles in ArXiv and …


title: Large Language Models for Relevance Judgement in Table Retrieval

snippet: … The tables come from the English subset of the Web Data Commons (WDC) table corpus 2015, which contains 50.8M HTML tables from the July 2015 Common Crawl The creators of the WTR collection further filtered the collection by identifying entities …


title: Large Language Models for Reliable Information Extraction

snippet: This project addresses the ongoing challenge of achieving reliable Information Extraction (IE), particularly in domains which require near-perfect precision. ELICIT (Butcher et al., 2023) introduced a novel approach that combines the processing speed of …


title: Large Language Models for Software Engineering: A Systematic Mapping Study

authors: MK Görmez, M Yılmaz, PM Clarke - European Conference on Software Process …, 2024

snippet: In this research, we aim to conduct a systematic mapping study on Large Language Models (LLMs) for Software Engineering (SE). The significantly enhanced capabilities of LLMs have led to their use in many fields, including the important …


title: LARGE LANGUAGE MODELS FOR TEXT CLASSIFICATION: FROM ZERO-SHOT LEARNING TO INSTRUCTION-TUNING

snippet: Advances in large language models (LLMs) have transformed the field of natural language processing and have enormous potential for social scientific analysis. We explore the application of LLMs to supervised text classification. As a case study, we …


title: Large language models in healthcare: from a systematic review on medical examinations to a comparative analysis on fundamentals of robotic surgery online test

authors: A Moglia, K Georgiou, P Cerveri, L Mainardi… - Artificial Intelligence Review, 2024

snippet: Large language models (LLMs) have the intrinsic potential to acquire medical knowledge. Several studies assessing LLMs on medical examinations have been published. However, there is no reported evidence on tests related to robot-assisted …


title: Large Language Models on Tabular Data--A Survey

snippet: Recent breakthroughs in large language modeling have facilitated rigorous exploration of their application in diverse tasks related to tabular data modeling, such as prediction, tabular data synthesis, question answering, and table …


title: Large Language Models Understand Layouts

authors: W Li, M Duan, D An, Y Shao - arXiv preprint arXiv:2407.05750, 2024

snippet: … We find that GLM, Llama, and GPT-3 all use datasets such as CommonCrawl, Wikipedia, and Books (Pile includes CommonCrawl, Wikipedia, and Books) in their pretraining. CommonCrawl is a large-scale, unstructured, multilingual web dataset …


title: Large Language Models, scientific knowledge and factuality: A framework to streamline human expert evaluation

authors: M Wysocka, O Wysocki, M Delmas, V Mutel, A Freitas - Journal of Biomedical …, 2024

snippet: Objective: The paper introduces a framework for the evaluation of the encoding of factual scientific knowledge, designed to streamline the manual evaluation process typically conducted by domain experts. Inferring over and extracting information from …


title: Large Language Models: A Case Study on Corporate Values and Transformative Technologies

snippet: Design (VSD) a technique that provides theories and methods to systematically “account for human values… throughout the design process”(Value Sensitive Design Lab, nd). This paper employs VSD as a theoretical lens to dissect the dual challenges of …


title: Large Language Models: A Deep Dive

authors: U Kamath, K Keenan, G Somers, S Sorenson

snippet: In the panorama of technological evolution, Large Language Models (LLMs) have emerged as a cornerstone, transforming our interaction with information, reshaping industries, and redefining the boundaries of artificial intelligence. As we stand on the …


title: Large Language Models: A Survey

authors: S Minaee, T Mikolov, N Nikzad, M Chenaghlu… - arXiv preprint arXiv …, 2024

snippet: … Common Crawlbased dataset consisting of texts in 101 languages. … CommonCrawl. They also released an extract of 600 billion tokens from our REFINEDWEB dataset, and 1.3/7.5B parameters language models trained on it. 27 …


title: Large Language Models: An Introduction

authors: U Kamath, K Keenan, G Somers, S Sorenson - Large Language Models: A Deep Dive …, 2024

snippet: This chapter begins with a discussion of the historical context and progression of natural language processing. Beginning with the origins of human linguistic capabilities, this chapter explains the gradual transition to computational language …


title: Large Malaysian Language Model Based on Mistral for Enhanced Local Language Understanding

authors: H Zolkepli, A Razak, K Adha, A Nazhan - arXiv preprint arXiv:2401.13565, 2024

snippet: … An analysis of the widely utilized Common Crawl dataset reveals a mere 0.0742% contribution from the Malay language based on CCMAIN-… We also replicated the same technique for Malaysia Hansard and MS CommonCrawl samples. All synthetic …


title: Large Scale Transfer Learning for Tabular Data via Language Modeling

authors: J Gardner, JC Perdomo, L Schmidt - arXiv preprint arXiv:2406.12031, 2024

snippet: … Given that T4 consists of 3.1Mtables sourced from public data sources (Common Crawl, Github) and that our evaluations are also comprised of public benchmarks, we investigate the extent and possible impact of data contamination – that is …


title: Large Vocabulary Size Improves Large Language Models

authors: S Takase, R Ri, S Kiyono, T Kato - arXiv preprint arXiv:2406.16508, 2024

snippet: This paper empirically investigates the relationship between subword vocabulary size and the performance of large language models (LLMs) to provide insights on how to define the vocabulary size. Experimental results show that larger vocabulary …


title: Large-language models facilitate discovery of the molecular signatures regulating sleep and activity

authors: D Peng, L Zheng, D Liu, C Han, X Wang, Y Yang… - Nature Communications, 2024

snippet: Sleep, locomotor and social activities are essential animal behaviors, but their reciprocal relationships and underlying mechanisms remain poorly understood. Here, we elicit information from a cutting-edge large-language model (LLM), generative pre-trained …


title: Large-Language-Models (LLM)-Based AI Chatbots: Architecture, In-Depth Analysis and Their Performance Evaluation

authors: V Kumar, P Srivastava, A Dwivedi, I Budhiraja… - International Conference on …, 2023

snippet: … LLaMA models span from 7 billion to 65 billion parameters and were trained on trillions of tokens derived from publicly accessible datasets, including Common Crawl and Wikipedia. The primary objective of LLaMA models is to serve as …


title: Large-scale machine unlearning in transformer-based pre-trained language models: Evaluation and adaptation of existing machine unlearning frameworks KGA and …

snippet: In the past few years, Machine Learning (ML) models such as GPT-4, utilized by the prominent chatbot ChatGPT, have faced data breaches, raising concerns about data privacy and awareness about the "right to be forgotten." Consequently, the concept …


title: Latxa: An Open Language Model and Evaluation Suite for Basque

authors: J Etxaniz, O Sainz, N Perez, I Aldabe, G Rigau, E Agirre… - arXiv preprint arXiv …, 2024

snippet: We introduce Latxa, a family of large language models for Basque ranging from 7 to 70 billion parameters. Latxa is based on Llama 2, which we continue pretraining on a new Basque corpus comprising 4.3M documents and 4.2B tokens. Addressing the …


title: Law and the Political Economy of AI Production

snippet: The governance of Artificial Intelligence (AI) is at a historical juncture. Legislative acts, global treaties, export controls, and technical standards are now dominating the discourse over what used to be a predominantly market-driven space. Amidst all this …


title: Law, Technology and Humans

snippet: … Around 60% of GPT-3 training data comes from a filtered version of Common Crawl—that is, a non-profit organisation that archives and provides open-access petabytes of data collected since 2011. [30] The remaining 40% comes from diverse …


title: LAWSUIT: a LArge expert-Written SUmmarization dataset of ITalian constitutional court verdicts

authors: L Ragazzi, G Moro, S Guidi, G Frisoni - Artificial Intelligence and Law, 2024

snippet: Large-scale public datasets are vital for driving the progress of abstractive summarization, especially in law, where documents have highly specialized jargon. However, the available resources are English-centered, limiting research …


title: Learning entity and relation representation for low-resource medical language processing

snippet: Recent advancements in natural language processing have led to growing interest in critical domains such as legal, finance, and healthcare. In particular, medical language processing has emerged as a focus of its own. As the number of …


title: Learning Fine-Grained Grounded Citations for Attributed Large Language Models

authors: L Huang, X Feng, W Ma, Y Gu, W Zhong, X Feng, W Yu… - arXiv preprint arXiv …, 2024

snippet: … 2021), a pre-processed and cleaned version of the Common Crawl corpus, serving as a proxy web search index. In particular, for a given user query sampled from the AQuAMuSe dataset, we initially retrieve the top 100 relevant documents …


title: Learning from Noisy Data for Cross Lingual Text Generation in Low-Resource Languages

snippet: With Large Language Models (LLMs) and Language models in general becoming a more significant part of our daily content consumption, it is paramount to ensure that languages with fewer resources do not get excluded. As most language models are …


title: Learning to Generate Answers with Citations via Factual Consistency Models

authors: R Aly, Z Tang, S Tan, G Karypis - arXiv preprint arXiv:2406.13124, 2024

snippet: Large Language Models (LLMs) frequently hallucinate, impeding their reliability in mission-critical situations. One approach to address this issue is to provide citations to relevant sources alongside generated content, enhancing the verifiability of …


title: Learning with Noisy Foundation Models

authors: H Chen, J Wang, Z Wang, R Tao, H Wei, X Xie… - arXiv preprint arXiv …, 2024

snippet: … GPT2 is pre-trained on WebText [29], a scraped web dataset from Common Crawl that contains low-quality raw texts. We also leverage OpenAI’s API service “text-ada-002”. We cannot use larger and more recent language models such as LLaMA [30], since …


title: Leveraging AI to Deliver Culturally Responsive Mental Health Care at Scale

authors: A Cerezo, D Cooper, V Palat, A Jolley, S Peregrine

snippet: … One of the most commonly used datasets to train LLMs is The Common Crawl dataset, a large collection of web pages scraped from the internet since 2014. These pages contain the gamut from high-quality news, science, and literature to the …


title: Leveraging distant supervision and deep learning for twitter sentiment and emotion classification

authors: M Kastrati, Z Kastrati, A Shariq Imran, M Biba - Journal of Intelligent Information …, 2024

snippet: Nowadays, various applications across industries, healthcare, and security have begun adopting automatic sentiment analysis and emotion detection in short texts, such as posts from social media. Twitter stands out as one of the most popular …


title: Leveraging Large Language Models to Measure Gender Bias in Gendered Languages

authors: E Derner, SS de la Fuente, Y Gutiérrez, P Moreda… - arXiv preprint arXiv …, 2024

snippet: Gender bias in text corpora used in various natural language processing (NLP) contexts, such as for training large language models (LLMs), can lead to the perpetuation and amplification of societal inequalities. This is particularly …


title: Leveraging LLMs for Collaborative Ontology Engineering in Parkinson Disease Monitoring and Alerting

authors: G Bouchouras, D Doumanas, A Soularidis, K Kotis…

snippet: This paper explores the integration of Large Language Models (LLMs) in the engineering of a Parkinson’s Disease (PD) monitoring and alerting ontology through four key methodologies: One Shot (OS) prompt techniques, Chain of Thought (CoT) …


title: Leveraging Open Large Language Models for Multilingual Policy Topic Classification: The Babel Machine Approach

authors: M Sebők, Á Máté, O Ring, V Kovács, R Lehoczki - Social Science Computer Review, 2024

snippet: The article presents an open-source and freely available natural language processing system for comparative policy studies. The CAP Babel Machine allows for the automated classification of input files based on the 21 major policy topics of …


title: Leveraging Parameter Efficient Training Methods for Low Resource Text Classification: A case study in Marathi

authors: P Deshmukh, N Kulkarni, S Kulkarni, K Manghani… - 2024 IEEE 9th International …, 2024

snippet: With the surge in digital content in low-resource languages, there is an escalating demand for advanced Natural Language Processing (NLP) techniques tailored to these languages. BERT (Bidirectional Encoder Representations from Transformers) …


title: Leveraging Universal Speech Representations for Detecting and Assessing the Severity of Mild Cognitive Impairment Across Languages

authors: A Favaro, T Cao, N Dehak, L Moro-Velazquez

snippet: This study examines the suitability of language-agnostic features for automatically detecting Mild Cognitive Impairment (MCI) and predicting Mini-Mental State Examination (MMSE) scores in a multilingual framework. We explored two methods …


title: Leveraging Zero-Shot Prompting for Efficient Language Model Distillation

authors: L Vöge, V Gurgul, S Lessmann - arXiv preprint arXiv:2403.15886, 2024

snippet: … Specifically, we use the T5 v1.1 models which were only pretrained on the C4 Common Crawl dataset for general language understanding without additional task-specific finetuning. The use of text-to-text models allows for straightforward processing, as …


title: LexEval: A Scalable LLM Evaluation Framework

snippet: As Large Language Models (LLMs) become more powerful and their parameter counts increase, it is crucial for them to exhibit consistent recall capabilities for terms that appear less frequently in their pre-training data. The importance of accurate …


title: Lib2Life–Digital Library Services Empowered with Advanced Natural Language Processing Techniques

authors: M Nitu, M Dascalu, MD Dascalu, LM Neagu…

snippet: Educational institutions are struggling to keep up with the accelerated technological advancements; hence, sustainable and supportive tools have become essential to reshape traditional models into intelligent learning systems. This paper introduces …


title: LiBERTa: Advancing Ukrainian Language Modeling through Pre-training from Scratch

authors: M Haltiuk, A Smywiński-Pohl - Proceedings of the Third Ukrainian Natural Language …, 2024

snippet: … CC-100, a multilingual text corpus sourced from Wikipedia and CommonCrawl, was processed following the CCNet1 methodology. The Ukrainian segment of CC-100 encompasses 6.5 billion tokens, equivalent to 84 GiB of data2. This corpus primarily …


title: Limitations of Language Models in The Oil & Gas Upstream Operations

authors: A Alsultan, FA Razak - International Petroleum Technology Conference, 2024

snippet: … Although the process is simple in principle, it is done on very large training corpra which can be as big as dataset provided by the Common Crawl project (428.81 TB as of the date of writing this paper). Thus, training these models using these large …


title: Limits to Predicting Online Speech Using Large Language Models

authors: M Remeli, M Hardt, RC Williamson - arXiv preprint arXiv:2407.12850, 2024

snippet: We study the predictability of online speech on social media, and whether predictability improves with information outside a user's own posts. Recent work suggests that the predictive information contained in posts written by a user's peers …


title: LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages

authors: AM Bean, S Hellsten, H Mayne, J Magomere, EA Chi… - arXiv preprint arXiv …, 2024

snippet: … ], while the second and fourth rows are sorted by Common Crawl Share [16]. The upper two … of pages tagged with a given language in the Common Crawl dataset [16]. Figure 12 show the … LLMs are trained on, using the Common Crawl share and …


title: Linguistic Profiling of Deepfakes: An Open Database for Next-Generation Deepfake Detection

authors: Y Wang, Z Huang, Z Ma, X Hong - arXiv preprint arXiv:2401.02335, 2024

snippet: The emergence of text-to-image generative models has revolutionized the field of deepfakes, enabling the creation of realistic and convincing visual content directly from textual descriptions. However, this advancement presents considerably greater …


title: Linguistic Structure from a Bottleneck on Sequential Information Processing

authors: R Futrell, M Hahn - arXiv preprint arXiv:2405.12109, 2024

snippet: … Table 3: English expressions for the given meanings, along with frequencies from the English Common Crawl web corpus [64]. Example … n-gram counts and language models from the common crawl. In Language Resources and Evaluation …


title: Linguistic_Hygenist at PAN 2024 TextDetox: HybridDetox-A Combination of Supervised and Unsupervised Methods for Effective Multilingual Text Detoxification

authors: S Gangopadhyay, MT Khan, H Jabeen - 2024

snippet: … To handle these, we used the XLM-RoBERTa large model [25], which is pretrained in a self-supervised manner on 2.5TB of filtered CommonCrawl data spanning 100 languages, including all languages featured in the competition. The model employs a …


title: LITS: An Optimized Learned Index for Strings (An Extended Version)

authors: Y Yang, S Chen - arXiv preprint arXiv:2407.11556, 2024

snippet: Index is an important component in database systems. Learned indexes have been shown to outperform traditional tree-based index structures for fixed-sized integer or floating point keys. However, the application of the learned solution to variable-length …


title: LITS: An Optimized Learned Index for Strings

authors: Y Yang, S Chen - Proceedings of the VLDB Endowment, 2024


title: Llama (LLM)

snippet: … Data was mostly from CommonCrawl and C4. Llama-2-70B saw 2T tokens but Llama-3-70B saw 15T tokens. Llama-3-8B and Llama-3-70B took 1.3M and 6.4M GPU hours for pretraining. …


title: LLaMA Beyond English: An Empirical Study on Language Capability Transfer

authors: J Zhao, Z Zhang, Q Zhang, T Gui, X Huang - arXiv preprint arXiv:2401.01055, 2024

snippet: In recent times, substantial advancements have been witnessed in large language models (LLMs), exemplified by ChatGPT, showcasing remarkable proficiency across a range of complex tasks. However, many mainstream LLMs (eg LLaMA) are …


title: LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training

authors: T Zhu, X Qu, D Dong, J Ruan, J Tong, C He, Y Cheng - arXiv preprint arXiv …, 2024

snippet: … As to the Dev-to-Train differences in Figure 9b, we find HellaSwag and ARC-c share the most similar expert preferences with CommonCrawl and C4, and GSM-8K is similar to arXiv. This may provide some insights for continual pre-training to further …


title: LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languages

authors: Y Lu, W Zhu, L Li, Y Qiao, F Yuan - arXiv preprint arXiv:2407.05975, 2024

snippet: Large Language Models~(LLMs) demonstrate remarkable translation capabilities in high-resource language tasks, yet their performance in low-resource languages is hindered by insufficient multilingual data during pre-training. To address this, we …


title: LLM-Assistance for Quality Control of LLM Output

authors: K Sandkuhl - Perspectives in Business Informatics Research: 23rd …

snippet: … GPT-3, the model behind ChatGPT, operates with 175 billion parameters and utilizes datasets that include nearly a trillion words sourced from various corpora, such as the Common Crawl, WebText2, internet-based books, and the English …


title: LLM-Based Edge Intelligence: A Comprehensive Survey on Architectures, Applications, Security and Trustworthiness

authors: O Friha, MA Ferrag, B Kantarci, B Cakmak, A Ozgun… - IEEE Open Journal of the …, 2024

snippet: The integration of Large Language Models (LLMs) and Edge Intelligence (EI) introduces a groundbreaking paradigm for intelligent edge devices. With their capacity for human-like language processing and generation, LLMs empower edge …


title: LLM-Datasets: An Open Framework for Pretraining Datasets of Large Language Models

authors: M Ostendorff, PO Suarez, LF Lage, G Rehm

snippet: … All of the numbers are given by Common Crawl dump. … 2 show the number of documents by both date of the CommonCrawl dump and language, the percentage of documents removed at each filtering step, and the final document and token …


title: LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs

authors: A Aizawa, E Aramaki, B Chen, F Cheng, H Deguchi… - arXiv preprint arXiv …, 2024

snippet: … The statistics of the Japanese corpus from Common Crawl dumps are presented in Figure 2. We used the publicly available Common Crawl dumps from 2013 to the middle of 2023. We merged the Common Crawl dumps from 2013 to 2016 because …


title: LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement

authors: N Lee, T Wattanawong, S Kim, K Mangalam, S Shen… - arXiv preprint arXiv …, 2024

snippet: … We assume that we are given an LLM model M (eg, GPT-3.5 or LLaMA2-7B) that is pre-trained on some source dataset (eg, Common Crawl). The goal is to adapt M (hereon called the student model) to a new target domain by using a small seed dataset D …


title: LLM4Schema. org: Generating Schema. org Markups with Large Language Models

authors: MH Dang, THT Pham, P Molli, H Skaf-Molli, A Gaignard

snippet: The integration of Schema. org markup in web pages has resulted in the creation of billions of RDF triples, yet approximately 75% of all web pages still lack these essential markups. Large Language Models (LLMs) offer a potential solution by …


title: LLMs achieve adult human performance on higher-order theory of mind tasks

authors: W Street, JO Siy, G Keeling, A Baranes, B Barnett… - arXiv preprint arXiv …, 2024

snippet: This paper examines the extent to which large language models (LLMs) have developed higher-order theory of mind (ToM); the human ability to reason about multiple mental and emotional states in a recursive manner (eg I think that you …


title: LLMs for Low Resource Languages in Multilingual, Multimodal and Dialectal Settings

authors: F Alam, SA Chowdhury, S Boughorbel, M Hasanain - … of the 18th Conference of the …, 2024

snippet: The recent breakthroughs in Artificial Intelligence (AI) can be attributed to the remarkable performance of Large Language Models (LLMs) across a spectrum of research areas (eg, machine translation, question-answering, automatic speech …


title: LLMs Still Can't Avoid Instanceof: An Investigation Into GPT-3.5, GPT-4 and Bard's Capacity to Handle Object-Oriented Programming Assignments

authors: BP Cipriano, P Alves - arXiv preprint arXiv:2403.06254, 2024

snippet: … are high-level descriptions of the training datasets (eg a filtered version of Common Crawl 14, English Wikipedia, among others) [3], it was not possible to perform this analysis for those models. 13https:github.com/camilaabrantes/CursoJava …


title: LMaaS: Exploring Pricing Strategy of Large Model as a Service for Communication

authors: P Wu, Q Liu, Y Dong, F Wang - arXiv preprint arXiv:2401.02675, 2024

snippet: The next generation of communication is envisioned to be intelligent communication, that can replace traditional symbolic communication, where highly condensed semantic information considering both source and channel will be extracted and …


title: LMCK: pre-trained language models enhanced with contextual knowledge for Vietnamese natural language inference

authors: NLT Nguyen, KTK Phan, TV Huynh, KV Nguyen - Multimedia Tools and Applications, 2024

snippet: … trained with a cross-lingual masked language modeling objective on data in 100 languages, including Vietnamese from Common Crawl. … a combination of span masking and sentence shuffling objectives on a subset of 25 languages, including …


title: LogogramNLP: Comparing Visual and Textual Representations of Ancient Logographic Writing Systems for NLP

authors: D Chen, F Shi, A Agarwal, J Myerston… - arXiv preprint arXiv …, 2024

snippet: … 2023), a pixel-based language model pre-trained on the Common Crawl dataset with a masked image modeling objective, to encode the … Among the aforementioned models, T5, ByT5, and PIXEL are pre-trained on large-scale text …


title: Long-input summarization using Large Language Models

snippet: … the colossal and cleaned version of common crawl CNN … T5 was pre-trained on the C4 (The Colossal and Cleaned version of Common Crawl) dataset. C4 contains text from 350M web pages; the dataset is 750 GB. A maximum sequence length of …


title: Long-Term Ad Memorability: Understanding & Generating Memorable Ads

authors: SI Harini, S Singh, Y Kumar, A Bhattacharyya, V Baths…

snippet: Marketers spend billions of dollars on advertisements, but to what end? At purchase time, if customers cannot recognize the brand for which they saw an ad, the money spent on the ad is essentially wasted. Despite its importance in marketing, until now …


title: LongWanjuan: Towards Systematic Measurement for Long Text Quality

authors: K Lv, X Liu, Q Guo, H Yan, C He, X Qiu, D Lin - arXiv preprint arXiv:2402.13583, 2024

snippet: … In the English data, the CommonCrawl domain predominates, accounting for over 50% of the data. Apart from a significant amount of aggregated texts in the CommonCrawl domain, the majority of data in other domains consists of holistic texts …


title: Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretraining

authors: Q Zhang, T Du, H Huang, Y Wang, Y Wang - Forty-first International Conference on Machine …

snippet: In recent years, the rise of generative self-supervised learning (SSL) paradigms has exhibited impressive performance across visual, language, and multi-modal domains. While the varied designs of generative SSL objectives lead to distinct properties in …


title: Look Hear: Gaze Prediction for Speech-directed Human Attention

authors: S Mondal, S Ahn, Z Yang, N Balasubramanian… - arXiv preprint arXiv …, 2024

snippet: For computer systems to effectively interact with humans using spoken language, they need to understand how the words being generated affect the users' moment-by-moment attention. Our study focuses on the incremental prediction of attention as a person is …


title: LoRA-Enhanced Language Alignments for Robust Code-Mixed Text Representation

authors: X Ran, Y Zhang, J Wang, D Xu, X Zhang - 2024 International Joint Conference on …, 2024

snippet: … • XLM-RoBERTa-Large [3] is a multilingual version of RoBERTa, which is pre-trained on a filtered Common Crawl dataset containing 100 languages. • XMFGM [31] applies adversarial training to construct some adversarial samples via adding …


title: LoRA-Guard: Parameter-Efficient Guardrail Adaptation for Content Moderation of Large Language Models

authors: H Elesedy, PM Esperança, SV Oprea, M Ozay - arXiv preprint arXiv:2407.02987, 2024

snippet: Guardrails have emerged as an alternative to safety alignment for content moderation of large language models (LLMs). Existing model-based guardrails have not been designed for resource-constrained computational portable devices, such …


title: Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training

authors: Z Zhong, M Xia, D Chen, M Lewis - arXiv preprint arXiv:2405.03133, 2024

snippet: … We suspect that segment-level routing may be particularly effective at handling out-of-domain evaluation data (we assume that there is only a very small part of Python code in Commoncrawl). Our analysis in Section 5.4 shows that segment-level routing …


title: Low-Complexity and Secure Clustering-Based Similarity Detection for Private Files

snippet: Detection of the similarity between files is a requirement for many practical applications, such as copyright protection, file management, plagiarism detection, and detecting duplicate submissions of scientific articles to multiple journals or …


title: M2SA: Multimodal and Multilingual Model for Sentiment Analysis of Tweets

authors: G Thakkar, S Hakimov, M Tadić - arXiv preprint arXiv:2404.01753, 2024

snippet: … 2020b) is a large multilingual language model trained on 2.5 TB of filtered Common Crawl data containing 100 languages. The model was trained with the Masked Language Modelling (MLM) objective, with 15% of the input words masked …


title: Machine Learning and the Analysis of Culture

snippet: The focus of this chapter is on how machine learning (ML) impacts the analysis of culture in sociology. It shows how ML has greatly advanced the analysis of culture, with new tools enabling a massive and fine-grained extraction of information from …


title: Machine Translation that Peeks at the Reference

snippet: Abstract Machine translation with lexical constraints is a popular research topic, especially for terminology translation. Existing approaches for lexical control in MT are usually complex and not easily applicable to all existing MT toolkits. We propose …


title: Machine-Learning Phishing Detection Model Used in the E-Banking Environment

authors: M Manala, J Jansen van Vuuren - IFIP International Conference on Human Choice …, 2024

snippet: … Rashid, Mahmood [20] proposed a machine-learning approach that uses Alexa and the Common Crawl archive dataset with 5000 URLs. … Geyik, Erensoy [21] used the PhishTank Alexa Common-Crawl datasets. The performance attained …


title: Machines Do See Color: A Guideline to Classify Different Forms of Racist Discourse in Large Corpora

authors: DD Gordillo, J Timoneda, SV Vera - arXiv preprint arXiv:2401.09333, 2024

snippet: Current methods to identify and classify racist language in text rely on small-n qualitative approaches or large-n approaches focusing exclusively on overt forms of racist discourse. This article provides a step-by-step generalizable guideline to …


title: MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions

authors: K Zhang, Y Luan, H Hu, K Lee, S Qiao, W Chen, Y Su… - arXiv preprint arXiv …, 2024

snippet: … We collect all images with the same URL from Common Crawl2 as a group of images from the same web page for potential pairing. Due to the inevitable noisy images introduced by simple grouping, we remove duplicated, low-resolution, and …


title: MAiDE-up: Multilingual Deception Detection of GPT-generated Hotel Reviews

authors: O Ignat, X Xu, R Mihalcea - arXiv preprint arXiv:2404.12938, 2024

snippet: Deceptive reviews are becoming increasingly common, especially given the increase in performance and the prevalence of LLMs. While work to date has addressed the development of models to differentiate between truthful and …


title: MaiNLP at SemEval-2024 Task 1: Analyzing Source Language Selection in Cross-Lingual Textual Relatedness

authors: S Zhou, H Shan, B Plank, R Litschko - arXiv preprint arXiv:2404.02570, 2024

snippet: This paper presents our system developed for the SemEval-2024 Task 1: Semantic Textual Relatedness (STR), on Track C: Cross-lingual. The task aims to detect semantic relatedness of two sentences in a given target language without access to …


title: MambaByte: Token-free Selective State Space Model

authors: J Wang, T Gangavarapu, JN Yan, AM Rush - arXiv preprint arXiv:2401.13660, 2024

snippet: Token-free language models learn directly from raw bytes and remove the bias of subword tokenization. Operating on bytes, however, results in significantly longer sequences, and standard autoregressive Transformers scale poorly in such settings …


title: MAmmoTH2: Scaling Instructions from the Web

authors: X Yue, T Zheng, G Zhang, W Chen - arXiv preprint arXiv:2405.03548, 2024

snippet: … We argue that the pre-training corpus (eg, Common Crawl) already contains a vast amount of high-quality instruction data for LLM reasoning. For example, the web corpus contains a large amount of educational materials in the form of instruction-following …


title: MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series

authors: G Zhang, S Qu, J Liu, C Zhang, C Lin, CL Yu, D Pan… - arXiv preprint arXiv …, 2024

snippet: … Inspired by [89], which adopts an iterative pipeline to facilitate the acquisition of large-scale, high-quality data from Common Crawl, we propose to select high-quality data for mathematics, scientific exam synthetic data, and wiki-based content in our Matrix. …


title: Mapping the media landscape: predicting factual reporting and political bias through web interactions

authors: D Sánchez-Cortés, S Burdisso, E Villatoro-Tello… - International Conference of …, 2024

snippet: … In this work we used the graph G built in [8] consisting of 17K news sources obtained after processing 100M news articles from Common Crawl News. Following [3, 8] we report 5-fold cross-validation evaluation results on our MBFC datasets …


title: Mapping the Positive Self-Bias Embedded in Human Languages

snippet: … Here, we employed the fastText algorithm 62, pre-trained on the Common Crawl and Wikipedia corpora, for the analyses across Studies 1a-c. We opted for this model due to its leading performance, widespread acceptance, and multilingual …


title: MapPool-Bubbling up an extremely large corpus of maps for AI

authors: R Schnürer - 2024 ICA Workshop on AI, Geovisualization, and …, 2024

snippet: MapPool is a dataset of 75 million potential maps and textual captions. It has been derived from CommonPool, a dataset consisting of 12 billion text-image pairs from the Internet. The images have been encoded by a vision transformer and classified …


title: Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking

authors: Y Fung, R Zhao, J Doo, C Sun, H Ji - arXiv preprint arXiv:2402.09369, 2024

snippet: Pretrained large language models have revolutionized many applications but still face challenges related to cultural bias and a lack of cultural commonsense knowledge crucial for guiding cross-culture communication and interactions …


title: MASTERARBEIT/MASTER'S THESIS

snippet: Today, a huge amount of unstructured data from various sources is available. Natural language processing (NLP) techniques using such data are applied to several tasks such as named entity recognition (NER). This study is designed to …


title: MASTERARBEIT| MASTER'S THESIS

snippet: The exponential growth of the social media user base has led to an alarming rise in the use of offensive language and profanity, necessitating effective detection mechanisms. This thesis explores the challenges and advancements in automatic …


title: Mastering Transformers: The Journey from BERT to Large Language Models and Stable Diffusion

snippet: Explore transformer-based language models from BERT to GPT, delving into NLP and computer vision tasks, while tackling challenges effectively Key Features Understand the complexity of deep learning architecture and transformers …


title: Materials science in the era of large language models: a perspective

authors: G Lei, R Docherty, SJ Cooper - arXiv preprint arXiv:2403.06949, 2024

snippet: Large Language Models (LLMs) have garnered considerable interest due to their impressive natural language capabilities, which in conjunction with various emergent properties make them versatile tools in workflows ranging from complex …


title: Matthew Coscia and

snippet: … URL: https://github.com/commoncrawl/ia-web-commons. [9] Image Details - Container Registry - Google Cloud Platform. 2022. …


title: MDSAA

snippet: In the field of digital marketing, efficient content gap analysis is crucial for developing effective SEO strategies. Traditional approaches to this task are time-consuming, representing a challenge for Organic Performance teams, who must process large …


title: Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse

authors: M Song, SH Sim, R Bhardwaj, HL Chieu, N Majumder… - arXiv preprint arXiv …, 2024

snippet: … To do this, we query Wikipedia and Common Crawl to retrieve the 100 most relevant documents. We filter seed question for which the retriever fails to retrieve relevant documents. Furthermore, we identify 5 documents that are equally effective …


title: Measuring and Improving the Energy Efficiency of Large Language Models Inference

authors: MF Argerich, M Patiño-Martínez - IEEE Access, 2024

snippet: … 4) RedPajama [42]: another family of models based on Pythia with 7B and 3B parameters, fine-tuned on a dataset with the same name as the model, with over 100B text documents coming from 84 CommonCrawl snapshots. Similar to Dolly …


title: Measuring complex psychological and sociological constructs in large-scale text

authors: A Herderich, J Lasser, S Aroyehun, D Garcia…

snippet: In recent years, there has been an increasing exchange between social and computational sciences. Methods from natural language processing enable social scientists to systematically process large amounts of text. Rich psycho-sociological …


title: Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domain

authors: I García-Ferrero, R Agerri, AA Salazar, E Cabrio… - arXiv preprint arXiv …, 2024

snippet: … mT5 was trained using mC4, a 1 Trillion token Common Crawl-based dataset covering 101 languages. The pre-training is based on a masked language modeling “spancorruption” objective, where consecutive spans of input tokens are replaced …


title: Meeting the challenge: A benchmark corpus for automated Urdu meeting summarization

authors: B Sadia, F Adeeba, S Shams, K Javed - Information Processing & Management, 2024

snippet: Meeting summarization has become crucial as the world is gradually shifting towards remote work. Nowadays, automation of meeting summary generation is really needed in order to minimize the time and effort. The surge in online meetings …


title: MELTing point: Mobile Evaluation of Language Transformers

authors: S Laskaridis, K Kateveas, L Minto, H Haddadi - arXiv preprint arXiv:2403.12844, 2024

snippet: Transformers have revolutionized the machine learning landscape, gradually making their way into everyday tasks and equipping our computers with ``sparks of intelligence''. However, their runtime requirements have prevented them from being …


title: Memorization and Privacy Risks in Domain-Specific Large Language Models

authors: X Yang, Z Wen, W Qu, Z Chen, Z Xiang, B Chen, H Yao - ICLR 2024 Workshop on Reliable …

snippet: … Common Crawl3 is a nonprofit organization that provides a large and open web crawl data repository for public use. It collects web pages from the internet every month and stores them on Amazon Web Services. Common Crawl’s … C4 cleanses …


title: Mentions of prejudice in news media–an international comparison

authors: D Rozado - Journal of Computational Social Science, 2024

snippet: Previous research has identified a post-2010 sharp increase of terms used to denounce prejudice (ie racism, sexism, homophobia, Islamophobia, anti-Semitism, etc.) in US and UK news media content. Here, we extend previous analysis to an …


title: MeSH2Matrix: Combining MeSH keywords and machine learning for biomedical relation classification based on PubMed

authors: H Turki, BFP Dossou, CC Emezue, A Toluwase…

snippet: Biomedical relation classification has been significantly improved by the application of advanced machine learning techniques on the raw texts of scholarly publications. Despite this improvement, the reliance on large chunks of raw text makes these …


title: Methods Futures Report–Sea, Sky, and Land: Engaging in Solidarity in Endangered Ecologies–4S Conference Nov 2023

snippet: This is a report of the 4S 2023 conference. The 4S (Society for the Social Studies of Science) conference 2023 was a large, disciplinarily diverse gathering with over 400 sessions spread over four days. Navigating a particular route through the conference …


title: mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval

authors: X Zhang, Y Zhang, D Long, W Xie, Z Dai, J Tang, H Lin… - arXiv preprint arXiv …, 2024

snippet: We present systematic efforts in building long-context multilingual text representation model (TRM) and reranker from scratch for text retrieval. We first introduce a text encoder (base size) enhanced with RoPE and unpadding, pre-trained …


title: Milestones in Bengali Sentiment Analysis leveraging Transformer-models: Fundamentals, Challenges and Future Directions

authors: S Sengupta, S Ghosh, P Mitra, TI Tamiti - arXiv preprint arXiv:2401.07847, 2024

snippet: Sentiment Analysis (SA) refers to the task of associating a view polarity (usually, positive, negative, or neutral; or even fine-grained such as slightly angry, sad, etc.) to a given text, essentially breaking it down to a supervised (since we have the view …


title: MindLLM: Lightweight large language model pre-training, evaluation and domain application

authors: Y Yang, H Sun, J Li, R Liu, Y Li, Y Liu, Y Gao, H Huang - AI Open, 2024

snippet: Large Language Models (LLMs) have demonstrated remarkable performance across various natural language tasks, marking significant strides towards general artificial intelligence. While general artificial intelligence is leveraged by developing …


title: MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies

authors: S Hu, Y Tu, X Han, C He, G Cui, X Long, Z Zheng… - arXiv preprint arXiv …, 2024

snippet: The burgeoning interest in developing Large Language Models (LLMs) with up to trillion parameters has been met with concerns regarding resource efficiency and practical expense, particularly given the immense cost of experimentation. This …


title: MINNESOTA JOURNAL OF LAW, SCIENCE & TECHNOLOGY

authors: Z Niesel, J Villasenor, J Perl, DE Ho, A Vallebueno

snippet: In administrative law, there is perhaps no more misunderstood phrase than the" arbitrary and capricious" standard of judicial review. American administrative law's cornerstone is the judicial review of agency decision-making-a review which has …


title: MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens

authors: A Awadalla, L Xue, O Lo, M Shu, H Lee, EK Guha… - arXiv preprint arXiv …, 2024

snippet: … We follow OBELICS’s method for extracting interleaved multimodal documents from CommonCrawl WARC files by parsing each WARC entry’s DOM tree. While OBELICS only processed documents from February 2020 to February 2023 …


title: Misinformation Resilient Search Rankings with Webgraph-based Interventions

authors: P Carragher, EM Williams, KM Carley - arXiv preprint arXiv:2404.08869, 2024

snippet: … To this end, we investigate the distribution of changes in CommonCrawl PageRank as a result of our interventions for all domains in the webgraph, not just news domains. We then take the domains whose PR score changes significantly ( 50%) …


title: Mission Critical--Satellite Data is a Distinct Modality in Machine Learning

authors: E Rolf, K Klemmer, C Robinson, H Kerner - arXiv preprint arXiv:2402.01444, 2024

snippet: Satellite data has the potential to inspire a seismic shift for machine learning -- one in which we rethink existing practices designed for traditional data modalities. As machine learning for satellite data (SatML) gains traction for its real-world impact …


title: Mitigating Catastrophic Forgetting in Language Transfer via Model Merging

authors: A Alexandrov, V Raychev, MN Müller, C Zhang… - arXiv preprint arXiv …, 2024

snippet: As open-weight large language models (LLMs) achieve ever more impressive performances across a wide range of tasks in English, practitioners aim to adapt these models to different languages. However, such language adaptation is often …


title: Mitigating the Impact of Adversarial Fine-tuning on Large Language Models through Perturbation Attenuation

authors: Z Jejesi, A McBride, F Kovacic, L O'Connor - 2024

snippet: The increasing complexity and scale of machine learning models have rendered them vulnerable to adversarial manipulation, particularly during fine-tuning processes where subtle perturbations can lead to significant degradation in model …


title: Mitigating the Privacy Issues in Retrieval-Augmented Generation (RAG) via Pure Synthetic Data

authors: S Zeng, J Zhang, P He, J Ren, T Zheng, H Lu, H Xu… - arXiv preprint arXiv …, 2024

snippet: … 2021), we randomly select segments from the Common Crawl dataset to function as the information component. However, the randomness of the input may affect the command component. To mitigate this issue, we limit the maximum length of the …


title: MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures

authors: J Ni, F Xue, X Yue, Y Deng, M Shah, K Jain, G Neubig… - arXiv preprint arXiv …, 2024

snippet: … In the detection phase, we train open-source LLMs on self-collected data to detect queries in Common Crawl splits. During filtering, we utilize GPT-4 Turbo to exclude non-query sentences. In classification, we categorize the filtered queries by input …


title: ML-EAT: A Multilevel Embedding Association Test for Interpretable and Transparent Social Science

authors: R Wolfe, A Hiniker, B Howe - arXiv preprint arXiv:2408.01966, 2024

snippet: This research introduces the Multilevel Embedding Association Test (ML-EAT), a method designed for interpretable and transparent measurement of intrinsic bias in language technologies. The ML-EAT addresses issues of ambiguity and difficulty in …