title: Revisiting Adversarial Training at Scale

authors: Z Wang, X Li, H Zhu, C Xie - arXiv preprint arXiv:2401.04727, 2024

snippet: … DataComp-1B is a more recent dataset with about 1.3B samples filtered from a candidate pool of 12.8B image-text pairs from Common Crawl, which has been recorded to yield superior performance for contrastive training. To summarize, our …


title: Revisiting Joke Comprehension with Surprisal and Contextual Similarity: Implication from N400 and P600 Components

authors: H Xu, M Nakanishi, S Coulson - Proceedings of the Annual Meeting of the Cognitive …, 2024

snippet: … Specifically, we used the version with a 2.2 million word vocabulary and 300-dimensional vectors trained on 840 billion tokens from the Common Crawl corpus. The contextual vector of all the words preceding the final word in each sentence was computed by …


title: Revisiting Pre-training of Embedding Layers in Transformer-based Neural Machine Translation

authors: M Neishi, N Yoshinaga - Journal of Natural Language Processing, 2024

snippet: Recent trends in the pre-training and fine-tuning paradigm have made significant advances in several natural language processing tasks, including machine translation (MT), particularly for low-resource situations. However, it is reported that …


title: Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruning

authors: S Sarkar, L Lausen, V Cevher, S Zha, T Brox, G Karypis - arXiv preprint arXiv …, 2024

snippet: … It is sourced from the Common Crawl, a web archive that captures vast amounts of text data from the internet. We … the Common Crawl, covering 108 languages [46]. It is a subset of the Colossal Clean Crawled Corpus (C4), which itself is a cleaned …


title: Revolutionizing College English Reading with ChatGPT: Current Predicaments and Future Directions

authors: W Li, Y Tang, S Wang - Transactions on Social Science, Education and …, 2024

snippet: … The training data mainly includes website data in Common Crawl, two internet-based books corpora, and Englishlanguage Wikipedia [11]. ChatGPT trains a large amount of Internet information without distinguishing the true from the false and unifying …


title: Revolutionizing Urdu Sentiment Analysis: Harnessing the Power of XLM-R and GPT-2

authors: MR Ashraf, M Hussain, MA Jaffar, WY Ramay… - IEEE Access, 2024

snippet: … In our proposed approach XLM-R leverages much larger CommonCrawl web data across 100 languages and uses a larger 250k SentencePiece vocabulary to improve subword coverage of Urdu Text. Beyond masked language modeling, XLM-R …


title: Rho-1: Not All Tokens Are What You Need

authors: Z Lin, Z Gou, Y Gong, X Liu, Y Shen, R Xu, C Lin… - arXiv preprint arXiv …, 2024

snippet: Previous language model pre-training methods have uniformly applied a next-token prediction loss to all training tokens. Challenging this norm, we posit that "Not all tokens in a corpus are equally important for language model training". Our initial …


title: Risk Mitigation Strategies for the Open Foundation Model Value Chain

snippet: Policymakers worldwide are designing interventions to ensure AI development is safe and responsible. In the EU, the newly formed AI Office is drafting codes of practice for general purpose AI models, while in the US, state legislatures are …


title: Risks, Causes, and Mitigations of Widespread Deployments of Large Language Models (LLMs): A Survey

authors: MN Sakib, MA Islam, R Pathak, MM Arifin - arXiv preprint arXiv:2408.04643, 2024

snippet: Recent advancements in Large Language Models (LLMs), such as ChatGPT and LLaMA, have significantly transformed Natural Language Processing (NLP) with their outstanding abilities in text generation, summarization, and classification …


title: RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs

authors: J Dang, A Ahmadian, K Marchisio, J Kreutzer, A Üstün… - arXiv preprint arXiv …, 2024

snippet: Preference optimization techniques have become a standard final stage for training state-of-art large language models (LLMs). However, despite widespread adoption, the vast majority of work to-date has focused on first-class citizen languages like …


title: Roberta-LightGBM: A hybrid model of deep fake detection with pre-trained and binary classification

authors: V Rajkumar - INFOCOMP Journal of Computer Science, 2024

snippet: In May 2023, a fake image of an explosion near the Pentagon gained widespread social media traction. It dragged down US markets momentarily, perhaps marking the first time an artificial intelligence (AI)-generated image has affected the market …


title: RoBERTaNET: Enhanced RoBERTa Transformer Based Model for Cyberbullying Detection with GloVe Features

authors: F Alrowais, AA Jamjoom, H Karamti, M Umer, S Alsubai… - IEEE Access, 2024

snippet: … 3) FastText FastText is a word illustration library, that encompasses 300dimensional 2 million common crawl words, resulting in 600 billion word vectors [31]. Particularly, hand-crafted ngrams are used as features alongside solo words. Its straightforward …


title: Robust Neural Machine Translation for Abugidas by Glyph Perturbation

authors: H Kaing, C Ding, H Tanaka, M Utiyama - Proceedings of the 18th Conference of the …, 2024

snippet: Neural machine translation (NMT) systems are vulnerable when trained on limited data. This is a common scenario in low-resource tasks in the real world. To increase robustness, a solution is to intently add realistic noise in the training phase. Noise …


title: RoCode: A Dataset for Measuring Code Intelligence from Problem Definitions in Romanian

authors: A Cosma, B Iordache, P Rosso - arXiv preprint arXiv:2402.13222, 2024

snippet: … Even if code is present in the dataset (from Common Crawl), it is described in English and comments and documentation are in English. The same argument can be made for mathematics and other scientific disciplines. A large, highly curated and …


title: RoMath: A Mathematical Reasoning Benchmark in Romanian

authors: A Cosma, AM Bucur, E Radoi - arXiv preprint arXiv:2409.11074, 2024

snippet: … For pretraining, the general approach is to filter Common Crawl web pages and PDFs to obtain high quality math tokens. For instance, … 2024b) that would arise from dumps such as Common Crawl. Figure 1 showcases our approach. After …


title: ROME: Memorization Insights from Text, Probability and Hidden State in Large Language Models

authors: B Li, Q Zhao, L Wen - arXiv preprint arXiv:2403.00510, 2024

snippet: Probing the memorization of large language models holds significant importance. Previous works have established metrics for quantifying memorization, explored various influencing factors, such as data duplication, model size, and prompt length …


title: RU22Fact: Optimizing Evidence for Multilingual Explainable Fact-Checking on Russia-Ukraine Conflict

authors: Y Zeng, X Ding, Y Zhao, X Li, J Zhang, C Yao, T Liu… - arXiv preprint arXiv …, 2024

snippet: Fact-checking is the task of verifying the factuality of a given claim by examining the available evidence. High-quality evidence plays a vital role in enhancing fact-checking systems and facilitating the generation of explanations that are understandable to …


title: RuBia: A Russian Language Bias Detection Dataset

authors: V Grigoreva, A Ivanova, I Alimova, E Artemova - arXiv preprint arXiv:2403.17553, 2024

snippet: Warning: this work contains upsetting or disturbing content. Large language models (LLMs) tend to learn the social and cultural biases present in the raw pre-training data. To test if an LLM's behavior is fair, functional datasets are employed, and due to their …


title: Sailor: Open Language Models for South-East Asia

authors: L Dou, Q Liu, G Zeng, J Guo, J Zhou, W Lu, M Lin - arXiv preprint arXiv:2404.03608, 2024

snippet: We present Sailor, a family of open language models ranging from 0.5B to 7B parameters, tailored for South-East Asian (SEA) languages. These models are continually pre-trained from Qwen1.5, a great language model for multilingual use …


title: Sampling-based Pseudo-Likelihood for Membership Inference Attacks

authors: M Kaneko, Y Ma, Y Wata, N Okazaki - arXiv preprint arXiv:2404.11262, 2024

snippet: … 2023): LLaMA-2 employs English CommonCrawl, C4, Github, Wikipedia, Books, ArXiv, and StackExchange as pre-training datasets. … or filtered to contain predominantly English text, but a small amount of non-English data is still present …


title: Scaffold-BPE: Enhancing Byte Pair Encoding with Simple and Effective Scaffold Token Removal

authors: H Lian, Y Xiong, J Niu, S Mo, Z Su, Z Lin, P Liu, H Chen… - arXiv preprint arXiv …, 2024

snippet: … The Pile is composed of 22 diverse and high-quality datasets, the models trained on which significantly outperform both raw and filtered Common Crawl [10] models. The data distribution for our model training is identical to those described in the …


title: Scalable Distributed String Sorting

authors: F Kurpicz, P Mehnert, P Sanders, M Schimek - arXiv preprint arXiv:2404.16517, 2024

snippet: … While different sampling and group assignment techniques may improve character balance, the CommonCrawl data set contains many duplicated strings (eg, standard legal disclaimers) which are always assigned to a single PE and cannot be …


title: Scalable Pre-training of Large Autoregressive Image Models

authors: A El-Nouby, M Klein, S Zhai, MA Bautista, A Toshev… - arXiv preprint arXiv …, 2024

snippet: This paper introduces AIM, a collection of vision models pre-trained with an autoregressive objective. These models are inspired by their textual counterparts, ie, Large Language Models (LLMs), and exhibit similar scaling properties. Specifically …


title: Scalable Range Search over Temporal and Numerical Expressions

authors: V Ohr, D Gupta - The 10th ACM SIGIR/The 14th International …

snippet: … a subset of Common Crawl amounting to … Common Crawl, published as C4 [30]. This document collection consists of approximately 13.81 million documents. The fourth and final document collection we use is the entire C4-En corpus derived from …


title: Scaling A Simple Approach to Zero-Shot Speech Recognition

authors: J Zhao, V Pratap, M Auli - arXiv preprint arXiv:2407.17852, 2024

snippet: … Figure 2 Accuracy on FLEURS dev languages when increasing the amount of CommonCrawl text data to build lexicons and unigram LMs compared to using in-domain text data (FLEURS-topline; 3k utterances). … Data is drawn from CommonCrawl (CC; …


title: Scaling Laws for Data Filtering--Data Curation cannot be Compute Agnostic

authors: S Goyal, P Maini, ZC Lipton, A Raghunathan, JZ Kolter - arXiv preprint arXiv …, 2024

snippet: … As we train for longer, the accuracy gains using the LAION-filtering subset that filters the common crawl to 10% of its initial size plateau. Surprisingly, even no-filtering of the common crawl is better than the popular LAION-filtering after seeing more …


title: Scaling neural machine translation to 200 languages

snippet: … To overcome this problem, we created training datasets through global bitext mining in publicly available web content (drawn from repositories such as CommonCrawl). The underlying idea of our bitext mining approach is first to learn a …


title: Scaling Retrieval-Based Language Models with a Trillion-Token Datastore

authors: R Shao, J He, A Asai, W Shi, T Dettmers, S Min… - arXiv preprint arXiv …, 2024

snippet: Scaling laws with respect to the amount of training data and the number of parameters allow us to predict the cost-benefit trade-offs of pretraining language models (LMs) in different configurations. In this paper, we consider another …


title: ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Laws

authors: R Li, Y Wei, M Zhang, N Yu, H Hu, H Peng - arXiv preprint arXiv:2408.08310, 2024

snippet: … We begin with five CommonCrawl … CommonCrawl data with no more than 25B tokens. Each dataset was used to train meta-models for 1 epoch. The results demonstrate that all experiments outperform the baseline of random selection. Training …


title: sCambiaMenti at ACTI: Ensemble model with majority voting for Automatic Conspiracy Detection

authors: S Bianco, D Salusso - EVALITA Proceedings of the Eighth Evaluation …, 2024

snippet: In this paper we describe the methodology that we have implemented to solve the subtask A of the Automatic Conspiracy Detection (ACTI) task (EVALITA 2023). We have developed different classifiers and then used a majority voting approach to …


title: ScenicNL: Generating Probabilistic Scenario Programs from Natural Language

authors: K Elmaaroufi, D Shanker, A Cismaru… - First Conference on Language …

snippet: … Most of these model’s are fine-tuned variants of a foundational model that was trained on a wider body of text such as Common Crawl. For instance Codex is ”a GPT language model fine-tuned on publicly available code from GitHub” and it’s …


title: SchemaPile: A Large Collection of Relational Database Schemas

authors: T Döhmen, R Geacu, M Hulsebos, S Schelter - … of the ACM on Management of Data, 2024

snippet: … such as the aforementioned code dataset The Stack [25] or web crawls such as CommonCrawl [12] as alternative data sources. Furthermore, enterprises could adapt our data … Common Crawl – a free, open repository of web crawl data that …


title: Scoping: Towards Streamlined Entity Collections for Multi-Sourced Entity Resolution with Self-Supervised Agents

snippet: Linking multiple entities to a real-world object is a time-consuming and error-prone task. Entity Resolution (ER) includes techniques for vectorizing entities (signature), grouping similar entities into partitions (blocking), and matching entity pairs based …


title: Screening Articles for Systematic Reviews with ChatGPT

snippet: … ChatGPT is a Chatbot Generative Pre-trained Transformer that employs an auto-regressive language model pre-trained on large datasets with bil250 lions of tokens from CommonCrawl, Wikipedia, and other publicly available text sources. ChatGPT relies …


title: SD-WEAT: Towards Robustly Measuring Bias in Input Embeddings

authors: M Gray, L Wu - International Conference on Human-Computer …, 2024

snippet: … 300d” from the Stanford NLP Group [9], which was trained on 840 billion tokens from Common Crawl data and generates 300-dimension vectors. Moreover, other input embedding methods, including those of BERT [10, 11], SciBERT [12, 13], and …


title: SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages

authors: H Lovenia, R Mahendra, SM Akbar, LJV Miranda… - arXiv preprint arXiv …, 2024

snippet: Southeast Asia (SEA) is a region rich in linguistic diversity and cultural variety, with over 1,300 indigenous languages and a population of 671 million people. However, prevailing AI models suffer from a significant lack of representation of texts, images …


title: SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models for Southeast Asian Languages

authors: W Zhang, HP Chan, Y Zhao, M Aljunied, J Wang, C Liu… - arXiv preprint arXiv …, 2024

snippet: Large Language Models (LLMs) have shown remarkable abilities across various tasks, yet their development has predominantly centered on high-resource languages like English and Chinese, leaving low-resource languages underserved …


title: Second Language (Arabic) Acquisition of LLMs via Progressive Vocabulary Expansion

authors: J Zhu, H Hunag, Z Lin, J Liang, Z Tang, K Almubarak…

snippet: This paper addresses the critical need for democratizing large language models (LLM) in the Arab world, a region that has seen slower progress in developing models comparable to state-of-the-art offerings like GPT-4 or ChatGPT 3.5, due to a …


title: Secret Collusion Among Generative AI Agents

authors: SR Motwani, M Baranchuk, M Strohmeier, V Bolina… - arXiv preprint arXiv …, 2024

snippet: Recent capability increases in large language models (LLMs) open up applications in which teams of communicating generative AI agents solve joint tasks. This poses privacy and security challenges concerning the unauthorised sharing of information …


title: SECTION ALIGNMENT TO DETECT DOCUMENT LINKS EFFICIENTLY

snippet: When searching for information, users often use search engines like Google1 or Bing2. These search engines have access to a wealth of publicly available information. To make finding the information even simpler and for processing and …


title: Secured Weight Release for Large Language Models via Taylor Expansion

authors: G Wang, YN Chuang, R Tang, S Zhong, J Yuan, H Jin…

snippet: Ensuring the security of released large language models (LLMs) poses a significant dilemma, as existing mechanisms either compromise ownership rights or raise data privacy concerns. To address this dilemma, we introduce TaylorMLP to protect the …


title: SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents

authors: K Cheng, Q Sun, Y Chu, F Xu, Y Li, J Zhang, Z Wu - arXiv preprint arXiv:2401.10935, 2024

snippet: … We collected approximately 300k web pages from the latest Common Crawl repository 2 to serve as our training data for web UI. For each webpage, we collect two types of elements from the HTML code: (1) elements that display visible text …


title: Seeds of Stereotypes: A Large-Scale Textual Analysis of Race and Gender Associations with Diseases in Online Sources

authors: L Hyldig Hansen, N Andersen, J Gallifant, LG McCoy… - arXiv e-prints, 2024

authors: LH Hansen, N Andersen, J Gallifant, LG McCoy… - arXiv preprint arXiv …, 2024

snippet: … Methods We conducted a large-scale textual analysis using a dataset comprising diverse web sources, including Arxiv, Wikipedia, and Common Crawl. The study analyzed the context in which various diseases are discussed alongside markers of …


title: Selecting Large Language Model to Fine-tune via Rectified Scaling Law

authors: H Lin, B Huang, H Ye, Q Chen, Z Wang, S Li, J Ma… - arXiv preprint arXiv …, 2024

snippet: The ever-growing ecosystem of LLMs has posed a challenge in selecting the most appropriate pre-trained model to fine-tune amidst a sea of options. Given constrained resources, fine-tuning all models and making selections afterward is …


title: Selective Metric Differential Privacy for Language Models

snippet: … We used a pre-trained GloVe model trained on Common Crawl data with 840B tokens with a 2.2M vocabulary size for the privatization mechanism. We chose to utilize this model because it has a large vocabulary and is case-sensitive. Since GPT-2 …


title: Self-Augmented In-Context Learning for Unsupervised Word Translation

authors: Y Li, A Korhonen, I Vulić - arXiv preprint arXiv:2402.10024, 2024

snippet: … 2017): the version pretrained on Wikipedia7 is used for XLING and the version pretrained with Wikipedia plus Common Crawl8 is used for PanLex-BLI, as recommended by XLING and PanLex-BLI, respectively. The same WEs are used for …


title: SEMANT-Feature Group Selection Utilizing FastText-Based Semantic Word Grouping, Scoring, and Modeling Approach for Text Classification

authors: D Voskergian, B Bakir-Gungor, M Yousef - International Conference on Database and …, 2024

snippet: Text classification presents a challenge due to its high-dimensional feature space. As such, devising an effective feature selection scheme is essential. In this study, we present SEMANT, a novel hybrid filter-wrapper feature selection method that utilizes …


title: SEMANTIC MATCHING FOR DATA CATALOGS

snippet: Modern organizations have a lot of data, and more is coming. This data is often spread across the entire organization, appearing in various forms, such as CSV files, databases, PPTX slides, emails, etc. To organize and manage the various forms of …


title: Semantic-Aware Methods for the Analysis of Bias and Underrepresentation in Language Resources

snippet: … Within this research I did not analyze Common Crawl5, even if it is mentioned in some LMs papers. This is because it is not a proper … CC-News is a snapshot of news texts derived from the CommonCrawl initiative9) that has been exploited for …


title: Semantic-guided spatio-temporal attention for few-shot action recognition

authors: J Wang, B Liu - Applied Intelligence, 2024

snippet: … We utilize GloVe [49] to extract text embeddings for action labels, which is pre-trained on Common Crawl dataset of 840B tokens. For labels with multiple words, we average the embeddings of all their words. Finally, one action label is represented …


title: Semantically Enriched Cross-Lingual Sentence Embeddings for Crisis-related Social Media Texts

authors: R Lamsal, MR Read, S Karunasekera - arXiv preprint arXiv:2403.16614, 2024

snippet: Tasks such as semantic search and clustering on crisis-related social media texts enhance our comprehension of crisis discourse, aiding decision-making and targeted interventions. Pre-trained language models have advanced performance in …


title: Semiotics of Machinic Co-Enunciation

snippet: 1 In this paper, we propose a semiotic study on generative artificial intelligences, considering the work of Midjourney and DALL· E. 1 The aim is to frame their functioning from a semiotic point of view and to describe the operations that can be …


title: SeNSe: embedding alignment via semantic anchors selection

authors: L Malandri, F Mercorio, M Mezzanzanica, F Pallucchini - International Journal of Data …, 2024

snippet: Word embeddings have proven extremely useful across many NLP applications in recent years. Several key linguistic tasks, such as machine translation and transfer learning, require comparing distributed representations of words belonging to …


title: SENTA: Sentence Simplification System for Slovene

authors: A Žagar, M Klemen, M Robnik-Šikonja, I Kosem - Proceedings of the 2024 Joint …, 2024

snippet: … It uses sentence-level paraphrase data obtained by mining paraphrases from Common Crawl in combination with controllable generation mechanisms to adjust the length and lexical complexity of a simplified candidate. …


title: Sentence Embeddings for Massively Multilingual Speech and Text Processing

snippet: … CommonCrawl raw text data corresponds to partial snapshots of the internet totalling terabytes of text data extracted from web pages in many languages. The authors used the open-source FAISS library (Johnson et al. 2019) in order to …


title: Sentence Level Analysis Model for Phishing Detection Using KNN

snippet: Phishing emails have experienced a rapid surge in cyber threats globally, especially following the emergence of the COVID-19 pandemic. This form of attack has led to substantial financial losses for numerous organizations. Although various models …


title: Sentiment Analysis and Rating Prediction for App Reviews Using Transformer-based Models

snippet: In this study, we present the sentiment analysis of Spotify app reviews, the implementation of Natural Language Processing (NLP) methods, and the use of transformer-based models including BERT, DistilBERT, RoBERTa, and XLM-RoBERTa …


title: Sentiment Analysis of Solar Energy in US Cities: A 10-Year Analysis Using Transformer-Based Deep Learning

snippet: This study examines US public sentiment toward solar energy from 2013 to 2022, analyzing 8 million social media posts using RoBERTa, a transformer-based deep learning algorithm. Sentiment has generally been positive but has declined since …


title: Sentiment analysis on the issue of COVID-19 vaccination using LSTM and fasttext

authors: WN Dewani, Y Azhar, N Hayatin - AIP Conference Proceedings, 2024

snippet: … The fasttext model is the result of a word vector previously trained from Common Crawl and Indonesian language Wikipedia data from Facebook. This model is trained using the Continuous Bag of Words (CBOW) method with position weight [13] …


title: Sequence-to-Sequence Architectures: Encoder-Decoders and Decoders

authors: PM Nugues - … for Natural Language Processing: Programming with …, 2024

snippet: The transformer consisted originally of two parts: the encoder and the decoder. This chapter describes this complete structure and, to make it concrete, illustrates it with a machine-translation program in PyTorch. For sake of feasibility, we replace the word …


title: SETEM: Self-ensemble training with Pre-trained Language Models for Entity Matching

authors: H Ding, C Dai, Y Wu, W Ma, H Zhou - Knowledge-Based Systems, 2024

snippet: … This benchmark is constructed by extracting schema.org data from the Common Crawl and using the following attributes: brand, title, currency and price. The WDC Products benchmark consists of overall nine training, nine validation and nine test …


title: Shapley Values-Powered Framework for Fair Reward Split in Content Produced by GenAI

authors: A Glinsky, A Sokolsky - arXiv preprint arXiv:2403.09700, 2024

snippet: … DATACOMP introduces a new candidate pool of 12.8 billion image-text pairs sourced from Common Crawl, which is broader than the collection in LAION-5B. This extensive pool offers a more diverse base for dataset creation. Unlike LAION-5B …


title: ShieldGemma: Generative AI Content Moderation Based on Gemma

authors: W Zeng, Y Liu, R Mullins, L Peran, J Fernandez… - arXiv preprint arXiv …, 2024

snippet: We present ShieldGemma, a comprehensive suite of LLM-based safety content moderation models built upon Gemma2. These models provide robust, state-of-the-art predictions of safety risks across key harm types (sexually explicit, dangerous …


title: Should we stay silent on violence? An ensemble approach to detect violent incidents in Spanish social media texts

authors: D Sharma, V Gupta, VK Singh, D Pinto - Natural Language Processing

snippet: … For training, more than two terabytes of filtered CommonCrawl data was used. The implementation of XLM-RoBERTa includes a preprocessing step, as described earlier. After preprocessing the data, the tweets are tokenized using the XLM-RoBERTa …


title: SIGN LANGUAGE RECOGNITION OF WORDS AND SENTENCE PREDICTION USING LSTM AND NLP

authors: R GAIKWAD, L ADMUTHE - Journal of Theoretical and Applied Information …, 2024

snippet: … The dataset used for the training of the T5 model is the C4 dataset called as Colossal Clean Crawled Corpus which is collected from Common Crawl, a publicly available web archive. It consists of 750 GB clean English text scraped from the …


title: Silencing the Risk, Not the Whistle: A Semi-automated Text Sanitization Tool for Mitigating the Risk of Whistleblower Re-Identification

authors: D Staufer, F Pallas, B Berendt - arXiv preprint arXiv:2405.01097, 2024

snippet: Whistleblowing is essential for ensuring transparency and accountability in both public and private sectors. However, (potential) whistleblowers often fear or face retaliation, even when reporting anonymously. The specific content of their …


title: SimLESS: A Secure Deduplication System over Similar Data in Cloud Media Sharing

authors: M Song, Z Hua, Y Zheng, T Xiang, X Jia - IEEE Transactions on Information Forensics …, 2024

snippet: With the growing popularity of cloud computing, sharing media data through the cloud has become a common practice. Due to high information redundancy, media data take up a significant amount of storage space. Moreover, similar media data …


title: SimLLM: Calculating Semantic Similarity in Code Summaries using a Large Language Model-Based Approach

authors: X Jin, Z Lin - Proceedings of the ACM on Software Engineering, 2024

snippet: Code summaries are pivotal in software engineering, serving to improve code readability, maintainability, and collaboration. While recent advancements in Large Language Models (LLMs) have opened new avenues for automatic code …


title: Simple and Scalable Strategies to Continually Pre-train Large Language Models

authors: A Ibrahim, B Thérien, K Gupta, ML Richter, Q Anthony… - arXiv preprint arXiv …, 2024

snippet: … We then generate a fixed token-length response for each of the models trained German Common Crawl. As a baseline, we also evaluate … of German-language outputs from the models trained on German Common Crawl when compared to the …


title: Simulating Policy Impacts: Developing a Generative Scenario Writing Method to Evaluate the Perceived Effects of Regulation

authors: J Barnett, K Kieslich, N Diakopoulos - arXiv preprint arXiv:2405.09679, 2024

snippet: … They utilize proprietary datasets as well as opensource web data such as the continuously growing Common Crawl, which contains more than 250 billion pages of over a decade’s worth of textual content. While these models primarily learn how …


title: Simulating Subjects: The Promise and Peril of AI Stand-ins for Social Agents and Interactions

snippet: Large Language Models (LLMs), through their exposure to massive collections of online text, learn to reproduce the perspectives and linguistic styles of diverse social and cultural groups. This capability suggests a powerful social scientific application–the …


title: Single Cells Are Biological Tokens: Towards Cell Language Models

snippet: The rapid advancement of single-cell technologies allows for simultaneous measurement of multiple molecular features within individual cells, providing unprecedented multimodal data through single-cell multi-omics and spatial omics …


title: Single Layer Single Gradient Unlearning

authors: Z Cai, Y Tan, MS Asif - arXiv preprint arXiv:2407.11867, 2024

snippet: … Large-scale web scrapers, such as Common Crawl, archive billions of webpages monthly, providing massive data sources to train such revolutionary AI models. While these large unsupervised datasets significantly enhance performance under …


title: Single Parent Family: A Spectrum of Family Members from a Single Pre-Trained Foundation Model

authors: H Hajimolahoseini, M Hassanpour, F Ataiefard, B Chen… - arXiv preprint arXiv …, 2024

snippet: This paper introduces a novel method of Progressive Low Rank Decomposition (PLRD) tailored for the compression of large language models. Our approach leverages a pre-trained model, which is then incrementally decompressed to smaller sizes using …


title: Sketch Grammar for Japanese

snippet: Sketch Grammar is a formalism used to extract frequent word collocations from text corpora. This thesis aims at analyzing the current sketch grammar for Japanese used in the Sketch Engine corpus manager and implementing a new version with …


title: Slight Corruption in Pre-training Data Makes Better Diffusion Models

authors: H Chen, Y Han, D Misra, X Li, K Hu, D Zou… - arXiv preprint arXiv …, 2024

snippet: … For example, Stable Diffusion [26] was pre-trained on LAION-2B [17], which contains billion-scale image-text pairs collected from Common Crawl [27]. Despite the heavy filtering mechanisms used in collecting pre-training datasets [17, 28], they …


title: Small Language Models and Large Language Models in Oppositional thinking analysis: Capabilities and Biases and Challenges

authors: Á Huertas-García, C Martí-González, J Muñoz… - Working Notes of CLEF, 2024

snippet: The proliferation of misinformation and conspiracy theories needs robust methods to differentiate legitimate critical discourse from harmful conspiratorial narratives. This study investigates discerning critical messages from conspiracy theories within …


title: Small Models, Big Insights: Leveraging Slim Proxy Models To Decide When and What to Retrieve for LLMs

authors: J Tan, Z Dou, Y Zhu, P Guo, K Fang, JR Wen - arXiv preprint arXiv:2402.12052, 2024

snippet: The integration of large language models (LLMs) and search engines represents a significant evolution in knowledge acquisition methodologies. However, determining the knowledge that an LLM already possesses and the knowledge that requires the …


title: Smaller Can Be Better: Efficient Data Selection for Pre-training Models

authors: G Fang, S Wang, M Wang, Y Yang, H Huang - Asia-Pacific Web (APWeb) and Web …, 2024

snippet: To reduce the resource consumption for pre-training models on large-scale corpora, existing research has primarily focused on refining network architectures and training methods. Nevertheless, the concomitant cost tends to be expensive. This …


title: Smaller Language Models are Better Zero-shot Machine-Generated Text Detectors

authors: N Mireshghallah, J Mattern, S Gao, R Shokri… - Proceedings of the 18th …, 2024

snippet: As large language models are becoming more embedded in different user-facing services, it is important to be able to distinguish between human-written and machine-generated text to verify the authenticity of news articles, product reviews, etc. Thus, in this paper …


title: Smart Automation Using LLM

authors: P Ethape, R Kane, G Gadekar, S Chimane - International Research Journal of …, 2023

snippet: … Such large-scale models can ingest massive amounts of data, often from the internet, but also from sources such as the Common Crawl, which comprises more than 50 billion web pages, and Wikipedia, which has approximately 57 million pages. …


title: Smart Bilingual Focused Crawling of Parallel Documents

authors: C García-Romero, M Esplà-Gomis, F Sánchez-Martínez - arXiv preprint arXiv …, 2024

snippet: Crawling parallel texts $\unicode{x2014}$texts that are mutual translations$\unicode{x2014}$ from the Internet is usually done following a brute-force approach: documents are massively downloaded in an unguided process, and only a fraction of them end up …


title: Smoky Quartz: A Transparent Bilingual Large Language Model

authors: Q Huang, C Zhang, M Tao, J Lin, S Huang, Y Feng

snippet: Recent advancements in large-scale models have been remarkable; however, a significant challenge persists in their lack of transparency, particularly concerning the training data utilized. This obscurity raises concerns about potential data leaks …


title: SMW Cloud: A Corpus of Domain-Specific Knowledge Graphs from Semantic MediaWikis

snippet: Semantic wikis have become an increasingly popular means of collaboratively managing Knowledge Graphs. They are powered by platforms such as Semantic MediaWiki and Wikibase, both of which enable MediaWiki to store and publish …


title: Social Evolution of Published Text and The Emergence of Artificial Intelligence Through Large Language Models and The Problem of Toxicity and Bias

authors: A Khan, P Saravanan, SK Venkatesan - arXiv preprint arXiv:2402.07166, 2024

snippet: We provide a birds eye view of the rapid developments in AI and Deep Learning that has led to the path-breaking emergence of AI in Large Language Models. The aim of this study is to place all these developments in a pragmatic broader historical social …


title: Social Media Sentiment Analysis on Multilingual Dataset

snippet: This thesis presents using unlabeled data and soft targets from a LLM to train a pre-trained model for sentiment analysis. Inspired by the study “What do LLMs Know about Financial Markets? A Case Study on Reddit Market Sentiment Analysis” by Deng et …


title: SocialQuotes: Learning Contextual Roles of Social Media Quotes on the Web

authors: J Palowitch, H Alvari, M Kazemi, T Amin, F Radlinski - arXiv preprint arXiv …, 2024

snippet: … We build and release SocialQuotes1, a data set computed from the Common Crawl with over 32M social media quotes, accompanied by role labels for 8.3k quotes obtained from human annotators. 3. We validate our main hypothesis with …


title: Sociolinguistically Informed Interpretability: A Case Study on Hinglish Emotion Classification

authors: K Tatariya, H Lent, J Bjerva, M de Lhoneux - arXiv preprint arXiv:2402.03137, 2024

snippet: Emotion classification is a challenging task in NLP due to the inherent idiosyncratic and subjective nature of linguistic expression, especially with code-mixed data. Pre-trained language models (PLMs) have achieved high performance for many tasks and …


title: SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training

authors: N He, W Xiong, H Liu, Y Liao, L Ding, K Zhang, G Tang… - arXiv preprint arXiv …, 2024

snippet: … In summary, our experiments on the RedPajama CommonCrawl dataset substantiate that the soft deduplication method is capable of reducing perplexity and enhancing the accuracy of downstream tasks more efficiently compared to the …


title: SoK: SSO-MONITOR—The Current State and Future Research Directions in Single Sign-On Security Measurements

authors: L Jannett, M Westers, T Wich, C Mainka, A Mayer…

snippet: Single Sign-On (SSO) with OAuth 2.0 and OpenID Connect 1.0 is essential for user authentication and authorization on the Internet. Billions of users rely on SSO services provided by Google, Facebook, and Apple. For large-scale measurements …


title: Sonnet or Not, Bot? Poetry Evaluation for Large Models and Datasets

authors: M Walsh, A Preus, M Antoniak - arXiv preprint arXiv:2406.18906, 2024

snippet: … About 30% of our poems are found in the Common Crawl data included in Dolma, with the C4 dataset close behind. Wikipedia and Semantic Scholar contain the fewest detected poems. Overall, if at least one line from a poem is detected, it is …


title: Sourceformer: Tool Integration into Transformer Large Language Models Through Source Code

snippet: LLMs have been shown to use tools well. By allowing specific tools to increase the capabilities that LLMs struggle with, these models can become much more useful. Previous works use handcrafted examples of simple tool use during self-training …


title: Spanning the Spectrum of Hatred Detection: A Persian Multi-Label Hate Speech Dataset with Annotator Rationales

authors: Z Delbari, NS Moosavi, MT Pilehvar - Proceedings of the AAAI Conference on …, 2024

snippet: With the alarming rise of hate speech in online communities, the demand for effective NLP models to identify instances of offensive language has reached a critical point. However, the development of such models heavily relies on the …


title: Spatial Language Understanding: Deep Learning, Reasoning, and Evaluation

snippet: … Pretrained language models (PLM) are large neural network models that have been trained on vast amounts of text data such as the entire text of Wikipedia or the Common Crawl web corpus in a un/supervised way. These models use transformers …


title: Speaker independent recognition of low-resourced multilingual Arabic spoken words through hybrid fusion

authors: S Mehra, V Ranga, R Agarwal, S Susan - Multimedia Tools and Applications, 2024

snippet: This article introduces a supervised strategy designed to enhance spoken word recognition within the constraints of a resource-limited multilingual dataset, specifically focusing on the Arabic language. Notably, existing methodologies often …


title: Specialising and Analysing Instruction-Tuned and Byte-Level Language Models for Organic Reaction Prediction

authors: J Pang, I Vulić - arXiv preprint arXiv:2405.10625, 2024

snippet: Transformer-based encoder-decoder models have demonstrated impressive results in chemical reaction prediction tasks. However, these models typically rely on pretraining using tens of millions of unlabelled molecules, which can be time-consuming …


title: Specialized Language Models with Cheap Inference from Limited Domain Data

authors: D Grangier, A Katharopoulos, P Ablin, A Hannun - arXiv preprint arXiv:2402.01093, 2024

snippet: Large language models have emerged as a versatile tool but are challenging to apply to tasks lacking large inference budgets and large in-domain training sets. This work formalizes these constraints and distinguishes four important variables …


title: Spectra: A Comprehensive Study of Ternary, Quantized, and FP16 Language Models

authors: A Kaushal, T Pandey, T Vaidhya, A Bhagat, I Rish - arXiv preprint arXiv:2407.12327, 2024

snippet: Post-training quantization is the leading method for addressing memory-related bottlenecks in LLM inference, but unfortunately, it suffers from significant performance degradation below 4-bit precision. An alternative approach involves …


title: SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models

authors: D Liu, R Zhang, L Qiu, S Huang, W Lin, S Zhao, S Geng… - Forty-first International Conference …

authors: P Gao, R Zhang, C Liu, L Qiu, S Huang, W Lin, S Zhao… - arXiv preprint arXiv …, 2024

snippet: … Specifically, we first collect large-scale PDF datasets from Common Crawl 1 and arXiv websites. Then, we utilize PyMuPDF 2 to get the rendering results of each page in the PDF file and also save all the text annotations along with their bounding …


title: Spike No More: Stabilizing the Pre-training of Large Language Models

authors: S Takase, S Kiyono, S Kobayashi, J Suzuki - arXiv preprint arXiv:2312.16903, 2023

snippet: … 2020) that consists of clean English texts extracted from Common Crawl7 as our LLM pre-training corpus. We also used the separated part of C4 as our validation data. We used GPT-2 vocabulary (Radford et al.… https://commoncrawl.org/ …


title: SPML: A DSL for Defending Language Models Against Prompt Attacks

authors: RK Sharma, V Gupta, D Grossman - arXiv preprint arXiv:2402.11755, 2024

snippet: … The pre-training data included trillions of tokens sourced from publicly available datasets, such as Common Crawl, Wikipedia, and public domain books from Project Gutenberg. We use the versions with 7 billion and 13 billion parameters, available …


title: Sports, crisis, and social media: a Twitter-based exploration of the Tokyo Olympics in the COVID-19 era

authors: V Mehra, P Singh, S Bharany, RS Sawhney - Social Network Analysis and Mining, 2024

snippet: The global landscape underwent a transformative shift due to the unexpected arrival of the COVID-19 pandemic, disrupting various aspects of life, including the staging of sports events. Notably, the Tokyo Olympics, a renowned international sports …


title: Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text

authors: A Hans, A Schwarzschild, V Cherepanova, H Kazemi… - arXiv preprint arXiv …, 2024

snippet: … When evaluating Binoculars on samples from languages that are not well represented in Common Crawl data (standard LLM pretraining data), we find that false-positives rates remain low, which is highly desirable from a harm reduction …


title: SRBerta—A Transformer Language Model for Serbian Cyrillic Legal Texts

authors: M Bogdanović, J Kocić, L Stoimenov - Information, 2024

snippet: … CommonCrawl [5] is a collection of texts whose size is measured in petabytes. This collection was … , CommonCrawl contains texts of lower quality, so development teams often resorted to processing texts before training and extracting …


title: Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training

authors: W Du, T Luo, Z Qiu, Z Huang, Y Shen, R Cheng, Y Guo… - arXiv preprint arXiv …, 2024

snippet: LLMs are computationally expensive to pre-train due to their large scale. Model growth emerges as a promising approach by leveraging smaller models to accelerate the training of larger ones. However, the viability of these model growth …


title: Stage-1 Report: Everything everywhere all at once? Exploring the richness of visual experience beyond words

authors: L Qianchen, M Yamada, Y Aizawa, N Tsuchiya

snippet: In consciousness research, understanding the nature and mechanism of visual experience has been one of the most debated topics. For humans, vision is arguably the primary sensory modality to understand the environment (Cole & Balcetis, 2021) …


title: StarCoder 2 and The Stack v2: The Next Generation

authors: A Lozhkov, R Li, LB Allal, F Cassano, J Lamy-Poirier… - arXiv preprint arXiv …, 2024

snippet: The BigCode project, an open-scientific collaboration focused on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder2. In partnership with Software Heritage (SWH), we build The Stack v2 on …


title: State of Title IX: A Knowledge Base for Title IX Documentation

snippet: Knowledge bases play a vital role in the modern world, offering a systematic and structured approach to integrate various entities, concepts, rules, and relationships associated with real-world information. In this study, we aggregated the official Title …


title: Statistical dataset evaluation: A case study on named entity recognition

authors: C Wang, Q Dong, X Wang, Z Sui - Natural Language Processing

snippet: Datasets serve as crucial training resources and model performance trackers. However, existing datasets have exposed a plethora of problems, inducing biased models and unreliable evaluation results. In this paper, we propose a model-agnostic …


title: STFL: Utilizing a Semi-Supervised, Transfer-Learning, Federated-Learning Approach to Detect Phishing URL Attacks

authors: I Sakazi, E Grolman, Y Elovici, A Shabtai - 2024 International Joint Conference on …, 2024

snippet: Phishing attacks are continually changing, so machine learning detection models must be continuously updated by collecting new URL data from users without compromising their privacy. Existing approaches for the detection of phishing URLs …


title: Strategies for Corpus Development for Low‐Resource Languages: Insights from Nepal

authors: BK Bal, B Prasain, RR Ghimire, P Acharya - Automatic Speech Recognition and …, 2024

snippet: … 25 whereas the monolingual dataset were derived from Wikipedia and Common Crawl for languages under consideration. They also experimented with four variations of training settings: fully supervised, fully unsupervised, semisupervised …


title: Stratified Graph Indexing for E cient Search in Deep Descriptor Databases

snippet: Searching for unseen objects in extensive visual archives is challenging, demanding efficient indexing methods that can support meaningful similarity retrievals. This research paper presents the Stratified Graph (SG) approach for …


title: Strengthening the WiC: New Polysemy Dataset in Hindi and Lack of Cross Lingual Transfer

authors: H Dubossarsky, F Dairkee - Proceedings of the 2024 Joint International Conference …, 2024

snippet: … TB of filtered CommonCrawl data as opposed to mBERT which was trained on Wikipedia data, and reported to outperform mBERT in a variety of cross-lingual benchmarks like XNLI, MLQA and NER. HindiBERT is a monolingual model trained …


title: Structured Packing in LLM Training Improves Long Context Utilization

authors: K Staniszewski, S Tworkowski, S Jaszczur… - arXiv preprint arXiv …, 2023

snippet: Recent advances in long-context Large Language Models (LCLMs) have generated significant interest, especially in applications such as querying scientific research papers. However, their potential is often limited by inadequate context utilization. We …


title: Studies in conversational AI: multilingual capabilities, world knowledge, and evaluation strategies

snippet: This thesis studies the evolving landscape of conversational AI. The main research objective is to improve the conversational abilities of conversational agents, with a focus on integrating real-time knowledge and expanding multilingual capabil-ities …


title: Studying Peer Effects in Divergent Thinking: Theory and Method

snippet: In designing technology that supports user learning, an important first step is to understand how interactions among humans shape mutual learning. Much qualitative research in the realm of peer-assisted learning (PAL) has advanced the …


title: SUBLLM: A Novel Efficient Architecture with Token Sequence Subsampling for LLM

authors: Q Wang, Y Yuan, X Yang, R Zhang, K Zhao, W Liu… - arXiv preprint arXiv …, 2024

snippet: While Large Language Models (LLMs) have achieved remarkable success in various fields, the efficiency of training and inference remains a major challenge. To address this issue, we propose SUBLLM, short for Subsampling-Upsampling-Bypass …


title: Supervised Learning

authors: ER Vankov, K Gadhoumi - Statistical Methods in Epilepsy, 2024

snippet: … A corpus such as Common Crawl, which contains roughly 840 billion tokens of over two million unique words, provides such scale. There are other corpora used in these unsupervised processes, one such common candidate being a collection of all …


title: SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs

authors: J Kim, J Nam, S Mo, J Park, SW Lee, M Seo, JW Ha… - arXiv preprint arXiv …, 2024

snippet: Large language models (LLMs) have made significant advancements in various natural language processing tasks, including question answering (QA) tasks. While incorporating new information with the retrieval of relevant passages is a promising …


title: Suri: Multi-constraint Instruction Following for Long-form Text Generation

authors: CM Pham, S Sun, M Iyyer - arXiv preprint arXiv:2406.19371, 2024

snippet: Existing research on instruction following largely focuses on tasks with simple instructions and short responses. In this work, we explore multi-constraint instruction following for generating long-form text. We create Suri, a dataset with 20K human-written …


title: Survey of different Large Language Model Ar-chitectures: Trends, Benchmarks, and Challenges

authors: M SHAO, A BASIT, R KARRI, M SHAFIQUE - structure

snippet: Large Language Models (LLMs) represent a class of deep learning models adept at understanding natural language and generating coherent text in response to prompts or queries. These models significantly surpass conventional neural …


title: Synthetic continued pretraining

authors: Z Yang, N Band, S Li, E Candès, T Hashimoto - arXiv preprint arXiv:2409.07431, 2024

snippet: Pretraining on large-scale, unstructured internet text has enabled language models to acquire a significant amount of world knowledge. However, this knowledge acquisition is data-inefficient -- to learn a given fact, models must be trained on …


title: Synthetic Data Generation and Joint Learning for Robust Code-Mixed Translation

authors: S Soni, A Kunchukuttan, T Chakraborty, MS Akhtar - arXiv preprint arXiv …, 2024

snippet: The widespread online communication in a modern multilingual world has provided opportunities to blend more than one language (aka code-mixed language) in a single utterance. This has resulted a formidable challenge for the computational …


title: Systematic Evaluation of Linear Transformations for Cross-lingual Semantic Spaces

snippet: In the field of natural language understanding (NLU), a fundamental element is the representation of word meanings, a process that is integral to a wide range of applications. These applications span numerous domains including machine …


title: Systematic exploration and in-depth analysis of ChatGPT architectures progression

authors: D Banik, N Pati, A Sharma - Artificial Intelligence Review, 2024

snippet: The fast evolution of artificial intelligence frameworks has resulted in the creation of increasingly sophisticated large language models (LLM), ChatGPT being the most famous one. This study paper dives into this LLM with a case study of ChatGPT’s …


title: T5G2P: Text-to-Text Transfer Transformer based Grapheme-to-Phoneme Conversion

authors: M Řezáčková, D Tihelka, J Matoušek - IEEE/ACM Transactions on Audio, Speech …, 2024

snippet: … For the Czech language, we collected the CommonCrawl corpus at the end of August 2020. It contains 47.9GB of clean text, 20.5 million … Similarly, for both Russian and German we collected CommonCrawl corpus in the season from August …


title: Tübingen-CL at SemEval-2024 Task 1: Ensemble Learning for Semantic Relatedness Estimation

authors: L Zhang, Ç Çöltekin - Proceedings of the 18th International Workshop on …, 2024

snippet: The paper introduces our system for SemEval-2024 Task 1, which aims to predict the relatedness of sentence pairs. Operating under the hypothesis that semantic relatedness is a broader concept that extends beyond mere similarity of sentences …


title: Table Representation Learning

snippet: The increasing amount of data being collected, stored, and analyzed, induces a need for efficient, scalable, and robust methods to handle this data. Representation learning, ie, the practice of leveraging neural networks to obtain generic vector …


title: Tackling an Unbalanced Dataset for Classifying Indonesian E-Commerce Reviews Using Multi Word Embedding Model

authors: R Adi, BR Irnawan, J Li - 2023 Eighth International Conference on Informatics …, 2023

snippet: … Our research used a pre-trained FastText that was trained on Indonesian Common Crawl and Wikipedia provided by [24]. 3) GloVe: GloVe is an unsupervised algorithm that can produce a vector representation of words created by [22]. Besides …


title: Take the Bull by the Horns: Hard Sample-Reweighted Continual Training Improves LLM Generalization

authors: X Chen, Z Wang, D Sow, J Yang, T Chen, Y Liang… - arXiv preprint arXiv …, 2024

snippet: … subset of the Common Crawl’s web crawl corpus. More recently, LLaMA (Touvron et al.… datasets to encompass multiple domains such as Common Crawl, C4, and source codes from … contents are mostly unintelligible” in the Common Crawl. All …


title: Tamil Language Computing: the Present and the Future

authors: K Sarveswaran - arXiv preprint arXiv:2407.08618, 2024

snippet: This paper delves into the text processing aspects of Language Computing, which enables computers to understand, interpret, and generate human language. Focusing on tasks such as speech recognition, machine translation, sentiment …


title: Task Oriented In-Domain Data Augmentation

authors: X Liang, X Hu, S Zuo, Y Gong, Q Lou, Y Liu, SL Huang… - arXiv preprint arXiv …, 2024

snippet: Large Language Models (LLMs) have shown superior performance in various applications and fields. To achieve better performance on specialized domains such as law and advertisement, LLMs are often continue pre-trained on in-domain data …


title: Tasneef: A Fast and Effective Hybrid Representation Approach for Arabic Text Classification

authors: M Louail, CKM Hamdi-Cherif, A Hamdi-Cherif - IEEE Access, 2024

snippet: The Arabic language role in actual global affairs entails sophisticated natural language processing techniques, especially in text classification. This paper presents Tasneef as a novel hybrid approach to tackle computational challenges by …


title: TEA+: A Novel Temporal Graph Random Walk Engine With Hybrid Storage Architecture

authors: C Huan, Y Liu, H Zhang, S Song, S Pandey, S Chen… - ACM Transactions on Architecture …

snippet: Many real-world networks are characterized by being temporal and dynamic, wherein the temporal information signifies the changes in connections, such as the addition or removal of links between nodes. Employing random walks on these …


title: Team Aditya at EXIST 2024—Detecting Sexism In Multilingual Tweets Using Contrastive Learning Approach

authors: A Shah, A Gokhale - Working Notes of CLEF, 2024

snippet: Due to the growing impact of social media, the necessity for automated mechanisms that can identify sexism and other forms of disrespectful and hateful conduct is rising, aiming to create a more inclusive and respectful digital space. However, it poses …


title: Team OpenWebSearch at CLEF 2024: LongEval

authors: D Alexander, M Fröbe, G Hendriksen, F Schlatt… - 2024

snippet: We describe the OpenWebSearch group’s participation in the CLEF 2024 LongEval IR track. Our submitted runs explore how historical data from the past can be transferred into future retrieval systems. Therefore, we incorporate relevance …


title: TeClass: A Human-Annotated Relevance-based Headline Classification and Generation Dataset for Telugu

authors: G Kanumolu, L Madasu, N Surange, M Shrivastava - arXiv preprint arXiv:2404.11349, 2024

snippet: … 2019) is a multilingual version of the RoBERTa model, and it was pre-trained on a vast 2.5TB CommonCrawl dataset, which included text from 100 languages. For our experiments, we utilized the xlm-roberta-base variant, boasting 270 million parameters. …


title: Tele-FLM Technical Report

authors: X Li, Y Yao, X Jiang, X Fang, C Wang, X Liu, Z Wang… - arXiv preprint arXiv …, 2024

snippet: … CommonCrawl1 is often considered to be a repository containing diverse human experience … CommonCrawl are primarily concentrated in the English segment, with the Chinese content exhibiting relatively lower information density and quality …


title: Tele-LLMs: A Series of Specialized Large Language Models for Telecommunications

authors: A Maatouk, KC Ampudia, R Ying, L Tassiulas - arXiv preprint arXiv:2409.05314, 2024

snippet: … is the Common Crawl dataset, which comprises web archives from across the internet. To avoid issues with duplicates, non-English content, and potential profanity found in raw dumps, we utilized the refined web dataset [26]. This curated …


title: TelecomGPT: A Framework to Build Telecom-Specfic Large Language Models

authors: H Zou, Q Zhao, Y Tian, L Bariah, F Bader, T Lestable… - arXiv preprint arXiv …, 2024

snippet: Large Language Models (LLMs) have the potential to revolutionize the Sixth Generation (6G) communication networks. However, current mainstream LLMs generally lack the specialized knowledge in telecom domain. In this paper, for the …


title: Tell me a story: a framework for critically investigating AI language models

authors: L Munn, L Henrickson - Learning, Media and Technology, 2024

snippet: Large language models are rapidly being rolled out into high-stakes fields like healthcare, law, and education. However, understanding of their design considerations, operational logics, and implicit biases remains limited. How might …


title: Temporal Blind Spots in Large Language Models

authors: J Wallat, A Jatowt, A Anand - arXiv preprint arXiv:2401.12078, 2024

snippet: … A mixture of training datasets has been used: a filtered version of CommonCrawl, an expanded version of WebText [44], two not further specified internet-based book corpora, and the English Wikipedia. The most recent information in text-davinci-003’s …


Temporally-Varied News Headline Generation with GPT-3


title: TEXAS INTELLECTUAL PROPERTY LAW JOURNAL

authors: J Nacht, A Neill, J Thomas, E Lee, KE Noonan… - 2024

snippet: … 4t Common Crawl contains information from over 250 billion web pages since 2007 using web crawlers … in Common Crawl was filtered before being incorporated into GPT-3 to maintain a higher level of quality for training purposes.Websites …


title: Text Emotion Recognition Using Fast Text Word Embedding in Bi-Directional Gated Recurrent Unit

authors: C Akalya Devi, D Karthika Renuka, T Harisudhan…

snippet: Emotions are states of readiness in the mind that result from evaluations of one's own thinking or events. Although almost all of the important events in our lives are marked by emotions, the nature, causes, and effects of emotions are some of the …


title: Text Filtering Classifiers for Medium-Resource Languages

authors: J Daðason, H Loftsson - Proceedings of the 2024 Joint International Conference …, 2024

snippet: … Online texts are often obtained from large datasets, such as those published by Common Crawl (CC).The GPT-3 model was pre-trained on 499B tokens, 410B of which were obtained from CC (Brown et al.… The Icelandic Common Crawl Corpus (IC3) …


title: Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challenges

authors: J Becker, JP Wahle, B Gipp, T Ruas - arXiv preprint arXiv:2405.15604, 2024

snippet: Text generation has become more accessible than ever, and the increasing interest in these systems, especially those using large language models, has spurred an increasing number of related publications. We provide a systematic literature review …


title: TEXT SUMMARIZATION SYSTEM USING NATURAL LANGUAGE PROCESSING AND MACHINE LEARNING

snippet: Information overload, caused by the rapid growth of the Internet, has become a significant issue. The abundance of information available online necessitates the simplification of relevant content through summarization. Manual summarization of …


title: Text-to-text generative approach for enhanced complex word identification

authors: P Śliwiak, SAA Shah - Neurocomputing, 2024

snippet: This paper presents a novel approach for solving the Complex Word Identification (CWI) task using the text-to-text generative model. The CWI task involves identifying complex words in text, which is a challenging Natural Language Processing task. To …


title: TextGram: Towards a Better Domain-Adaptive Pretraining

authors: O Dhekane, S Hiwarkhedkar, R Joshi, A Ladkat - … , Perundurai, Erode, India, December 6–8 …

snippet: … The RealNews Dataset is a large corpus of news articles scraped from Common Crawl [16]. It consists of 5000 news domains that are indexed by Google news. IMDb Movie Reviews is a binary sentiment analysis dataset made up of 50000 reviews …


title: Textual Coverage of Eventive Entries in Lexical Semantic Resources

authors: E Fucikova, CF Alcaina, J Hajic, Z Urešová - Proceedings of the 2024 Joint …, 2024

snippet: This short paper focuses on the coverage of eventive entries (verbs, predicates, etc.) of some well-known lexical semantic resources when applied to random running texts taken from the internet. While coverage gaps are often reported for manually …


title: Textual emotion detection–A systematic literature review

snippet: Textual emotion detection is a critical area of study with significant applications in business, education, and healthcare. Despite substantial theoretical advancements over the years, there is a notable gap in the practical implementation of these …


title: The advantages of context specific language models: the case of the Erasmian Language Model

authors: J Gonçalves, N Jelicic, M Murgia, E Stamhuis - arXiv preprint arXiv:2408.06931, 2024

snippet: … The dominant trend in improving language models seems to be based on scaling up the number of parameters (eg GPT4 has approximately 1.7 trillion parameters) or the amount of training data fed into the model (eg the Common Crawl dataset …


title: The African Woman is Rhythmic and Soulful: Evaluation of Open-ended Generation for Implicit Biases

authors: S Lim - arXiv preprint arXiv:2407.01270, 2024

snippet: This study investigates the subtle and often concealed biases present in Large Language Models (LLMs), which, despite passing explicit bias tests, can still exhibit implicit biases akin to those observed in humans who profess egalitarian beliefs yet …


title: The AI Companion in Education: Analyzing the Pedagogical Potential of ChatGPT in Computer Science and Engineering

authors: Z He, T Nguyen, T Miari, M Aliasgari, S Rafatirad…

snippet: … ChatGPT3 employs a pre-trained large language model [29] trained over the Common Crawl dataset [30]. During its training process, it first preprocess data from the Common Crawl dataset, data quality is refined by eliminating irrelevant or noisy …


title: The AI Did My Job Well

snippet: Artificial intelligence (AI), an idea first proposed back in 1955 (McCarthy, Minsky, & Rochester) seemingly has burst into public consciousness nearly 70 years later. Like the chained prisoners emerging from Plato’s allegorical cave, many of us find …


title: The Aranea Corpora Family: Ten+ Years of Processing Web-Crawled Data

authors: V Benko - International Conference on Text, Speech, and …, 2024

snippet: Aranea is a project to create a family of web-crawled corpora for languages taught at Slovak Universities. Since 2013, more than two dozen languages have been added to the project: they are often represented with Gigaword+ size corpora and …


title: The BEA 2024 Shared Task on the Multilingual Lexical Simplification Pipeline

authors: M Shardlow, F Alva-Manchego, RT Batista-Navarro… - … on Innovative Use of NLP for …, 2024

snippet: We report the findings of the 2024 Multilingual Lexical Simplification Pipeline shared task. We released a new dataset comprising 5,927 instances of lexical complexity prediction and lexical simplification on common contexts across 10 languages, split …


title: The Case For Data Centre Hyperloops

authors: G López-Paradís, IM Hair, S Kannan, R Rabbat… - 2024 ACM/IEEE 51st …, 2024

authors: G López-Paradıs, IM Hair, S Kannan, R Rabbat…

snippet: Data movement is a hot-button topic today, with workloads like machine learning (ML) training, graph processing, and data analytics consuming datasets as large as 30PB. Such a dataset would take almost a week to transfer at 400gbps while consuming …


title: The Case for Developing a Foundation Model for Planning-like Tasks from Scratch

authors: B Srivastava, V Pallagani - arXiv preprint arXiv:2404.04540, 2024

snippet: Foundation Models (FMs) have revolutionized many areas of computing, including Automated Planning and Scheduling (APS). For example, a recent study found them useful for planning problems: plan generation, language translation, model …


title: The Case for Nurturing AI Literacy in Law Schools

authors: S Migliorini, JI Moreira - Asian Journal of Legal Education, 2024

snippet: The debate surrounding the permissibility of generative artificial intelligence (AI) tools in legal education has garnered widespread attention. However, this discourse has largely oscillated between the advantages and disadvantages of generative AI …


title: The correlation between nativelike selection and prototypicality: a multilingual onomasiological case study using semantic embedding

authors: H Zhang - arXiv preprint arXiv:2405.13529, 2024

snippet: In native speakers' lexical choices, a concept can be more readily expressed by one expression over another grammatical one, a phenomenon known as nativelike selection (NLS). In previous research, arbitrary chunks such as collocations have …


title: The Dark Side of Dataset Scaling: Evaluating Racial Classification in Multimodal Models

authors: A Birhane, S Dehdashtian, VU Prabhu, V Boddeti - arXiv preprint arXiv:2405.04623, 2024

snippet: … This is particularly important in the context of multimodal datasets whose main source is the World Wide Web, condensed and packaged as the Common Crawl dump, which is known to exhibit numerous drawbacks. In this paper, we evaluate the …


title: The Dark Side of Language Models: Exploring the Potential of LLMs in Multimedia Disinformation Generation and Dissemination

authors: D Barman, Z Guo, O Conlan - Machine Learning with Applications, 2024

snippet: Disinformation - the deliberate spread of false or misleading information poses a significant threat to our society by undermining trust, exacerbating polarization, and manipulating public opinion. With the rapid advancement of artificial intelligence and …


title: The diagnostic and triage accuracy of the GPT-3 artificial intelligence model: an observational study

authors: DM Levine, R Tuwani, B Kompa, A Varma… - The Lancet Digital Health, 2024

snippet: … GPT-3 is one of the largest AI models ever created, containing over 175 billion model parameters, and was trained on approximately 570 gigabytes of data from the common crawl dataset, which contains nearly all unstructured text available on the …


title: The effect of generative artificial intelligence on managerial behavior across functions

snippet: GAI is becoming readily available and easily accessible for those who are willing to take advantage of it. As a result, the world is witnessing a transformation regarding how managers perform their daily tasks. This study aims to understand the …


title: THE ENHANCEMENT OF OPEN DATA USABILITY BY DEFINING A CATEGORIZATION METHOD BASED ON THE OPEN DATA PORTAL METADATA

snippet: Due to numerous data transparency and open government initiatives, a large volume of data was published on open data portals. To make it more accessible and visible, these portals have introduced data filtering by category, tags, format …


title: The ethical situation of DALL-E 2

authors: E Hogea, J Rocafortf - arXiv preprint arXiv:2405.19176, 2024

snippet: A hot topic of Artificial Intelligence right now is image generation from prompts. DALL-E 2 is one of the biggest names in this domain, as it allows people to create images from simple text inputs, to even more complicated ones. The company that made this …


title: The Ethics and Challenges of Legal Personhood for AI

snippet: Hon. Katherine B. Forrest (Fmr.) abstract. AI’s increasing cognitive abilities will raise challenges for judges.“Legal personhood” is a flexible and political concept that has evolved throughout American history. In determining whether to expand that concept …


title: The Evolution of Multimodal Model Architectures

authors: SN Wadekar, A Chaurasia, A Chadha, E Culurciello - arXiv preprint arXiv:2405.17927, 2024

snippet: This work uniquely identifies and characterizes four prevalent multimodal model architectural patterns in the contemporary multimodal landscape. Systematically categorizing models by architecture type facilitates monitoring of developments in …


title: The Fill-Mask Association Test (FMAT): Measuring Propositions in Natural Language

authors: HWS Bao - Journal of Personality and Social Psychology, 2024

authors: L Lin, B Wang, X Wang, ZX Wang, A Wiśniowski…

snippet: Recent advances in large language models are enabling the computational intelligent analysis of psychology in natural language. Here, the Fill-Mask Association Test (FMAT) is introduced as a novel and integrative method leveraging …


title: The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

authors: G Penedo, H Kydlíček, A Lozhkov, M Mitchell, C Raffel… - arXiv preprint arXiv …, 2024

snippet: … In this work, we introduce FineWeb, a 15-trillion token dataset derived from 96 Common Crawl snapshots that produces better-performing LLMs than other open pretraining datasets. To advance the understanding of how best to curate high-quality …


title: The Future of Chatbots: Survey on Generative Pretrained Models, Ernie, and Gemini

authors: A Raut, J Katti - 2024 International Conference on Innovations and …, 2024

snippet: The evolution of chatbots from rule-based, retrieval-based, contextual systems, to hybrid and generative AIpowered models. It aimsto solve complex natural language processing and task-specific challenges. The finding explores innovative self-attention …


title: The Future of Data Science in Materials Science

authors: P Chundi, V Bommanapally, V Gadhamshetty - Machine Learning in 2D Materials …, 2023

snippet: … GPT-3 is an autoregressive model built with 96 layers [31], increased embedding vector dimension of 12888, trained with 175B parameters on five datasets including Common crawl, WebText2, Books1, Books2, and Wikipedia. Though GPT-3 has …


title: The Future of Intelligent Healthcare: A Systematic Analysis and Discussion on the Integration and Impact of Robots Using Large Language Models for Healthcare

snippet: … Namely, datasets used for training LLMs such as Common Crawl contain petabytes of data [260]. Therefore, it is not possible to have complete human oversight over the data cleaning process. Hence, this is why heuristic methods are …


title: The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)

authors: S Zeng, J Zhang, P He, Y Xing, Y Liu, H Xu, J Ren… - arXiv preprint arXiv …, 2024

snippet: … 2021), we randomly select chunks from the Common Crawl dataset to serve as the {information} component. … 2021), we randomly select chunks from the Common Crawl dataset to serve as the {information} component. Due to the random …


title: The High Stakes of AI Ethics: Evaluating OpenAI's Potential Shift to “NSFW” Content and Other Concerns

snippet: … These address concerns surrounding the dubious curation practices used to generate these datasets, the sordid quality of alt-text data available on the world wide web, the problematic content of the CommonCrawl dataset often used as a …


title: The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysis

authors: M Zhang, V Gautam, M Wang, JO Alabi, X Shen… - arXiv preprint arXiv …, 2024

snippet: … The order of languages follows their data ratio in the CommonCrawl corpus10 from high-resource to low-resource. We observe large variations in model performance across different languages. For instance, there exists a large …


title: The Impact of Geometric Complexity on Neural Collapse in Transfer Learning

authors: M Munn, B Dherin, J Gonzalvo - arXiv preprint arXiv:2405.15706, 2024

snippet: Many of the recent remarkable advances in computer vision and language models can be attributed to the success of transfer learning via the pre-training of large foundation models. However, a theoretical framework which explains this empirical …


title: The Impact of Infrastructure: Considerations of Generative AI in the Classroom

authors: ES Kehlenbach - Journal of Political Science Education, 2024

snippet: How are we to grapple with the increasing intrusion of technology in the classroom? What considerations need to be addressed before we begin bringing advanced technological systems like ChatGPT into our classroom spaces? A close attention to …


title: The impact of language family on D2T generation in under-resourced languages

snippet: The paper delves deep into the challenges and methods of generating text from structured data(RDF triples) for under-resourced languages based on the WebNLG challenge. The main question of this paper is to assess how important language …


title: The Impact of Online Censorship on LLMs

authors: M Ahmed, J Knockel - Free and Open Communications on the Internet, 2024

snippet: … the Common Crawl contains censored content, there may be downstream effects on the models. To investigate whether the Common Crawl … Of which, we found pages indexed from 325 of the censored websites in the Common Crawl dataset that …


title: The Impact of OpenAI on the Work Environment located in Finland

snippet: In the modern era, technology has undergone a rapid transformation that touches every aspect of our lives. At the forefront of this transformation stands artificial intelligence (AI), a field rapidly evolving through groundbreaking innovations, and …


title: The life cycle of large language models in education: A framework for understanding sources of bias

authors: J Lee, Y Hicke, R Yu, C Brooks, RF Kizilcec - British Journal of Educational …, 2024

snippet: Large language models (LLMs) are increasingly adopted in educational contexts to provide personalized support to students and teachers. The unprecedented capacity of LLM‐based applications to understand and generate natural language can …


title: The model student: GPT-4 performance on graduate biomedical science exams

authors: D Stribling, Y Xia, MK Amer, KS Graim, CJ Mulligan… - Scientific Reports, 2024

snippet: … As these study materials may have been incorporated into the GPT-4 training data such as the Common Crawl 43 , standardized examinations may not be an accurate assessment of domain-specific model knowledgebase and capability …


title: The OECD-UNSD Multinational Enterprise Information Platform

snippet: … (3) Common Crawl is an open source initiative to provide a monthly snapshot of the internet through web scraping. As part of the project, they provide a quarterly network graph of hyperlinks between domains, which can be used to determine …


title: The Open Web Index

authors: G Hendriksen, M Dinzinger, SM Farzana, NA Fathima…

snippet: … To operationalize our crawling pipeline, we introduce a re-crawl mechanism specifically for Common Crawl dumps. Unlike Common Crawl, we crawl on a daily basis and save the results as WARC (Web ARChive) files.This approach allows us …


title: The Open Web Index: Crawling and Indexing the Web for Public Use

authors: G Hendriksen, M Dinzinger, SM Farzana, NA Fathima… - European Conference on …, 2024

snippet: … To operationalize our crawling pipeline, we introduce a re-crawl mechanism specifically for Common Crawl dumps. Unlike Common Crawl, we crawl on a daily basis and save the results as WARC (Web ARChive) files. Footnote 11 This …


title: The Power of Absence: Thinking with Archival Theory in Algorithmic Design

authors: J Sherman, R Morrison, L Klein, DK Rosner - arXiv preprint arXiv:2405.05420, 2024

snippet: … , for example–but also of datasets like the Colossal Clean Common Crawl corpus, or C4, which explicitly documents its decision to include … [33] have shown how the quality filters imposed on the original Common Crawl dataset exhibit a preference …


title: The Practical Epistemologies of Design and Artificial Intelligence

authors: W Billingsley - Science & Education, 2024

snippet: This article explores the epistemological trade-offs that practical and technology design fields make by exploring past philosophical discussions of design, practitioner research, and pragmatism. It argues that as technologists apply Artificial …


title: The principal components of meaning, revisited

authors: C Westbury, M Yang, K Anderson - Psychonomic Bulletin & Review, 2024

snippet: … the Common Crawl corpus (https://commoncrawl.org/), a web-derived corpus. As its name suggests, the model has vectors of length 300. … ://nlp.stanford.edu/projects/glove/), a matrix with vectors of length 300 that was trained on 840 billion tokens from …


title: The research landscape on generative artificial intelligence: a bibliometric analysis of transformer-based models

snippet: Purpose The aim of this study is to offer valuable insights to businesses and facilitate better understanding on transformer-based models (TBMs), which are among the widely employed generative artificial intelligence (GAI) models, garnering …


title: The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources

authors: S Longpre, S Biderman, A Albalak, H Schoelkopf… - arXiv preprint arXiv …, 2024

snippet: … These corpora often merge and deduplicate multiple years of common crawl scrapes, or supplement additional sources including biomedical text (PubMed), legal text (Freelaw, … 2023) provides a 3 trillion token, 2023 processed split of …


title: The Revealed Preferences of Pre-authorized Licenses and Their Ethical Implications for Generative Models

authors: VM Suriyakumar, P Menell, D Hadfield-Menell…

snippet: This work examines the revealed preferences of creators as reflected in open and quasi-open licensing regimes based on the most commonly used licenses by copyright holders of images in the Creative Commons and copyright holders of code …


title: The Right Prompts for the Job: Repair Code-Review Defects with Large Language Model

authors: Z Zhao, Z Xu, J Zhu, P Di, Y Yao, X Ma - arXiv preprint arXiv:2312.17485, 2023

snippet: Automatic program repair (APR) techniques have the potential to reduce manual efforts in uncovering and repairing program defects during the code review (CR) process. However, the limited accuracy and considerable time costs associated with …


title: The Right to Scrap Data on the Internet: From the US Case hiQLabs, Inc. v. LinkedIn Corp. to the ChatGPT Scraping Cases: Differences Between US and EU Law [pre …

authors: JTS y López - Global Privacy Law Review, 2024

snippet: … a doctor when she was undergoing treatment for a rare genetic condition ended up in the Common Crawl’s archive and LAION’s dataset, and therefore in the GhatGPT datasets (because the Common Crawl is one of the ChatGPT training …


title: The Role and Prospects of Using ChatGPT Artificial Intelligence-Based Chatbots in Healthcare Improvement

authors: NA Mumtaz - … OF INTERNATIONAL SEMINAR AND WORKSHOP ON …

snippet: In the age of digital transformation, healthcare is undergoing a revolution through information technology and artificial intelligence (AI). This article explores the role of AI-based chatbots, like ChatGPT, in healthcare enhancement. Utilizing a literature …


title: The Science of Data Filtering: Data Curation cannot be Compute Agnostic

authors: S Goyal, P Maini, ZC Lipton, A Raghunathan, JZ Kolter - ICLR 2024 Workshop on …

snippet: … In fact, we show that even a model trained on *unfiltered common crawl* obtains higher accuracy than that trained on the LAION dataset post 40 or more repetitions. While past research in neural scaling laws has considered web data to be …


title: The science of implicit race bias: Evidence from the Implicit Association Test

authors: KN Morehouse, MR Banaji - Daedalus, 2024

snippet: Beginning in the mid-1980s, scientific psychology underwent a revolution–the implicit revolution–that led to the development of methods to capture implicit bias: attitudes, stereotypes, and identities that operate without full conscious awareness …


title: The Self in Action: Narrating Agentic Moments

authors: S Zilberstein, E Ayala-Hurtado, M Sanchez, D Robey - RSF: The Russell Sage …, 2024

snippet: This article develops a cultural and contextual approach to studying agency that attends to variation in how people narrate their experiences. Drawing on the large-scale, nationally representative American Voices Project data, the article uses …


title: The Shape of Word Embeddings: Recognizing Language Phylogenies through Topological Data Analysis

authors: O Draganov, S Skiena - arXiv preprint arXiv:2404.00500, 2024

snippet: Word embeddings represent language vocabularies as clouds of $d$-dimensional points. We investigate how information is conveyed by the general shape of these clouds, outside of representing the semantic meaning of each token. Specifically, we …


title: The Silicone Ceiling: Auditing GPT's Race and Gender Biases in Hiring

authors: L Armstrong, A Liu, S MacNeil, D Metaxa - arXiv preprint arXiv:2405.04412, 2024

snippet: Large language models (LLMs) are increasingly being introduced in workplace settings, with the goals of improving efficiency and fairness. However, concerns have arisen regarding these models' potential to reflect or exacerbate social biases …


title: The Sociolinguistic Foundations of Language Modeling

authors: J Grieve, S Bartl, M Fuoli, J Grafmiller, W Huang… - arXiv preprint arXiv …, 2024

snippet: In this paper, we introduce a sociolinguistic perspective on language modeling. We claim that large language models are inherently models of varieties of language, and we consider how this insight can inform the development and deployment of …


title: The Tokenization Problem: Understanding Generative AI's Computational Language Bias

authors: M Asprovska, N Hunter - Ubiquity Proceedings, 2024

snippet: … This paper used the Common Crawl corpus as a stand-in due to its wide linguistic representation, but the actual distribution within the pre-training dataset may differ. The discrepancies between these datasets might significantly influence tokenization …


title: The Trade-off between Performance, Efficiency, and Fairness in Adapter Modules for Text Classification

authors: MD Bui, K von der Wense - arXiv preprint arXiv:2405.02010, 2024

snippet: … 2019) is derived from 393,423 online biographies in English from the Common Crawl corpus, each including the subject’s occupation and gender. The dataset contains 28 occupations, assuming a binary gender classification. Gender identification …


title: The use of machine learning and deep learning models in detecting depression on social media: A systematic literature review

authors: WA Gadzama, D Gabi, MS Argungu, HU Suru - Personalized Medicine in Psychiatry, 2024

snippet: Depression is regarded as one of the world's primary concerns. Recent researchers use artificial intelligence techniques like machine learning and deep learning to identify depressive symptoms automatically. This literature review focuses on using …


title: The Web Data Commons Schema. org Table Corpora

authors: R Peeters, A Brinkmann, C Bizer - Companion Proceedings of the ACM on Web …, 2024

snippet: … This paper presented the WDC Schema.org Table Corpora that we extracted from the Common Crawl. To the best of our knowledge, the corpora are the largest public table corpora that contain tables from many different sources which all share a …


title: The Web Is Missing an Essential Part of Infrastructure: An Open Web Index A proposal for building an index of the Web that separates the infrastructure part of the …

snippet: The web as it currently exists would not be possible without search engines. They are an integral part of the Web and can also be seen as a part of the Web’s infrastructure. Google alone now serves over two trillion search queries per year. 11 …


title: The work of art in the age of artificial intelligibility

snippet: The emergence of complex deep-learning models capable of producing novel images on a practically innumerable number of subjects and in an equally wide variety of artistic styles is beginning to highlight serious inadequacies in the ethical …


title: Themes and sentiments in conversations about food waste on Twitter: Proposal of a framework using neural topic modeling

authors: MW Barbosa, A Gomes - Food Quality and Preference, 2024

snippet: … In this database, Twitter data was extracted from CommonCrawl. This method resulted in a collection spanning 80,000 distinct users. Historical snapshots from CommonCrawl, which spanned two years, were also considered to augment the …


title: Theoretical and Methodological Framework for Studying Texts Produced by Large Language Models

authors: J Milička - arXiv preprint arXiv:2408.16740, 2024

snippet: This paper addresses the conceptual, methodological and technical challenges in studying large language models (LLMs) and the texts they produce from a quantitative linguistics perspective. It builds on a theoretical framework that …


title: Threat Modelling and Risk Analysis for Large Language Model (LLM)-Powered Applications

authors: SB Tete - arXiv preprint arXiv:2406.11007, 2024

snippet: The advent of Large Language Models (LLMs) has revolutionized various applications by providing advanced natural language processing capabilities. However, this innovation introduces new cybersecurity challenges. This paper …


title: Time for a Pause: Without Effective Public Oversight, AI in Schools Will Do More Harm Than Good.

authors: B Williamson, A Molnar, F Boninger - 2024

snippet: Public education is a public and private good essential to democratic civic life. The public must, therefore, be able to provide meaningful direction over schools through transparent democratic governance structures. And yet artificial intelligence (AI) 1 …


title: Time Machine GPT

authors: F Drinkall, E Rahimikia, JB Pierrehumbert, S Zohren - arXiv preprint arXiv:2404.18543, 2024

snippet: … To scale to even larger models, processing the annual Common Crawl datasets is a necessary step, though the dataset has proved problematic due to its scale and lack of … Aside from this, cleaning Common Crawl would also demand significant …


title: Timely Quality Problem Resolution in Peer-Production Systems: The Impact of Bots, Policy Citations, and Contributor Experience

authors: V Mindel, A Aaltonen, A Rai, L Mathiassen, W Jabr - Information Systems Research, 2024

snippet: Although online peer-production systems have proven to be effective in producing high-quality content, their open call for participation makes them susceptible to ongoing quality problems. A key concern is that the problems should be addressed …


title: To Code, or Not To Code? Exploring Impact of Code in Pre-training

authors: V Aryabumi, Y Su, R Ma, A Morisot, I Zhang, A Locatelli… - arXiv preprint arXiv …, 2024

snippet: Including code in the pre-training data mixture, even for models not specifically designed for code, has become a common practice in LLMs pre-training. While there has been anecdotal consensus among practitioners that code data plays a vital role …


title: To store or not: Online cost optimization for running big data jobs on the cloud

authors: X Fu, L Pan, S Liu - Future Generation Computer Systems, 2024

snippet: As businesses increasingly rely on cloud-based big data analytics services to drive insights, reducing the cost of storing and analyzing large volumes of data in the cloud has become a major concern. During the execution of big data analysis jobs …


title: TOFU: A Task of Fictitious Unlearning for LLMs

authors: P Maini, Z Feng, A Schwarzschild, ZC Lipton, JZ Kolter - arXiv preprint arXiv …, 2024

snippet: … LLMs trained on Common Crawl data1 may be able to correctly answer factual questions about this person and they may wish to have their data removed from an LLM. In fact, regulations around the Right to be Forgotten that focus on this situation …


title: TOMGPT: Reliable Text-Only Training Approach for Cost-Effective Multi-modal Large Language Model

authors: Y Chen, Q Wang, S Wu, Y Gao, T Xu, Y Hu - ACM Transactions on Knowledge …, 2024

snippet: … For instance, c4 [37] dataset, a colossal and cleaned version of the web crawl corpus from Common Crawl, serves as a suitable option. Since the CLIP text encoder has limitations in processing long text, we subdivided the text in the c4 …


title: Tonirodriguez at CheckThat! 2024: Is it Possible to Use Zero-Shot Cross-Lingual Methods for Subjectivity Detection in Low-Resources Languages?

authors: A Rodríguez, E Golobardes, J Suau - 2024

snippet: … on 2.5TB of filtered CommonCrawl data containing 100 languages. Following the work of XLM and RoBERTa, the XLM-RoBERTa or XLM-R model takes multilingual pretraining one step further by massively upscaling the training data [19]. Using the …


title: Tooling for big data extraction

authors: R Beilich - HPI Future SOC Lab–Proceedings 2020, 2024

snippet: … , provided by Common Crawl, what may causes libraries to crumble and others to rise. … The Common Crawl Foundation created “an open repository of web crawl data that can be … [5] For the Common Crawl dataset this results in three types of …


title: Topic Modelling Swahili Using LDA and Contextualized Embeddings

authors: DW Kahenya, W Mwangi, R Rimiru - 2024 Intelligent Methods, Systems, and …, 2024

snippet: … The XLMRoBERTa model, developed by [19] in their study on Unsupervised Cross-lingual Representation Learning at Scale, was trained on 2.5TB of filtered CommonCrawl data containing 100 languages. Flair framework [20] was used to …


title: Toward Better Education Quality through Students' Sentiment Analysis Using AutoML

authors: C Simionescu, D Marcu, MS Măciucă - … . Broad Research in Artificial Intelligence and …, 2024

snippet: Sentiment analysis from students' interactions with learning environments is a topic of interest for researchers in the field of education because it can make important contributions to improving the quality of instructional processes through …


title: Toward Inference-optimal Mixture-of-Expert Large Language Models

authors: L Yun, Y Zhuang, Y Fu, EP Xing, H Zhang - arXiv preprint arXiv:2404.02852, 2024

snippet: Mixture-of-Expert (MoE) based large language models (LLMs), such as the recent Mixtral and DeepSeek-MoE, have shown great promise in scaling model size without suffering from the quadratic growth of training cost of dense transformers …


title: Toward Robust Arabic AI-Generated Text Detection: Tackling Diacritics Challenges

authors: H Alshammari, K Elleithy - Information, 2024

snippet: Current AI detection systems often struggle to distinguish between Arabic human-written text (HWT) and AI-generated text (AIGT) due to the small marks present above and below the Arabic text called diacritics. This study introduces robust Arabic text …


title: Toward Understanding Training Data Attacks In Artificial Intelligence And Machine Learning Models

snippet: … With all of these changes, the most accessible source of training data continues to be open source data, such as the Common Crawl datasets(Tom B. Brown, 2020) and others that have been made publicly available. Only extremely large technology …


title: Towards a Framework for Openness in Foundation Models: Proceedings from the Columbia Convening on Openness in Artificial Intelligence

authors: A Basdevant, C François, V Storchan, K Bankston… - arXiv preprint arXiv …, 2024

snippet: Over the past year, there has been a robust debate about the benefits and risks of open sourcing foundation models. However, this discussion has often taken place at a high level of generality or with a narrow focus on specific technical attributes. In …


title: Towards a Scalable Geoparsing Approach for the Web

snippet: The ongoing surge in web data generation and storage, coupled with embedded geographic information, holds immense potential for enhancing search applications across diverse domains. However, extracting geographic information for further …


title: Towards a small language model powered chain‐of‐reasoning for open‐domain question answering

authors: J Roh, M Kim, K Bae - ETRI Journal, 2024

snippet: … Unlike the ONUS approach [10], which extracts simple sub-question examples from a Common Crawl, our method emphasizes the autonomous generation of a dataset without human intervention facilitated by the use of LLMs. Consequently, we …


title: Towards a sociology of recurrent events. Constellations of cultural change around Eurovision in 18 countries (1981–2021)

authors: L Carbone, J Mijs, T van Dooremalen, S Daenekindt - Poetics, 2024

snippet: … First, we used CMD with a corpus that was pre-trained on Common Crawl data using fastText English embeddings. Common Crawl is a vast and heterogeneous source of information that crawls web archives and freely provides their data to the …


title: Towards a Unified Terminology: A study on Siemens Energy Technical Information Systems

snippet: … The dataset includes the Common Crawl archive, an extensive collection of web-extracted text [5], as well as the WebText dataset, books … GPT-3’s training includes a staggering 400 billion tokens from Common Crawl, 19 billion from WebText, 12 …


title: Towards adaptive support for self‐regulated learning of causal relations: Evaluating four Dutch word vector models

authors: HJ Pijeira‐Díaz, S Braumann, J van de Pol, T van Gog… - British Journal of …, 2024

snippet: Advances in computational language models increasingly enable adaptive support for self‐regulated learning (SRL) in digital learning environments (DLEs; eg, via automated feedback). However, the accuracy of those models is a common concern …


title: Towards Automated Solution Recipe Generation for Industrial Asset Management with LLM

authors: N Zhou, D Patel, S Lin, F O'Donncha - arXiv preprint arXiv:2407.18992, 2024

snippet: This study introduces a novel approach to Industrial Asset Management (IAM) by incorporating Conditional-Based Management (CBM) principles with the latest advancements in Large Language Models (LLMs). Our research introduces an …


title: Towards automatic question generation using pre-trained model in academic field for Bahasa Indonesia

authors: D Suhartono, MRN Majiid, R Fredyan - Education and Information Technologies, 2024

snippet: … This model is a potent tool for transforming our text data into numerical representations because it was previously trained on the sizable Common Crawl Dataset in Indonesian. These embeddings are essential for later modeling and …


title: Towards Better Statistical Understanding of Watermarking LLMs

authors: Z Cai, S Liu, H Wang, H Zhong, X Li - arXiv preprint arXiv:2403.13027, 2024

snippet: In this paper, we study the problem of watermarking large language models (LLMs). We consider the trade-off between model distortion and detection ability and formulate it as a constrained optimization problem based on the green-red algorithm …


title: Towards Building Multilingual Language Model for Medicine

authors: P Qiu, C Wu, X Zhang, W Lin, H Wang, Y Zhang… - arXiv preprint arXiv …, 2024

snippet: In this paper, we aim to develop an open-source, multilingual language model for medicine, that the benefits a wider, linguistically diverse audience from different regions. In general, we present the contribution from the following aspects: first, for …


title: Towards Chapterisation of Podcasts Detection of Host and Structuring Questions in Radio Transcripts

snippet: This Master thesis investigates the application of Bidirectional Encoder Representations from Transformers (BERT) on podcast to identify the host and detect structuring questions within each episode. This research is conducted on an …


title: Towards detection and explanation of factual inconsistencies

snippet: Factual inconsistencies in text, which include a range of errors from minor inaccuracies to substantial distortions, present a significant challenge in the realm of information dissemination. These inconsistencies, whether unintentional or the …


title: Towards Disfluency Annotated Corpora for Indian Languages

authors: C Kochar, V Mujadia, P Mishra, DM Sharma

snippet: In the natural course of spoken language, individuals often engage in thinking and self-correction during speech production. These instances of interruption or correction are commonly referred to as disfluencies. When preparing data for …


title: Towards Effective Time-Aware Language Representation: Exploring Enhanced Temporal Understanding in Language Models

authors: J Wang, A Jatowt, Y Cai - arXiv preprint arXiv:2406.01863, 2024

snippet: In the evolving field of Natural Language Processing, understanding the temporal context of text is increasingly crucial. This study investigates methods to incorporate temporal information during pre-training, aiming to achieve effective time-aware …


title: Towards Estimating Personal Values in Song Lyrics

authors: AM Demetriou, J Kim, S Manolios, C Liem - arXiv preprint arXiv:2408.12694, 2024

snippet: Most music widely consumed in Western Countries contains song lyrics, with US samples reporting almost all of their song libraries contain lyrics. In parallel, social science theory suggests that personal values - the abstract goals that guide our …


title: Towards Green AI. A methodological survey of the scientific literature

authors: E Barbierato, A Gatti - IEEE Access, 2024

snippet: The pervasive deployment of Deep Learning models has recently prompted apprehensions regarding their ecological footprint, owing to the exorbitant levels of energy consumption necessitated by the training and inference processes. The term …


title: Towards interactive information seeking: conversational question answering

snippet: … To further advance the field, additional conversational search benchmarks have been introduced, such as ORQuAC [93], which extends CAsT and QuAC with more conversations constructed from Natural Questions and additional passages from …


title: Towards Large Language Models for Everyone: Instruction Following, Knowledge Retrieval and Multilingualism

snippet: Large language models (LLMs) have significantly advanced the field of Natural Language Processing and demonstrated the potential to fuel a variety of AI applications. Nonetheless, building them in a way that maximally benefits the very …


title: Towards Machine Translation Based on Monolingual Texts

snippet: … The CommonCrawl1 project carries out periodic web crawls and publishes the crawled data in an open … the CommonCrawl corpus. For example, the open source OSCAR2 project compiled a large multilingual corpus by language …