title: BIS Papers

authors: I Aldasoro, S Doerr, L Gambacorta, S Notra, T Oliviero… - 2024

snippet: Generative artificial intelligence (gen AI) introduces novel opportunities to strengthen central banks’ cyber security but also presents new risks. We use data from a unique survey among cyber security experts at major central banks to shed …


title: BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages

authors: J Myung, N Lee, Y Zhou, J Jin, RA Putri, D Antypas… - arXiv preprint arXiv …, 2024

snippet: … [21] introduce a methodology to extract large-scale cultural commonsense knowledge from the Common Crawl corpus on geography, religion, and occupations. CREHate [16] is a cross-cultural English hate speech dataset covering annotations …


title: Blind Baselines Beat Membership Inference Attacks for Foundation Models

authors: D Das, J Zhang, F Tramèr - arXiv preprint arXiv:2406.16201, 2024

snippet: … More precisely, these benchmarks introduce two datasets, CommonPool and DCLM-POOL, which contain respectively 12.8 billion image-text pairs and 240 trillion text tokens sampled from Common Crawl. To enable experiments at …


title: BlockLLM: Memory-Efficient Adaptation of LLMs by Selecting and Optimizing the Right Coordinate Blocks

authors: AV Ramesh, V Ganapathiraman, IH Laradji, M Schmidt - arXiv preprint arXiv …, 2024

snippet: … The C4 dataset is a large-scale, cleaned version of the Common Crawl web corpus used for pre-training language models, featuring diverse and high-quality text from the internet. Our experiment setup is similar to [34], following the setup from …


title: Boost Large Language Model Performance through Self-Training with Reward Guided Tree Search

authors: J Hargreaves, E Fairweather, O Bellingham - 2024

snippet: Large language models have changed the field of natural language processing by enabling sophisticated language generation tasks, yet there remains a persistent challenge in enhancing their performance through autonomous learning …


title: Botlitica: A generative AI-based tool to assist journalists in navigating political propaganda campaigns

authors: E Musi, EEG Aguilar, L Federico - Studies in Communication Sciences, 2024

snippet: The hype on generative AI has raised concerns about the spread of disinformation but also opened up new opportunities for hybrid journalism. The proliferation of political propaganda campaigns spread across digital media during election periods …


title: BOTS-LM: Training Large Language Models for Setswana

authors: N Brown, V Marivate - arXiv preprint arXiv:2408.02239, 2024

snippet: In this work we present BOTS-LM, a series of bilingual language models proficient in both Setswana and English. Leveraging recent advancements in data availability and efficient fine-tuning, BOTS-LM achieves performance similar to models …


title: Brain Rush

snippet: … LLM developers could tap free data libraries – such as the Common Crawl repository of web data – supplemented by licensing foreign-language …


title: Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM

authors: S Sukhbaatar, O Golovneva, V Sharma, H Xu, XV Lin… - arXiv preprint arXiv …, 2024

snippet: We investigate efficient methods for training Large Language Models (LLMs) to possess capabilities in multiple specialized domains, such as coding, math reasoning and world knowledge. Our method, named Branch-Train-MiX (BTX), starts …


title: Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensembling

authors: YC Yu, CC Kuo, Z Ye, YC Chang, YS Li - arXiv preprint arXiv:2406.12585, 2024

snippet: Ensembling multiple models has always been an effective approach to push the limits of existing performance and is widely used in classification tasks by simply averaging the classification probability vectors from multiple classifiers to achieve …


title: Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models

authors: T Blevins, T Limisiewicz, S Gururangan, M Li, H Gonen… - arXiv preprint arXiv …, 2024

snippet: Despite their popularity in non-English NLP, multilingual language models often underperform monolingual ones due to inter-language competition for model parameters. We propose Cross-lingual Expert Language Models (X-ELM), which …


title: Brevity is the soul of wit: Pruning long files for code generation

authors: AK Singh, Y Yang, K Tirumala, M Elhoushi, AS Morcos - arXiv preprint arXiv …, 2024

snippet: Data curation is commonly considered a "secret-sauce" for LLM training, with higher quality data usually leading to better LLM performance. Given the scale of internet-scraped corpora, data pruning has become a larger and larger focus. Specifically, many have …


title: Bridging large language model disparities: Skill tagging of multilingual educational content

authors: Y Kwak, ZA Pardos - British Journal of Educational Technology, 2024

snippet: The adoption of large language models (LLMs) in education holds much promise. However, like many technological innovations before them, adoption and access can often be inequitable from the outset, creating more divides than they bridge. In …


title: Bridging the Bosphorus: Advancing Turkish Large Language Models through Strategies for Low-Resource Language Adaptation and Benchmarking

authors: EC Acikgoz, M Erdogan, D Yuret - arXiv preprint arXiv:2405.04685, 2024

snippet: … This dataset was generated by extracting content from 71 monthly snapshots of the internet via Common Crawl (CC). CulturaX contains version … However, the diversity of these datasets should be sufficient, including common crawl, book …


title: Browsing and Searching Metadata of TREC

authors: T Breuer, EM Voorhees, I Soboroff - Proceedings of the 47th International ACM SIGIR …, 2024

snippet: Information Retrieval (IR) research is deeply rooted in experimentation and evaluation, and the Text REtrieval Conference (TREC) has been playing a central role in making that possible since its inauguration in 1992. TREC's mission centers …


title: Building a Large Japanese Web Corpus for Large Language Models

authors: N Okazaki, K Hattori, H Shota, H Iida, M Ohi, K Fujii… - arXiv preprint arXiv …, 2024

snippet: … The corpus developed in this study is made from 21 snapshots of Common Crawl (from CC-MAIN-2020-40 to CC-MAIN-2023-23), and the size of the corpus after cleaning is 173,350,375 pages and 312,093,428,689 characters. To confirm the …


title: Building a Specialized ChatGpt-Like System

snippet: … be examined, which was sampled from a mix primarily made up of the Common Crawl dataset [11]. This dataset is large enough to train the … This means that Common Crawl and Books2 datasets are sampled less than once during training …


title: Building and AI Chatbot using LLM

snippet: In our digital world, where communication is key, the emergence of AI-powered chatbots has revolutionized the way we interact with technology. This project delves into the development of an AI chatbot utilizing Large Language Models (LLM), a …


title: Building and better understanding vision-language models: insights and future directions

authors: H Laurençon, A Marafioti, V Sanh, L Tronchon - arXiv preprint arXiv:2408.12637, 2024

snippet: The field of vision-language models (VLMs), which take images and texts as inputs and output texts, is rapidly evolving and has yet to reach consensus on several key aspects of the development pipeline, including data, architecture, and training …


title: Building Better Datasets: Seven Recommendations for Responsible Design from Dataset Creators

authors: W Orr, K Crawford - arXiv preprint arXiv:2409.00252, 2024

snippet: The increasing demand for high-quality datasets in machine learning has raised concerns about the ethical and responsible creation of these datasets. Dataset creators play a crucial role in developing responsible practices, yet their …


title: Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers

authors: X Wei, S Moalla, R Pascanu, C Gulcehre - arXiv preprint arXiv:2406.16450, 2024

snippet: … Dataset We use the RefinedWeb dataset [44], a carefully curated subset of CommonCrawl, processed for optimal filtering and deduplication. It provides 600B tokens available for public use. As it is a substantial dataset, we shuffle, extract, and …


title: Building Question-Answer Data Using Web Register Identification

authors: A Eskelinen, A Myntti, E Henriksson, S Pyysalo… - Proceedings of the 2024 …, 2024

snippet: … (2015), is a corpus of Finnish collected between 2015 and 2016 from Common Crawl and by crawling .fi domains. As all of these resources are Common Crawl based, in the post-processing stage we make sure that duplicate QA pairs are …


title: By Senior Judge Stephanie Domitrovich and Judge Herbert B. Dixon Jr.

authors: CJJ Federal Judiciary - Judges Journal, 2024

snippet: … For example, the datasets used to train ChatGPT 3 included text from the following sources: (1) Common Crawl, containing petabytes of data collected from the internet since 2008; (2) WebText2, text from webpages with highly ranked Reddit …


title: Bye-bye Bias: What to Consider When Training Generative AI Models on Subjective Marketing Metrics

snippet: … However, these models are trained on vast data sets scraped from the Internet, such as Common Crawl or LAION, that contain little information on the types of perceptual measures marketing is often interested in, such as perceived brand …


title: Cache Me If You Can: The Case For Retrieval Augmentation in Federated Learning

authors: A Muhamed, P Thaker, MT Diab, V Smith - Privacy Regulation and Protection in Machine …

snippet: We propose retrieval augmentation (RA) as an enhancement to federated learning (FL) that can improve privacy protection and ensure regulatory compliance. FL, primarily designed for data privacy preservation, faces challenges with conventional …


title: CAM 2.0: End-to-End Open Domain Comparative Question Answering System

authors: A Shallouf, H Herasimchyk, M Salnikov, RG Veliz…

snippet: … After extracting objects and aspects, we look for their matches in the Common Crawl text. It is also important to mention, that for the final … (lemmatized) corpus containing 14.3 billion English sentences from the Common Crawl. To retrieve …


title: Can ChatGPT learn Chinese or Swahili? Considering how large language models might act differently if trained in different languages.

snippet: … Thien Nguyen, a professor of computer science at the University of Oregon, looked at the CommonCrawl corpus, a selection of text scraped … Then, because only a small percentage of the CommonCrawl corpus is in Korean, the company …


title: Can ChatGPT Plan Your Retirement?: Generative AI andFinancial Advice

snippet: We identify some of the most pressing issues facing the adoption of large language models (LLMs) in practical settings, and propose a research agenda to reach the next technological inflection point in generative AI. We focus on three challenges …


title: Can cross-domain term extraction benefit from cross-lingual transfer and nested term labeling?

authors: HTH Tran, M Martinc, A Repar, N Ljubešić, A Doucet… - Machine Learning, 2024

snippet: Automatic term extraction (ATE) is a natural language processing task that eases the effort of manually identifying terms from domain-specific corpora by providing a list of candidate terms. In this paper, we treat ATE as a sequence-labeling task and …


title: Can Large Language Models Accelerate Digital Transformation by Generating Expert-Like Systems Engineering Artifacts? Insights from an Empirical Exploration

authors: M Husain, P Wach, TG Topcu - The Proceedings of the 2024 Conference on Systems …

snippet: The old saga of complex system development continues as the vast majority of government and industry programs continue to result in cost and schedule overruns [1–7]. While numerous researchers and government reports attribute this trend to the ever-increasing …


title: Can large language models understand molecules?

authors: S Sadeghi, A Bui, A Forooghi, J Lu, A Ngom - BMC bioinformatics, 2024

snippet: Purpose Large Language Models (LLMs) like Generative Pre-trained Transformer (GPT) from OpenAI and LLaMA (Large Language Model Meta AI) from Meta AI are increasingly recognized for their potential in the field of cheminformatics, particularly …


title: Can Models Learn Skill Composition from Examples?

authors: H Zhao, S Kaur, D Yu, A Goyal, S Arora - ICML 2024 Workshop on LLMs and Cognition

snippet: As large language models (LLMs) become increasingly capable, their ability to exhibit *compositional generalization* of skills has garnered significant attention. Yu et al. 2023 recently introduced Skill-Mix evaluation, where models are tasked with …


title: Cascaded transformer-based networks for wikipedia large-scale image-caption matching

authors: N Messina, DA Coccomini, A Esuli, F Falchi - Multimedia Tools and Applications, 2024

snippet: … It uses an XLM-RoBERTa masked language model pre-trained on the CommonCrawl dataset and fine-tuned on the image URL-caption match classification task. The CLS token in output from XLM-RoBERTa is attached to a feed-forward …


title: Causality-driven multivariate stock movement forecasting

authors: A Díaz Berenguer, Y Da, MN Bossa, MC Oveneke… - PloS one, 2024

snippet: Our study aims to investigate the interdependence between international stock markets and sentiments from financial news in stock forecasting. We adopt the Temporal Fusion Transformers (TFT) to incorporate intra and inter-market …


title: CC-GPX: Extracting High-Quality Annotated Geospatial Data from Common Crawl

authors: I Ilyankou, J Haworth, S Cavazzi - arXiv preprint arXiv:2405.11039, 2024

snippet: The Common Crawl (CC) corpus is the largest open web crawl dataset containing 9.5+ petabytes of data captured since 2008. The dataset is instrumental in training large language models, and as such it has been studied for (un)desirable content …


title: CCSUM: A large-scale and high-quality dataset for abstractive news summarization

snippet: … Accordingly, among 35 million CommonCrawl News articles, we identify pairs of articles about the same news story and use one article’s first sentence as the summary for the other article. To ensure high quality, we apply strict filters whose …


title: CERM: Context-aware Literature-based Discovery via Sentiment Analysis

authors: JC Young, U Akujuobi - arXiv preprint arXiv:2402.01724, 2024

snippet: Driven by the abundance of biomedical publications, we introduce a sentiment analysis task to understand food-health relationship. Prior attempts to incorporate health into recipe recommendation and analysis systems have primarily focused on …


title: Challenge design roadmap

authors: HJE Balderas, I Guyon, A Howard, W Reade, S Treguer - AI Competitions and …, 2023

snippet: Challenges can be seen as a type of game that motivates participants to solve serious tasks. As a result, competition organizers must develop effective game rules. However, these rules have multiple objectives beyond making the game enjoyable …


title: Challenges and Contributions in Table-to-text Generation: A Survey

snippet: … The pre-training of T5 involves an unsupervised “span masking” objective applied to Common Crawl data, where spans of text are masked and the model learns to predict them, thus gaining a deep understanding of language patterns and …


title: Changes and challenges of legal education in the era of generative artificial intelligence: Chinese experience

authors: W Wang, Z Xu, Z Xu - Journal of Infrastructure, Policy and Development, 2024

snippet: Using generative artificial intelligence systems in the classroom for law case analysis teaching can enhance the efficiency and accuracy of knowledge delivery. They can create interactive learning environments that are appropriate, immersive …


title: Chatbots in Airport Customer Service—Exploring Use Cases and Technology Acceptance

authors: I Auer, S Schlögl, G Glowka - Future Internet, 2024

snippet: … Those pre-trained transformer systems like BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer) have been trained on vast language corpora such as Wikipedia and Common Crawl …


title: ChatGPT Alternative Solutions: Large Language Models Survey

authors: H Alipour, N Pendar, K Roy - arXiv preprint arXiv:2403.14469, 2024

snippet: In recent times, the grandeur of Large Language Models (LLMs) has not only shone in the realm of natural language processing but has also cast its brilliance across a vast array of applications. This remarkable display of LLM capabilities has ignited a …


title: ChatGPT and GPT-4: utilities in the legal sector

snippet: Artificial intelligence systems such as ChatGPT, the OpenAI chatbot, based on the language model family GPT (generative pre-trained transformers), as well as other solutions built on this technology and fine-tuned for specific tasks, have generated …


title: ChatGPT and Language Translation: A Small Case Study Evaluating English–Mandarin Translation

authors: C Woodrum - International Conference on Human-Computer …, 2024

snippet: … has been informed speculation that it relies heavily on common crawl [13], a compendium of publicly available … common crawl. The \(15^{th}\) most common language, Indonesian, makes up less than 1% of the data set [14, 15]. It is important …


title: ChatGPT as the Marketplace of Ideas: Should Truth-Seeking Be the Goal of AI Content Governance?

authors: J Zhang - arXiv preprint arXiv:2405.18636, 2024

snippet: As one of the most enduring metaphors within legal discourse, the marketplace of ideas has wielded considerable influence over the jurisprudential landscape for decades. A century after the inception of this theory, ChatGPT emerged as a …


title: ChatGPT in Cyber Onslaught and Fortification: Past, Present, and Future

authors: M Gunda, V Manda, P Naradasu, S Mekala… - 2024 IEEE 9th International …, 2024

snippet: … GPT-1 was unveiled in 2018 and trained on a pair of data sets: over 11,000 books from Book Corpus and the enormous Common Crawl … With 1.5 billion features and a much more extensive and diversified training dataset comprising Common Crawl …


title: ChatGPT versus NASS clinical guidelines for degenerative spondylolisthesis: a comparative analysis

authors: W Ahmed, M Saturno, R Rajjoub, AH Duey, B Zaidat… - European Spine Journal, 2024

snippet: … Furthermore, the largest dataset used to train both ChatGPT 3.5 and 4.0 is CommonCrawl, a platform that does not include PubMed and consists predominantly of open access articles, some of which may be of less rigorous if not …


title: ChatGPT vs Gemini vs LLaMA on Multilingual Sentiment Analysis

authors: A Buscemi, D Proverbio - arXiv preprint arXiv:2402.01715, 2024

snippet: … It relied on a dataset referred to as the Common Crawl [30], a publicly accessible repository that comprises billions of web pages, making it one of the most extensive text databases available. It is important to note that the choice of dataset significantly …


title: ChatGPT's applications in marketing: a topic modeling approach

authors: W Tafesse, A Wien - Marketing Intelligence & Planning, 2024

snippet: Purpose ChatGPT is a versatile technology with practical use cases spanning many professional disciplines including marketing. Being a recent innovation, however, there is a lack of academic insight into its tangible applications in the marketing …


title: ChatGPT, the perfect virtual teaching assistant? Ideological bias in learner-chatbot interactions

authors: M Van Poucke - Computers and Composition, 2024

snippet: This paper examines ChatGPT's use of evaluative language and engagement strategies while addressing information-seeking queries. It assesses the chatbot's role as a virtual teaching assistant (VTA) across various educational settings. By …


title: ChatGPT-3.5 can create and justify short Afrikaans poems: Implications for practice

authors: CJ van Staden - Suid-Afrikaanse Tydskrif vir Natuurwetenskap en …, 2024

snippet: ChatGPT-3.5 can create poems but the implications thereof has not yet been sufficiently investigated. The purpose of this explorative case study was to determine to which extent ChatGPT-3.5 can create and justify Afrikaans poems. Because …


title: ChatQA: Building GPT-4 Level Conversational QA Models

authors: Z Liu, W Ping, R Roy, P Xu, M Shoeybi, B Catanzaro - arXiv preprint arXiv …, 2024

snippet: … We collect 7k documents (average ∼1k words per document) from common crawl, which cover a wide range of domains. Each document is used for generation of a single conversational QA sample, which leads to a total of 7k multi-turn QA dialogues …


title: Chatting Over Course Material: The Role of Retrieval Augmented Generation Systems in Enhancing Academic Chatbots.

snippet: Large Language Models (LLMs) have the potential to enhance learning among students. These tools can be used in chatbot systems allowing students to ask questions about course material, in particular when plugged with the so-called …


title: Cheap Learning: Maximising Performance of Language Models for Social Data Science Using Minimal Data

authors: L Castro-Gonzalez, YL Chung, HR Kirk, J Francis… - arXiv preprint arXiv …, 2024

snippet: The field of machine learning has recently made significant progress in reducing the requirements for labelled training data when building new models. These `cheaper' learning techniques hold significant potential for the social sciences, where …


title: Check for Extending the Comparative Argumentative Machine: Multilingualism and Stance Detection

authors: I Nikishina¹, A Bondarenko, S Zaczek¹, OL Haag… - Robust Argumentation Machines …

snippet: The comparative argumentative machine CAM can retrieve arguments that answer comparative questions questions that ask which of several to-be-compared options should be favored in some scenario. In this paper, we describe how we equipped …


title: Check for updates A Study of Transductive Graph-Based Regression

authors: ADB Valejo - Intelligent Systems Design and Applications: Deep …

snippet: Regression methods play an important role in many realworld applications such as econometric, pattern recognition, and prediction of protein chains to cite a few tasks. In the semi-supervised transductive learning problem, some studies present graph-based …


title: Check for updates Digilog: Enhancing Website Embedding on Local Governments-A Comparative Analysis

authors: J Gerber, B Kreiner, J Saxer, A Weiler - Foundations of Intelligent Systems: 27th International …

snippet: The ability to understand and process websites, known as website embedding, is crucial across various domains. It lays the foundation for machine understanding of websites. Specifically, website embedding proves invaluable when monitoring local …


title: CheckIT!: A Corpus of Expert Fact-checked Claims for Italian Jacopo Gili', Lucia Passaro² and Tommaso Caselli³ ¹Department of Computer Science, University of Turin …

authors: LP Jacopo Gili, T Caselli - Proceedings of the 9th Italian Conference on …, 2024

snippet: This paper introduces CheckIT!, a resource of expert fact-checked claims, filling a gap for the development of fact-checking pipelines in Italian. We further investigate the use of three state-of-the-art generative text models to create variations of claims …


title: Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model

authors: X Du, Z Yu, S Gao, D Pan, Y Cheng, Z Ma, R Yuan… - arXiv preprint arXiv …, 2024

snippet: In this study, we introduce CT-LLM, a 2B large language model (LLM) that illustrates a pivotal shift towards prioritizing the Chinese language in developing LLMs. Uniquely initiated from scratch, CT-LLM diverges from the conventional …


title: CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systems

authors: S Rosenthal, A Sil, R Florian, S Roukos - arXiv preprint arXiv:2404.02103, 2024

snippet: … However, they use sentencelevel matching (by encoding sentences for semantic similarity comparisons) to retrieve up to top 7 documents from Common Crawl while avoiding exact matches as the abstractive dataset. In the extractive version, the …


title: CLASSLA-web: Comparable Web Corpora of South Slavic Languages Enriched with Linguistic and Genre Annotation

authors: N Ljubešić, T Kuzman - arXiv preprint arXiv:2403.12721, 2024

snippet: This paper presents a collection of highly comparable web corpora of Slovenian, Croatian, Bosnian, Montenegrin, Serbian, Macedonian, and Bulgarian, covering thereby the whole spectrum of official languages in the South Slavic language space …


title: Cleaner Pretraining Corpus Curation with Neural Web Scraping

authors: Z Xu, Z Liu, Y Yan, Z Liu, C Xiong, G Yu - arXiv preprint arXiv:2402.14652, 2024

snippet: … The web-crawled datasets, such as Common Crawl, have been widely used for pretraining, facilitating the development of language … htmlparser serves as the text pre-extraction tool for CommonCrawl6 WET file (containing pre-extracted text), while …


title: CLEARNESS: Coreference Resolution for Generating and Ranking Arguments Extracted from Debate Portals for Queries

snippet: … Refining Arg-CTRL with data from Common Crawl leads to a higher quality of generated arguments compared to using user discussions from Reddit comments. In the mentioned studies, new arguments are generated through the use of knowledge …


title: CLIP and the City: Addressing the Artificial Encoding of Cities in Multimodal Foundation Deep Learning Models

snippet: In this project, we propose and explore a computational pipeline to examine urban cultural landscapes through the lens of artificial intelligence, and for questioning modes of embedding culture in machine learning models. By employing machine …


title: CLIP-CID: Efficient CLIP Distillation via Cluster-Instance Discrimination

authors: K Yang, T Gu, X An, H Jiang, X Dai, Z Feng, W Cai… - arXiv preprint arXiv …, 2024

snippet: Contrastive Language-Image Pre-training (CLIP) has achieved excellent performance over a wide range of tasks. However, the effectiveness of CLIP heavily relies on a substantial corpus of pre-training data, resulting in notable consumption …


title: Co-constructing AI Authoring Using Ethical Theories

authors: B Jones - 88th Annual International Conference

snippet: We must show ethical humility, historically contextualizing ethical standards, as we implement AI tools in courses. The pace of change requires agile approaches to defining standards for the ethical use of AI. We invited students to co-construct a …


title: CoastTerm: a Corpus for Multidisciplinary Term Extraction in Coastal Scientific Literature

authors: J Delaunay, HTH Tran, CE González-Gallardo… - arXiv preprint arXiv …, 2024

snippet: … – Multilingual pre-trained model: We opt for XLMR [9], a transformer-based model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. This multilingual version of RoBERTa, achieves benchmark performance in ATE for rich-resourced …


title: Community OSCAR: A Community Effort for Multilingual Web Data

authors: M Brack, M Ostendorff, PO Suarez, JJ Saiz, IL Castilla…

snippet: The development of large language models (LLMs) relies heavily on extensive, highquality datasets. Publicly available datasets focus predominantly on English, leaving other language communities behind. To address this issue, we introduce …


title: Comparative Analysis of Cognitive Agreement between Human Analysts and Generative AI in Construction Safety Risk Assessment

authors: U Ray, C Arteaga, JW Park - ISARC. Proceedings of the International Symposium on …, 2024

snippet: The construction industry struggles with safety risk assessment complexities due to evolving work environments, diverse labor forces, time constraints, regulatory intricacies, and inconsistent practices. While previous studies have highlighted the …


title: Comparative Analysis of Generative AI Tools in Enhancing Educational Engagement

authors: F Borović, K Aleksić-Maslać, P Vranešić - 2024 47th MIPRO ICT and Electronics …, 2024

snippet: Technological development is continuously reshaping the traditional educational setting, presenting various possibilities for educational refinement. It is pivotal to motivate active engagement among all participants, and generative AI tools emerge …


title: Comparative evaluation of machine learning algorithms for phishing site detection

authors: NF Almujahid, MA Haq, M Alshehri - PeerJ Computer Science, 2024

snippet: The advent of Internet technologies has resulted in the proliferation of electronic trading and the use of the Internet for electronic transactions, leading to a rise in unauthorized access to sensitive user information and the depletion of resources for …


title: Comparative Performance of Advanced NLP Models and LLMs in Multilingual Geo-Entity Detection

authors: K Kopanov - Proceedings of the Cognitive Models and Artificial …, 2024

snippet: The integration of advanced Natural Language Processing (NLP) methodologies and Large Language Models (LLMs) has significantly enhanced the extraction and analysis of geospatial data from multilingual texts, impacting sectors such as …


title: Comparative Study on Synthetic and Natural Error Analysis with BART & MarianMT

authors: R Rohit, SA Gandheesh, GS Sannala, PB Pati - 2024 IEEE 9th International …, 2024

snippet: Text is essential for communication, information sharing, knowledge acquisition, and analysis. It shapes public opinion, supports education, and drives online content, making it crucial in various domains. While there are various language models …


title: Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization

authors: H Bansal, A Suvarna, G Bhatt, N Peng, KW Chang… - arXiv preprint arXiv …, 2024

snippet: A common technique for aligning large language models (LLMs) relies on acquiring human preferences by comparing multiple generations conditioned on a fixed context. This only leverages the pairwise comparisons when the generations are …


title: Comparing Explanation Faithfulness between Multilingual and Monolingual Fine-tuned Language Models

authors: Z Zhao, N Aletras - arXiv preprint arXiv:2403.12809, 2024

snippet: In many real natural language processing application scenarios, practitioners not only aim to maximize predictive performance but also seek faithful explanations for the model predictions. Rationales and importance distribution given by feature …


title: Comparing Fine-Tuning, Zero and Few-Shot Strategies with Large Language Models in Hate Speech Detection in English

authors: R Pan, JA García-Díaz, R Valencia-García

snippet: … This is a multilingual model based on RoBERTa, pre-trained with a 2.5 TB filtered CommonCrawl dataset containing 100 languages, using an MLM goal, where it randomly masks 15% of the words in the input, runs the entire masked sentence …


title: Comparing Foundations: Insights into the Construction of Financial Causal Knowledge Graphs with and without Ontology

authors: Z Xu, R Ichise - 人工知能学会全国大会論文集 第 38 回 (2024), 2024

snippet: Simplicity in information navigation, interpretation, and reasoning has positioned mono-relation knowledge graphs (KGs) as a focus of attention, particularly in targeted scenarios. However, their deficiency in hierarchical interactions between …


title: Comparing generative and retrieval-based chatbots in answering patient questions regarding age-related macular degeneration and diabetic retinopathy

authors: KX Cheong, C Zhang, TE Tan, BJ Fenner, WM Wong… - British Journal of …, 2024

snippet: … ChatGPT was trained on multiple extremely large datasets, of which Common Crawl (open repository of web crawl data) constituted the bulk of the datasets.30 31 Google Bard was trained on Infiniset (data comprising public forum dialogue …


title: Comparison of Common Crawl News & GDELT

authors: A El Ouadi, D Beskow - 2024 IEEE International Systems Conference (SysCon), 2024

snippet: … Common Crawl data has proven valuable for research and training artificial intelligence, noting that 82% of raw tokens used to train GPT-3 come from the Common Crawl dataset. CC-News, a derived corpus of news data from Common …


title: Comparison of Machine Learning Algorithms and Large Language Models for Product Categorization

authors: A İhsanoğlu, M Zaval, OT Yıldız - 2024 32nd Signal Processing and Communications …, 2024

snippet: This study explores the efficacy of traditional machine learning algorithms and Large Language Models (LLMs) in automating product categorization for online e-commerce platforms. By comparing these methodologies, we assess their performance in …


title: Compass: Large Multilingual Language Model for South-east Asia

authors: S Maria - arXiv preprint arXiv:2404.09220, 2024

snippet: … We categorized these sources into seven types: CommonCrawl, C4, Wikipedia, WebText, Academic, Books, and Code. We upsampled and … We expect that the introduction of biases may originate from CommonCrawl, as this substantial dataset …


title: Competition concerns with foundation models: a new feast for big tech?

authors: S Mitra - European Competition Journal, 2024

snippet: … Some well-known FMs such as GPT-3 and LLaMA have been trained on data from public sources (such as Common Crawl). However, there are concerns that publicly available data would eventually have been completely saturated for training …


title: Compilation of a Synthetic Judeo-French Corpus

authors: I Nikolova-Stoupak, G Lejeune, E Schaeffer-Lacroix - Proceedings of the 8th Joint …, 2024

snippet: This is a short paper describing the process of derivation of synthetic Judeo-French text. Judeo-French is one of a number of rare languages used in speaking and writing by Jewish communities as confined to a particular temporal and …


title: Complex Word Identification for Italian Language: a dictionary-based approach

authors: L Occhipinti - Sixth International Conference, 2024

snippet: … The model used in our study was trained on Wikipedia and Common Crawl datasets6. These embeddings provided vector representations for each word in our dataset, primarily comprising isolated items, allowing us to incorporate contextual …


title: Compositional Text-to-Image Generation with Dense Blob Representations

authors: W Nie, S Liu, M Mardani, C Liu, B Eckart, A Vahdat - arXiv preprint arXiv:2405.08246, 2024

snippet: Existing text-to-image models struggle to follow complex text prompts, raising the need for extra grounding inputs for better controllability. In this work, we propose to decompose a scene into visual primitives - denoted as dense blob representations …


title: Comprehensive analysis of natural language processing

authors: RK Yadav, A Madaan - Global Journal of Engineering and Technology …, 2024

snippet: … Huge datasets like the Penn Treebank and the Common Crawl drove the development of more sophisticated NLP models. Transfer learning techniques like BERT enabled finetuning of pre-trained models for specific tasks. TensorFlow and …


title: Comprehensive review and comparative analysis of transformer models in sentiment analysis

authors: H Bashiri, H Naderi - Knowledge and Information Systems, 2024

snippet: Sentiment analysis has become an important task in natural language processing because it is used in many different areas. This paper gives a detailed review of sentiment analysis, including its definition, challenges, and uses. Different …


title: COMPREHENSIVE STUDY OF CLINICAL ENTITY EXTRACTION AND CLASSIFICATION USING LARGE LANGUAGE MODELS

authors: M Faedi, P Torroni, DA Galassi, DG Grundler…

snippet: This project aims to evaluate various techniques for extraction and categorization of clinical terms in unstructured documents. The purpose of this study is to assist automated systems operating in the biomedical domain by highlighting relevant …


title: Comprehensive Study on Performance Evaluation and Optimization of Model Compression: Bridging Traditional Deep Learning and Large Language Models

authors: A Saxena, AK Bishwas, AA Mishra, R Armstrong - arXiv preprint arXiv:2407.15904, 2024

snippet: … For pruning language models, we used XLM-RoBERTA [30] trained on 2.5Tb of filtered CommonCrawl data containing 100 languages, BERTsmall [31] trained on disaster tweets dataset [32] and GPT-2 small [33] finetuned on E2E NLG [34]. …


title: Compressing Lengthy Context With UltraGist

authors: P Zhang, Z Liu, S Xiao, N Shao, Q Ye, Z Dou - arXiv preprint arXiv:2405.16635, 2024

snippet: Compressing lengthy context is a critical but technically challenging problem. In this paper, we propose a new method called UltraGist, which is distinguished for its high-quality compression of lengthy context due to the innovative design of the compression and …


title: Compression Represents Intelligence Linearly

authors: Y Huang, J Zhang, Z Shan, J He - arXiv preprint arXiv:2404.09937, 2024

snippet: … Concretely, for assessing knowledge and commonsense, we have compiled texts from the latest Common Crawl dataset. To evaluate coding ability, we have sourced data from GitHub repositories mainly on the Python language since the downstream …


title: Computational Collective Intelligence: 16th International Conference, ICCCI 2024, Leipzig, Germany, September 9–11, 2024, Proceedings, Part II

snippet: This volume contains the first part of the proceedings of the 16th International Conference on Computational Collective Intelligence (ICCCI 2024), held in Leipzig, Germany from 9–11 September 2024. The conference was organized in a hybrid …


title: Computational criminology: at-scale quantitative analysis of the evolution of cybercrime forums

snippet: Cybercrime forums and marketplaces are used by members to share hacking techniques, general community-building discussions, and trade hacking tools. While there is a large corpus of literature studying these platforms, from a cross-forum …


title: Computational Measures of Language Variation in Textual Utterances

snippet: The use of language as a tool for self-expression and social communication is marked by extensive variation in the way that language is used by different people and communities. The computational modeling of this variation studies the social …


title: Computerized diagnostic decision support systems–a comparative performance study of Isabel Pro vs. ChatGPT4

snippet: … ChatGPT4 is trained on Common Crawl, a publicly available dataset and one of the most extensive text datasets. While not trained explicitly for medical diagnosis, ChatGPT4 accesses medical textbooks, medical websites, medical papers, and …


title: Comquest: an Adaptive Crawler for User Comments on the Web

snippet: This thesis introduces Comquest, an adaptive framework designed for the large-scale collection and integration of user comments from the Web. User comments are featured on many websites and there is growing interest in mining and studying user …


title: Comquest: Large Scale User Comment Crawling and Integration

authors: Z Chen, L He, A Mukherjee, E Dragut - 2024

snippet: … To train the model, we collect an extensive dataset from the Common Crawl Project. We filter the webpages that show apparent code signals (eg, loading the commenting system libraries) of the supported commenting systems. …


title: Confounders in Instance Variation for the Analysis of Data Contamination

authors: B Mehrbakhsha, D Garigliottid, F Martínez-Plumeda…

snippet: … 2020) the same approach has been taken and all data points from the evaluation sets that had a 13-gram collision in the pre-training Common Crawl (C4) dataset were removed to tackle contamination. As contamination can involve minor …


title: Consent in Crisis: The Rapid Decline of the AI Data Commons

authors: S Longpre, R Mahari, A Lee, C Lund, H Oderinwale… - arXiv preprint arXiv …, 2024

snippet: … In our study, we focus on three popular, open-source, and permissively licensed data sources which are derived from Common Crawl, the largest publicly available crawl of the web, which has collected and stored hundreds of billions of web pages …


title: Considering the Role of Fairness in Copyright Fair Use

snippet: Moral intuitions regarding when it is "fair" to engage in unauthorized copying or derivation from a copyrighted work may often take into account factors that the fair use statute, 17 USC § 107, does not enumerate among fair use considerations. This …


title: Conspiracy Theory Detection using Transformers with Multi-task and Multilingual Approaches

snippet: The COVID-19 pandemic sparked a new age of conspiracy theories in society. This has become an issue, especially since these theories are mixed in with reasonable arguments that criticize the measures taken by governments and their effects. A way …


title: Construction of Text Summarization Corpus in Economics Domain and Baseline Models

authors: S Jumpathong, A Takhom, P Boonkwan… - 2024

snippet: … It was pre-trained with a data collection of 101 languages from Common Crawl Due to time and resource constraints, we chose mT5-small… It was pre-trained with a data collection of 101 languages from Common Crawl. Due to time and resource …


title: Context-aware embeddings for robust multiclass fraudulent URL detection in online social platforms

authors: S Afzal, M Asim, MO Beg, T Baker, AI Awad, N Shamim - Computers and Electrical …, 2024

snippet: … Their dataset comprised 2 million URLs from Common Crawl and PhishTank, and they applied three-fold cross-validation with 90% training data. Feature-ranking techniques were used to identify the most important among 14 features. They …


title: Context-aware Transliteration of Romanized South Asian Languages

authors: C Kirov, C Johny, A Katanova, A Gutkin, B Roark - Computational Linguistics, 2024

snippet: While most transliteration research is focused on single tokens such as named entities – eg, transliteration of “અમદાવાદ” from the Gujarati script to the Latin script “Ahmedabad” – the informal romanization prevalent in South Asia and elsewhere …


title: Contextual Chart Generation for Cyber Deception

authors: DD Nguyen, D Liebowitz, S Nepal, SS Kanhere… - arXiv preprint arXiv …, 2024

snippet: … (1) HPN-T5 is based on Text-to-Text Transfer Transformer (T5) [42], a highly scalable sequence-to-sequence Transformer pre-trained on a large corpus harvested from Common Crawl. T5 introduced a novel pre-training objective that …


title: Contextual Word Embedding for Biomedical Knowledge Extraction: a Rapid Review and Case Study

authors: D Vithanage, P Yu, L Wang, C Deng - Journal of Healthcare Informatics Research, 2024

snippet: Recent advancements in natural language processing (NLP), particularly contextual word embedding models, have improved knowledge extraction from biomedical and healthcare texts. However, limited comprehensive research compares these models …


title: Contextually Enriched Meta-Learning Ensemble Model for Urdu Sentiment Analysis Symmetry 2023, 15, 645

authors: K Ahmed, MI Nadeem, D Li, Z Zheng, N Al-Kahtani… - 2023

snippet: … On the other hand, this model makes use of pre-trained word vectors that were trained on “common crawl” and “Wikipedia” through the use of the fastText model [82]. The CBOW algorithm with position-weighting is used to train this word vector. After …


title: Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities

authors: K Fujii, T Nakamura, M Loem, H Iida, M Ohi, K Hattori… - arXiv preprint arXiv …, 2024

snippet: Cross-lingual continual pre-training of large language models (LLMs) initially trained on English corpus allows us to leverage the vast amount of English language resources and reduce the pre-training cost. In this study, we constructed Swallow, an …


title: Controllable Sentence Simplification in Dutch

authors: T Seidl, V Vandeghinste - Computational Linguistics in the Netherlands Journal, 2024

snippet: Text simplification aims to reduce complexity in vocabulary and syntax, enhancing the readability and comprehension of text. This paper presents a supervised sentence simplification approach for Dutch using a pre-trained large language …


title: Controllable Sentence Simplification in Swedish Using Control Prefixes and Mined Paraphrases

authors: J Monsen, A Jönsson - Proceedings of the 2024 Joint International Conference …, 2024

snippet: … 2020), an opensource repository with tools to download and clean Common Crawl snapshots. Each file in the CC-100 corpus contains … The Common Crawl snapshots contain a large portion of low-quality data, including offensive and …


title: Copilot for Microsoft 365: A Comprehensive End-user Training Plan for Organizations

snippet: This thesis presents a comprehensive end user training plan for Copilot for Microsoft 365, a generative AI tool that integrates with Microsoft’s productivity suite of applications. The research is conducted within the context of Sulava Ltd., a Finnish …


title: Copyleft for Alleviating AIGC Copyright Dilemma: What-if Analysis, Public Perception and Implications

authors: X Guo, Y Li, Y Peng, X Wei - arXiv preprint arXiv:2402.12216, 2024

snippet: As AIGC has impacted our society profoundly in the past years, ethical issues have received tremendous attention. The most urgent one is the AIGC copyright dilemma, which can immensely stifle the development of AIGC and greatly cost the entire …


title: CoRePooL—Corpus for Resource‐Poor Languages: Badaga Speech Corpus

authors: HB Barathi Ganesh, G Jyothish Lal, R Jairam… - … Speech Recognition and …, 2024

snippet: This chapter presents a corpus named CoRePooL that stands for Corpus for Resource‐Poor Languages. As voice‐specific human‐machine interaction applications are accelerated by deep learning algorithms, the lack of resources …


title: Corpus Considerations for Annotator Modeling and Scaling

authors: OO Sarumi, B Neuendorf, J Plepi, L Flek, J Schlötterer… - arXiv preprint arXiv …, 2024

snippet: Recent trends in natural language processing research and annotation tasks affirm a paradigm shift from the traditional reliance on a single ground truth to a focus on individual perspectives, particularly in subjective tasks. In scenarios where …


title: CorpusNÓS: A massive Galician corpus for training large language models

authors: I de-Dios-Flores, SP Suárez, CC Pérez, DB Outeiriño… - Proceedings of the 16th …, 2024

snippet: CorpusNÓS is a massive Galician corpus made up of 2.1 B words primarily devised for training large language models. The corpus sources are varied and represent a relatively wide range of genres. CorpusNÓS is, to the best of our knowledge, the …


title: Correcting Auditory Spelling Mistakes in Jordanian Dialect Using Machine Learning Techniques

authors: M Smadi, G Abandah - 2024 15th International Conference on Information and …, 2024

snippet: This paper explores the application of machine learning techniques, specifically leveraging an efficient transformer model known as Byt5, to rectify Arabic auditory spelling mistakes. The model is trained using datasets with synthetic errors …


title: Cost-Effective Big Data Orchestration Using Dagster: A Multi-Platform Approach

authors: H Picatto, G Heiler, P Klimek - arXiv preprint arXiv:2408.11635, 2024

snippet: … The Common Crawl dataset is a particular promising avenue for this kind of research, as it also contains historic data which allows one to … The temporal partitioning aligns with the Common Crawl1 dataset used, facilitating efficient data …


title: Cost-Effective Event Mining on the Web via Event Source Page Discovery and Data API Construction

authors: YH Lin, CH Chang, HM Chuang, XS Lin, T Yeh… - IEEE Access, 2024

snippet: … • Compared to the less than 1% of event pages found in the Common Crawl and Clueweb datasets when crawling the entire web, a significantly higher proportion of events, 13.3%, were identified when specifically targeting the websites of potential …


title: Cost-efficient prompt engineering for unsupervised entity resolution in the product matching domain

authors: N Nananukul, K Sisaengsuwanchai, M Kejriwal - Discover Artificial Intelligence, 2024

snippet: … Web Data Commons (WDC) [52]: The full version of this dataset contains 26 million products and descriptions from different e-commerce websites scraped from the Web as part of the Web Data Commons (Common Crawl). The products are …


title: Counter (media) Intelligence and Visioning: An Interview with Adam Harvey

snippet: … Among the largest and most popular datasets used in AI research projects are Common Crawl (for NLP), and ImageNet and COCO (for computer vision), which are all derived from user-generated content. In my research project Exposing.ai, I’ve …


title: Crafting clarity: Leveraging large language models to decode consumer reviews

authors: SV Praveen, P Gajjar, RK Ray, A Dutt - Journal of Retailing and Consumer Services, 2024

snippet: Large Language Models (LLMs) have emerged as powerful tools for understanding consumer perceptions and extracting insights from unstructured textual data. This study investigates the effectiveness of LLMs in comprehending consumer opinions …


title: Creating Ad Campaigns Using Generative AI Check for updates

authors: A Bulut, B Arslan - Applications of Generative AI

snippet: Search campaigns consist of ad groups. An ad group contains a related set of keywords and ads. During an online campaign, search advertisers experiment with different marketing messages such as subtle vs. strong being used in ad copies, with …


title: Creating Ad Campaigns Using Generative AI

authors: A Bulut, B Arslan - Applications of Generative AI, 2024

snippet: … In particular, T5 and PEGASUS models were trained on 750 GB of English-language text from the public Common Crawl web scrape while the BART model was trained on the CNN/Daily Mail dataset, which contains roughly 300K unique news articles …


title: Creating Parallel Corpora for Ukrainian: a German-Ukrainian Parallel Corpus (ParaRook|| DE-UK)

authors: M Shvedova, A Lukashevskyi - Proceedings of the Third Ukrainian Natural Language …, 2024

snippet: Parallel corpora are currently a popular and vibrantly developing category of linguistic resources, used both in literature and translation studies, as well as in the field of NLP. For Ukrainian, though, there are still not enough significant parallel …


title: CroissantLLM: A Truly Bilingual French-English Language Model

authors: M Faysse, P Fernandes, N Guerreiro, A Loison, D Alves… - arXiv preprint arXiv …, 2024

snippet: We introduce CroissantLLM, a 1.3B language model pretrained on a set of 3T English and French tokens, to bring to the research and industrial community a high-performance, fully open-sourced bilingual model that runs swiftly on consumer-grade local …


title: Cross-cultural Inspiration Detection and Analysis in Real and LLM-generated Social Media Data

authors: O Ignat, GG Lakshmy, R Mihalcea - arXiv preprint arXiv:2404.12933, 2024

snippet: Inspiration is linked to various positive outcomes, such as increased creativity, productivity, and happiness. Although inspiration has great potential, there has been limited effort toward identifying content that is inspiring, as opposed to just engaging …


title: Cross-Language Harmonization of Linguistic Resources

snippet: The presented work consists of two parts. In the first part I summarize the main directions of my research since the defense of my PhD thesis in 2005. I start with cross-language transfer of parsing models to languages that have little or no …


title: Cross-Lingual Cross-Modal Retrieval with Noise-Robust Fine-Tuning

authors: R Cai, J Dong, T Liang, Y Liang, Y Wang, X Yang… - IEEE Transactions on …, 2024

snippet: … from snapshots of the CommonCrawl public dataset2. Experiments in this paper concern parallel data between English and other four languages, namely Chinese (ZH), French (FR), German (DE) and Czech (CZ). Evaluation metrics. … https://commoncrawl.org/ …


title: Cross-lingual Editing in Multilingual Language Models

authors: H Beniwal, M Singh - arXiv preprint arXiv:2401.10521, 2024

snippet: The training of large language models (LLMs) necessitates substantial data and computational resources, and updating outdated LLMs entails significant efforts and resources. While numerous model editing techniques (METs) have emerged to …


title: Cross-lingual Transfer and Multilingual Learning for Detecting Harmful Behaviour in African Under-Resourced Language Dialogue

authors: TO Ajayi, M Arcan, P Buitelaar - Proceedings of the 25th Annual Meeting of the …, 2024

snippet: Most harmful dialogue detection models are developed for high-resourced languages. Consequently, users who speak under-resourced languages cannot fully benefit from these models in terms of usage, development, detection and …


title: Cross-lingual transfer for the annotation of the SynSemClass ontology

snippet: This work compares two approaches to automatic preannotation of semantic class to verbs in a sentence for the purpose of adding a new language to the SynSemClass ontology. Both approaches rely on a multilingual deep learning classification model …


title: Cross-lingual transfer of multilingual models on low resource African Languages

authors: H Thangaraj, A Chenat, JS Walia, V Marivate - arXiv preprint arXiv:2409.10965, 2024

snippet: … 2019) introduce XLM-R, a large-scale multilingual language model trained on 100 languages using two terabytes of CommonCrawl data. XLM-R offers better performance than models such as mBERT, particularly in low-resource languages such as …


title: Current strategies to address data scarcity in artificial intelligence-based drug discovery: A comprehensive review

authors: A Gangwal, A Ansari, I Ahmad, AK Azad… - Computers in Biology and …, 2024

snippet: Artificial intelligence (AI) has played a vital role in computer-aided drug design (CADD). This development has been further accelerated with the increasing use of machine learning (ML), mainly deep learning (DL), and computing hardware and software …


title: Cyber Risks of Machine Translation Critical Errors: Arabic Mental Health Tweets as a Case Study

authors: H Saadany, A Tantawy, C Orasan - arXiv preprint arXiv:2405.11668, 2024

snippet: With the advent of Neural Machine Translation (NMT) systems, the MT output has reached unprecedented accuracy levels which resulted in the ubiquity of MT tools on almost all online platforms with multilingual content. However, NMT systems, like …


title: Cyberbullying Detection Using PCA Extracted GLOVE Features and RoBERTaNet Transformer Learning Model

authors: M Umer, EA Alabdulqader, AA Alarfaj, L Cascone… - IEEE Transactions on …, 2024

snippet: Online platforms are nurturing social interactions, yet regrettably, they have also led to the proliferation of antisocial behaviors such as cyberbullying, trolling, and hate speech on a global scale. The identification of hate speech and aggression has …


title: D3. 2: Report on Citizen discourses and attitudes towards controversies

authors: P Troullinou, I Bonavita, FU TRI, FT ERI, S Stoyanov…

snippet: … Social media listening analysed data from the open web repository, CommonCrawl, for the period 2013-2021. The analysis indicated which … Social listening The findings of the analysis conducted on the CommonCrawl database will …


title: DANSK: Domain Generalization of Danish Named Entity Recognition

authors: K Enevoldsen, ET Jessen, R Baglini - Northern European Journal of Language …, 2024

snippet: Named entity recognition is an important application within Danish NLP, essential within both industry and research. However, Danish NER is inhibited by a lack coverage across domains and entity types. As a consequence, no current models …


title: Data Analysis of Twitter's Nasdaq100 Sentiments and Topics as Indicators for News Articles Retrieval: Fine-Tuning RoBERTa and RAG

snippet: This study explores the combination of sentiment analysis with vader-lexicon and semantic analysis with latent dirichlet allocation to identify real-life events, particularly in the context of Twitter datasets. While sentiment analysis alone may not …


title: Data Analyst Competencies: A Theory-Driven Investigation of Industry Requirements in the Field of Data Analytics

authors: CA Collier, AL Powell - Journal of Information Systems Education, 2024

snippet: … First, the job postings utilized in Dong and Triche’s study were not comprehensive but rather only those captured by the Common Crawl tool to facilitate the longitudinal nature of their study. Second, the wildcard searches used by Dong and Triche …


title: Data Augmentation and Large Language Model for Legal Case Retrieval and Entailment

authors: MQ Bui, DT Do, NK Le, DH Nguyen, KVH Nguyen… - The Review of Socionetwork …, 2024

snippet: The Competition on Legal Information Extraction and Entailment (COLIEE) is a well-known international competition organized each year with the goal of applying machine learning algorithms and techniques in the analysis and understanding of legal …


title: Data augmentation for language generation inspired by machine translation

snippet: The field of natural language processing has witnessed a surge in the adoption of deep learning, which faces notable hurdles when the training data is scarce. This thesis aims to study automatic data augmentation for language generation tasks …


title: Data Augmentation with Semantic Enrichment for Deep Learning Invoice Text Classification

authors: WW Chi, TY Tang, NM Salleh, M Mukred, H AlSalman… - IEEE Access, 2024

snippet: Natural language processing (NLP) is a research field that provides huge potential to automate accounting tasks dealing with text data. This research studies the application of NLP in automatically categorizing invoices based on the invoice text …


title: Data Authenticity, Consent, & Provenance for AI are all broken: what will it take to fix them?

authors: S Longpre, R Mahari, N Obeng-Marnu, W Brannon… - arXiv preprint arXiv …, 2024

snippet: … Common Crawl is accurate with wide coverage, but not detailed. Hugging Face can be inaccurate with varying levels of detail, but it has extensive coverage. The Data Provenance Initiative is highly accurate and detailed but is currently limited in scope. …


title: Data Collection Pipeline for Low-Resource Languages: A Case Study on Constructing a Tetun Text Corpus

authors: G de Jesus, SS Nunes - Proceedings of the 2024 Joint International Conference …, 2024

snippet: This paper proposes Labadain Crawler, a data collection pipeline tailored to automate and optimize the process of constructing textual corpora from the web, with a specific target to low-resource languages. The system is built on top of Nutch, an …


title: Data Contamination Report from the 2024 CONDA Shared Task

authors: O Sainz, I García-Ferrero, A Jacovi, JA Campos… - arXiv preprint arXiv …, 2024

snippet: … The reported corpora are mainly based on CommonCrawl snapshots, GitHub, or a mix of sources. For CommonCrawlbased corpora, there are 35 events reported for C4 (Raffel et al.… Ortiz Su’arez et al., 2019) and 6 for CommonCrawl (Rana…


title: Data Engineering for Scaling Language Models to 128K Context

authors: Y Fu, R Panda, X Niu, X Yue, H Hajishirzi, Y Kim… - arXiv preprint arXiv …, 2024

snippet: We study the continual pretraining recipe for scaling language models' context lengths to 128K, with a focus on data engineering. We hypothesize that long context modeling, in particular \textit{the ability to utilize information at arbitrary input …


title: Data Mixing Made Efficient: A Bivariate Scaling Law for Language Model Pretraining

authors: C Ge, Z Ma, D Chen, Y Li, B Ding - arXiv preprint arXiv:2405.14908, 2024

snippet: Large language models exhibit exceptional generalization capabilities, primarily attributed to the utilization of diversely sourced data. However, conventional practices in integrating this diverse data heavily rely on heuristic schemes, lacking …


title: Data Mixture Inference Attack: BPE Tokenizers Reveal Training Data Compositions

authors: J Hayase, A Liu, Y Choi, S Oh, NA Smith - ICML 2024 Workshop on Foundation Models in …

snippet: … 2022), which is based on the Nov/Dec 2022 dump from Common Crawl. We consider the 112 languages with at least 1MB of data. … , containing English Wikipedia dumps from Jun-Aug 2022, Web, Common Crawl data that was de-duplicated …


title: Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?

authors: J Hayase, A Liu, Y Choi, S Oh, NA Smith - arXiv preprint arXiv:2407.16607, 2024

snippet: … on the Nov/Dec 2022 dump from Common Crawl. We consider the 112 languages with at least 1 MB of data. … The training data is reportedly sourced from Common Crawl … November/December 2022 dump of Common Crawl. We only …


title: Data Selection for Generalization in Unimodal & Multimodal Models

snippet: In this thesis, I present research on improving datasets for deep learning models with automateddata transformation methods. The immediate goal of this work is to maximize the in-domainperformance, out-of-domain generalization, and robustness …


title: Data, Data Everywhere: A Guide for Pretraining Dataset Construction

authors: J Parmar, S Prabhumoye, J Jennings, B Liu… - arXiv preprint arXiv …, 2024

snippet: … We close this gap by conducting a large-scale analysis on over 90 Common Crawl web snapshots for the attributes of domain, quality, toxicity, and type of speech. We then show how such data attributes can aid in pretraining set construction to …


title: Data-Centric Methods for Decentralizing Large Language Models

snippet: … from large web crawl services like Common Crawl. Web crawl is perceived to be wholly representative of activity on the Internet [Brügger… Indeed, web dumps like Common Crawl offer the promise of more diverse text than what is available in …


title: Data-to-Text Generation with Neural Language Models

snippet: Data-to-text generation systems need to produce texts with high levels of semantic accuracy. Rule-based systems can guarantee this aspect, but their fluency and adaptability to new domains remain limited. Meanwhile, neural language models …


title: DataComp Challenge

snippet: … billion sample image-text dataset filtered from a comparable pool of samples to Common Crawl. For the creation of the LAION-2B dataset, first language filtering is applied to the original Common Crawl pool so that only samples with English texts …


title: DataComp-LM: In search of the next generation of training sets for language models

authors: J Li, A Fang, G Smyrnis, M Ivgi, M Jordan, S Gadre… - arXiv preprint arXiv …, 2024

snippet: … As part of DCLM, we provide a standardized corpus of 240T tokens extracted from Common Crawl, effective pretraining recipes based on the OpenLM framework, and a broad suite of 53 downstream evaluations. Participants in the DCLM benchmark …


title: DataSculpt: Crafting Data Landscapes for LLM Post-Training through Multi-objective Partitioning

authors: K Lu, Z Liang, X Nie, D Pan, S Zhang, K Zhao, W Chen… - arXiv preprint arXiv …, 2024

snippet: … For instance, less than 5% of documents within CommonCrawl exceed 2k tokens in length [8], which hinders LLMs’ exposure to complex narratives. Despite these challenges, significant strides have been made toward enhancing LLMs’ capabilities …


title: Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum

authors: H Pouransari, CL Li, JHR Chang, PKA Vasu, C Koc… - arXiv preprint arXiv …, 2024

snippet: … Although the benefits of ICLM with large-scale Common Crawl data (used in our experiments) are marginal in regular evaluation, it significantly boosts long-context evaluation metrics from 27.5 to 28.7 by constructing long data sequences through …


title: Dataset Growth

authors: Z Qin, Z Xu, Y Zhou, Z Zheng, Z Cheng, H Tang… - arXiv preprint arXiv …, 2024

snippet: Deep learning benefits from the growing abundance of available data. Meanwhile, efficiently dealing with the growing data scale has become a challenge. Data publicly available are from different sources with various qualities, and it is …


title: Datasets for Large Language Models: A Comprehensive Survey

authors: Y Liu, J Cao, C Liu, K Ding, L Jin - arXiv preprint arXiv:2402.18041, 2024

snippet: … The first method involves building upon Common Crawl1. Common Crawl is a massive, unstructured, multilingual web corpus that provides public access to web archives by regularly crawling and storing webpage data from the Internet. However …


title: Dataverse: Open-Source ETL (Extract, Transform, Load) Pipeline for Large Language Models

authors: H Park, S Lee, G Gim, Y Kim, D Kim, C Park - arXiv preprint arXiv:2403.19340, 2024

snippet: To address the challenges associated with data processing at scale, we propose Dataverse, a unified open-source Extract-Transform-Load (ETL) pipeline for large language models (LLMs) with a user-friendly design at its core. Easy addition of …


title: Dated Data: Tracing Knowledge Cutoffs in Large Language Models

authors: J Cheng, M Marone, O Weller, D Lawrie, D Khashabi… - arXiv preprint arXiv …, 2024

snippet: … Our analysis reveals two reasons for these inconsistencies: (1) temporal biases of CommonCrawl data due to non-trivial amounts of old data in new dumps and (2) complications in LLM deduplication schemes involving semantic duplicates and …


title: DDK: Distilling Domain Knowledge for Efficient Large Language Models

authors: J Liu, C Zhang, J Guo, Y Zhang, H Que, K Deng, Z Bai… - arXiv preprint arXiv …, 2024

snippet: Despite the advanced intelligence abilities of large language models (LLMs) in various applications, they still face significant computational and storage demands. Knowledge Distillation (KD) has emerged as an effective strategy to improve the …


title: DE-COP: Detecting Copyrighted Content in Language Models Training Data

authors: AV Duarte, X Zhao, AL Oliveira, L Li - arXiv preprint arXiv:2402.09910, 2024

snippet: How can we detect if copyrighted content was used in the training process of a language model, considering that the training data is typically undisclosed? We are motivated by the premise that a language model is likely to identify verbatim …


title: Dear GPT-3: Collaborative Writing with Neural Networks

authors: J Becker - Artificial Intelligence-Intelligent Art?: Human-Machine …, 2024

snippet: GPT-3 and I engaged in this dialogue about four months ago. I wanted to develop a collaborative writing project, not only to collectively realise a novel, but also to negotiate poetological questions which might arise during the process. These …


title: DeBERTa-BiLSTM: A multi-label classification model of arabic medical questions using pre-trained models and deep learning

authors: BS Al-Smadi - Computers in Biology and Medicine, 2024

snippet: … Using data from CommonCrawl in 100 different languages, the authors in [38] introduce XLM-RoBERTa, an advanced masked multilingual language model. XLM-RoBERTa outperformed previous multilingual models such as XLM and mBERT. [39] …


title: Deciphering the Role of Representation Disentanglement: Investigating Compositional Generalization in CLIP Models

authors: R Abbasi, MH Rohban, MS Baghshah - arXiv preprint arXiv:2407.05897, 2024

snippet: … In our experiments, we evaluate CLIP models trained on a diverse selection of datasets, including OpenAI’s private dataset, LAION, YFCC15m, CC12m, DataComp, DFN-5B, WebLI, and CommonCrawl. These models leverage a variety of backbone …


title: Decoding the Diversity: A Review of the Indic AI Research Landscape

authors: S KJ, V Jain, S Bhaduri, T Roy, A Chadha - arXiv preprint arXiv:2406.09559, 2024

snippet: This review paper provides a comprehensive overview of large language model (LLM) research directions within Indic languages. Indic languages are those spoken in the Indian subcontinent, including India, Pakistan, Bangladesh, Sri Lanka, Nepal, and …


title: Deconstructing AI literacy for librarians

snippet: In this presentation, Samantha shares ways we can think about AI literacy-how it is similar and dissimilar from traditional information literacy, what we need to know about the mechanics of AI tools, and how SMU Libraries support the different users …


title: Deconstructing In-Context Learning: Understanding Prompts via Corruption

authors: N Shivagunde, V Lialin, S Muckatira, A Rumshisky - arXiv preprint arXiv:2404.02054, 2024

snippet: The ability of large language models (LLMs) to "learn in context" based on the provided prompt has led to an explosive growth in their use, culminating in the proliferation of AI assistants such as ChatGPT, Claude, and Bard. These AI …


title: Decoupled Vocabulary Learning Enables Zero-Shot Translation from Unseen Languages

snippet: Multilingual neural machine translation systems learn to map sentences of different languages into a common representation space. Intuitively, with a growing number of seen languages the encoder sentence representation grows more flexible and …


title: Deep learning aided clinical decision support

snippet: Medical professionals create vast amounts of clinical texts during patient care. Often, these documents describe medical cases from anamnesis to the final clinical outcome. Automated understanding and selection of relevant medical records pose …


title: Deep Learning Based Multi-document Summarization

snippet: In this era of rapidly advancing technology, the exponential increase of data availability makes analyzing and understanding text files a tedious, labor-intensive, and time-consuming task. Multi-document summarization (MDS) is an effective tool …


title: Deep Learning based Visually Rich Document Content Understanding: A Survey

authors: Y Ding, J Lee, SC Han - arXiv preprint arXiv:2408.01287, 2024

snippet: Visually Rich Documents (VRDs) are essential in academia, finance, medical fields, and marketing due to their multimodal information content. Traditional methods for extracting information from VRDs depend on expert knowledge and manual labor …


title: Deep Learning for Cross-Domain Data Fusion in Urban Computing: Taxonomy, Advances, and Outlook

authors: X Zou, Y Yan, X Hao, Y Hu, H Wen, E Liu, J Zhang, Y Li… - arXiv preprint arXiv …, 2024

snippet: As cities continue to burgeon, Urban Computing emerges as a pivotal discipline for sustainable development by harnessing the power of cross-domain data fusion from diverse sources (eg, geographical, traffic, social media, and environmental data) …


title: Deep Learning for Economists

authors: M Dell - arXiv preprint arXiv:2407.15339, 2024

snippet: Deep learning provides powerful methods to impute structured information from large-scale, unstructured text and image datasets. For example, economists might wish to detect the presence of economic activity in satellite images, or to measure the topics or …


title: Deep Learning Impacts in the Field of Artificial Intelligence

authors: R Gulwani, M Aggarwal - Deep Learning Concepts in Operations Research, 2024

snippet: This book chapter explores the profound impacts of deep learning on the field of artificial intelligence (AI). Deep learning has revolutionized various domains within AI, enabling significant advancements in computer vision, natural language processing …


title: Deep Learning Model for Tamil Part-of-Speech Tagging

authors: H Visuwalingam, R Sakuntharaj, J Alawatugoda… - The Computer Journal, 2024

snippet: … These pre-trained word embeddings are trained on Common Crawl and Wikipedia. This model was trained using CBOW with position-weights, in dimension 300, with character n-grams of length 5, a window of size 5 and 10 negatives. …


title: DeepSeek LLM: Scaling Open-Source Language Models with Longtermism

authors: X Bi, D Chen, G Chen, S Chen, D Dai, C Deng, H Ding… - arXiv preprint arXiv …, 2024

snippet: … Our analysis revealed that deduplicating the entire Common Crawl corpus results in higher removal of duplicate instances compared to deduplicating within a single dump. Table 1 illustrates that deduplicating across 91 dumps eliminates four times …


title: DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence

authors: Q Zhu, D Guo, Z Shao, D Yang, P Wang, R Xu, Y Wu… - arXiv preprint arXiv …, 2024

snippet: … For the math corpus, we collect 221B math-related tokens sourced from CommonCrawl using the same pipeline, which approximately doubles the size of the 120B DeepSeekMath corpus (Shao et al., 2024), while for the natural language …


title: DeepSeek-VL: Towards Real-World Vision-Language Understanding

authors: H Lu, W Liu, B Zhang, B Wang, K Dong, B Liu, J Sun… - arXiv preprint arXiv …, 2024

snippet: We present DeepSeek-VL, an open-source Vision-Language (VL) Model designed for real-world vision and language understanding applications. Our approach is structured around three key dimensions: We strive to ensure our data is diverse …


title: DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

authors: Z Shao, P Wang, Q Zhu, R Xu, J Song, M Zhang, YK Li… - arXiv preprint arXiv …, 2024

snippet: … the DeepSeekMath Corpus from Common Crawl. As depicted in Figure 2, … Common Crawl, we employ URL-based deduplication and near-deduplication techniques, resulting in 40B HTML web pages. We then recall mathematical web …


title: Defining Boundaries: The Impact of Domain Specification on Cross-Language and Cross-Domain Transfer in Machine Translation

authors: L Shahnazaryan, M Beloucif - arXiv preprint arXiv:2408.11926, 2024

snippet: … 2021) and mined from the Common Crawl corpus.While all languages used in our experiments, except for Italian, are considered bridge languages in the M2M-100 model, the lack of information on the exact data sizes mined for each language limits …


title: Deliverable 3.1: WP3 1st Interim technical report

snippet: 1.1. Background This document is the first Interim Technical Report (deliverable 3.1) prepared within the WP3 New usecases for the ESSnet Trusted Smart Statistics–Web Intelligence Network project (ESSnet TSS-WIN). The report covers the period from …


title: Deriving the Meaning of Out-of-Vocabulary Words

snippet: … It consists of the articles that were collected using CommonCrawl[34]. The articles originated from five Czech news portals: … It is trained on more than 2TB of data from the filtered CommonCrawl[34] containing 100 languages. They report that the …


title: Design of Artificial Intelligence Companion Chatbot

snippet: With the development of cities and the prevalence of networks, interpersonal relationships have become increasingly distant. When people crave communication, they hope to find someone to confide in. With the rapid advancement of deep …


title: Design, Implementation and Evaluation of a Chatbot for Accounting Firm: A Fine-Tuning Approach With Two Novel Dataset

snippet: Artificial intelligence, particularly in the field of Chatbots, is fundamentally reshaping learning, communication, and work paradigms. This phenomenon has sparked growing interest among businesses, viewing Chatbots as a means to streamline …


title: Designing a course for pre-service science teachers using ChatGPT: what ChatGPT brings to the table

authors: HZ Okulu, N Muslu - Interactive Learning Environments, 2024

snippet: ChatGPT holds significant potential for enhancing learning through integration into education as an advanced chatbot. With the goal of harnessing this potential, our research focused on exploring the utilization of ChatGPT in designing a course plan …


title: Designing an Intelligent System to Map Global Connections

authors: E Bellamy, K Farrell, A Hopping, J Pinter, M Saju… - 2024

snippet: … Extended Abstract: This study develops a knowledge graph from the Common Crawl News Dataset to provide situational awareness and … We developed a data pipeline to extract semantic content from the Common Crawl News feed and filter it …


title: Designing and developing a dedicated Natural Language Processing Framework for Healthcare Information Technology Management and Assessment

snippet: The escalating complexity of the hospital environment, propelled by technological advancements, necessitates a comprehensive exploration of the integration and management of diverse tools and technologies in healthcare settings. In this context …


title: Designing Heterogeneous LLM Agents for Financial Sentiment Analysis

authors: F Xing - arXiv preprint arXiv:2401.05799, 2024

snippet: … GPT-3.5 was trained mainly on the Common Crawl corpus [2], which archives the web. BLOOMZ was trained on an even larger Open-science Open-collaboration Text Sources corpus [19], which is mainly crowd-sourced scientific datasets. The five …


title: Detecting Dementia from Transcribed Speech in Slovak using Pre-trained BERT Models

authors: J Staš, D Hládek, A Kopnický - 2024 34th International Conference Radioelektronika …, 2024

snippet: … TB of texts written in 100 languages from the Common Crawl dataset [14]; • RemBERT model6: a Rebalanced multilingual BERT model pre-trained on a large unlabeled text created by the combination of mC4 Common Crawl and Wikipedia …


title: Detecting Depression and Anxiety on Social Media Using Selective Masking

snippet: Mental health problems are one of the major problems in the world. It is estimated that once in their life, at least one mental health condition will affect one in four people. Depression is the most common condition, with 5\% of adults suffering from …


title: Detecting Hidden Meaning in Stock Images

snippet: OBJECTIVE − Automated extraction of hidden meaning in stock images− Is it possible to distinguish what is shown from what is meant?− Examine the divergence of text and image− Analyze the usage of stock images and textual descriptions on the web …


title: Detecting Machine-Generated News Using Fine-Tuned Transformers

snippet: … The dataset for this research was a collection of pre-generated GPT-2 news articles, along with real news articles gathered from the internet using Common Crawl. This data combined with the latest popular LLMs such as Llama, Gemma …


title: Detection of adversarial phishing attack using machine learning techniques

authors: KM Sudar, M Rohan, K Vignesh - Sādhanā, 2024

snippet: The frequency of cyberattacks, particularly phishing attacks, is increasing exponentially every day. Many users fall victim to clicking on malicious URLs, leading to the exploitation of their information. The traditional methodologies …


title: Detection of bullying with MachineLearning: Using Supervised Machine Learning and LLMs to classify bullying in text

snippet: In recent years, there has been a increase in the issue of bullying, particularly in academic settings [1]. This degree project examines the use of supervised machine learning techniques to identify bullying in text data from school surveys provided by …


title: Detection of Hate Speech and Offensive Language CodeMix Text in Dravidian Languages using Cost-Sensitive Learning Approach

authors: K Sreelakshmi, B Premjith, BR Chakravarthi… - IEEE Access, 2024

snippet: … It was trained using more than two terabytes of filtered CommonCrawl data. It has significantly improved performance on various cross-lingual transfer tasks and outperformed the mBERT model. XLM-R has 24 layers and 16 attention heads …


title: Detection of phishing addresses and pages with a data set balancing approach by generative adversarial network (GAN) and convolutional neural network (CNN) …

authors: S Jafari, N Aghaee‐Maybodi - Concurrency and Computation: Practice and …

snippet: Phishing attacks have a remarkable ability to steal user information by using simple techniques. Phishing attacks steal valuable information, such as user names and passwords. The loss caused by phishing attacks is significant, and every year …


title: Detection of Sarcasm in Urdu Tweets using Deep Learning and Transformer based Hybrid Approaches

authors: ME Hassan, M Hussain, I Maab, U Habib, MA Khan… - IEEE Access, 2024

snippet: Sarcasm has a significant role in human communication especially on social media platforms where users express their sentiments through humor, satire, and criticism. The identification of sarcasm is crucial in comprehending the sentiment and the …


title: Detection of Twitter Spam Using GLoVe Vocabulary Features, Bidirectional LSTM and Convolution Neural Network

authors: P Manasa, A Malik, I Batra - SN Computer Science, 2024

snippet: … The training of the GloVe model involves a substantial text corpus, like Wikipedia or Common Crawl, which facilitates the learning of connections between words and phrases. For every word in the corpus, the model creates a vector, wherein each …


title: Developing and Deploying AI Applications on NVIDIA Jetson Orin NX: A Comprehensive Guide

snippet: The NVIDIA Jetson Orin NX is a cutting-edge platform that brings powerful AI capabilities to the edge, enabling a wide range of applications from real-time computer vision to advanced robotics. This comprehensive guide provides detailed …


title: Developing ChatGPT for Biology and Medicine: A Complete Review of Biomedical Question Answering

authors: Q Li, L Li, Y Li - arXiv preprint arXiv:2401.07510, 2024

snippet: ChatGPT explores a strategic blueprint of question answering (QA) in delivering medical diagnosis, treatment recommendations, and other healthcare support. This is achieved through the increasing incorporation of medical domain data via natural …


title: Development and Evaluation of a German Language Model for the Financial Domain

authors: N Kozaeva, S Hamotskyi, C Hänig - Proceedings of the Joint Workshop of the 7th …, 2024

snippet: … The German colossal, cleaned Common Crawl corpus15 was employed, comprising texts of varying lengths. Further the results of LM performance for mixed datasets (financial corpus mixed with common language sentences and financial …


title: Development and Evaluation of Pre-trained Language Models for Historical Danish and Norwegian Literary Texts

authors: A Al-Laith, A Conroy, J Bjerring-Hansen… - Proceedings of the 2024 …, 2024

snippet: We develop and evaluate the first pre-trained language models specifically tailored for historical Danish and Norwegian texts. Three models are trained on a corpus of 19th-century Danish and Norwegian literature: two directly on the corpus with no …


title: Development of a Geographical Question-Answering System in the Kazakh Language

authors: A Mukanova, A Barlybayev, A Nazyrova, L Kussepova… - IEEE Access, 2024

snippet: … Among the pre-trained models, the KazBERT model, trained on cleansed CommonCrawl data in the Kazakh language, achieved the most favorable outcomes. For the test dataset, the KazBERT model demonstrated an Exact Match (EM) …


title: Development of an Adaptive Multi-Domain Artificial Intelligence System Built using Machine Learning and Expert Systems Technologies

authors: J Straub - arXiv preprint arXiv:2406.11272, 2024

snippet: Producing an artificial general intelligence (AGI) has been an elusive goal in artificial intelligence (AI) research for some time. An AGI would have the capability, like a human, to be exposed to a new problem domain, learn about it and then use …


title: Development of an Interface for Multi-Dimensional Clinical Language Analysis of Dementia and Assessment of its Market Viability

snippet: … This model, from Microsoft, is initialized from the transformer model XLM-RoBERTa-Large, which is pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages [24]. CommonCrawl data consists of a comprehensive corpus of web …


title: Development of Natural Language Processing Tools and Resources for Assamese Text

snippet: It is my immense pleasure to convey my heartfelt gratitude to my supervisors, Prof. Sukumar Nandi and Prof. Priyankoo Sarmah, for their unwavering support and guidance during my research work. I am grateful to both of my supervisors for …


title: Diagonalization & Forcing FLEX: From Cantor to Cohen and Beyond

snippet: Background: Maybe very high energy cosmic radiation started a synaptic butterfly cascade 2... who knows? For one reason or another, I decided to embark on a journey of exploring and discussing the possible role of Artificial Intelligence (AI) on …


title: Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning

authors: XY Liu, R Zhu, D Zha, J Gao, S Zhong, M Qiu - arXiv preprint arXiv:2312.17493, 2023

snippet: The surge in interest and application of large language models (LLMs) has sparked a drive to fine-tune these models to suit specific applications, such as finance and medical science. However, concerns regarding data privacy have emerged …


title: Diffused Seeing: The Epistemological Challenge

snippet: This article examines the transformation of the relationship between seeing and understanding in humans and machines by the technologies of machine learning known as ‘generative AI’. Taking Stable Diffusion as the main case study, while also …


title: Digilog: Enhancing Website Embedding on Local Governments-A Comparative Analysis

authors: J Gerber, B Kreiner, J Saxer, A Weiler - International Symposium on Methodologies …, 2024

snippet: The ability to understand and process websites, known as website embedding, is crucial across various domains. It lays the foundation for machine understanding of websites. Specifically, website embedding proves invaluable when monitoring local …


title: Digital Protection and Innovative Development Path of Red Culture Resources Based on Distributed Machine Learning Supported by Intelligent Information

authors: M Huang, X Zeng - Journal of Combinatorial Mathematics and …

snippet: As a product of the revolutionary war years, red culture, with its strong vitality, strong cohesion and extraordinary charm, with its incomparable positive energy, resists vulgar and flattering culture, promotes people to rebuild their faith, purify their minds …


title: Digital Threats

authors: S Samtani, E Raff, H Anderson, E Domschot… - 2024

snippet: … First, we used the common crawl GloVe embedding, created by Stanford [28]. This embedding was trained over 42 billion tokens, with a 300-dimensional vector space, and was used as the embedding layer of the BiLSTM model. Results are …


title: DiPaCo: Distributed Path Composition

authors: A Douillard, Q Feng, AA Rusu, A Kuncoro, Y Donchev… - arXiv preprint arXiv …, 2024

snippet: Progress in machine learning (ML) has been fueled by scaling neural network models. This scaling has been enabled by ever more heroic feats of engineering, necessary for accommodating ML approaches that require high bandwidth …


title: Disambiguating Homographs and Homophones Simultaneously: A Regrouping Method for Japanese

authors: Y Sato - Proceedings of the 2024 Joint International Conference …, 2024

snippet: We present a method that re-groups surface forms into clusters representing synonyms, and help disambiguate homographs as well as homophone. The method is applied post-hoc to trained contextual word embeddings. It is beneficial to …


title: Disambiguating natural language via aligning meaningful descriptions

snippet: Artificial Intelligence (AI) technologies are increasingly pervading aspects of our lives. Because people use natural language to communicate with each other, computers should also use natural language to communicate with us. One of the principal …


title: Display options

snippet: Chemical patents are a commonly used channel for disclosing novel compounds and reactions, and hence represent important resources for chemical and pharmaceutical research. Key chemical data in patents is often presented in tables …


title: Dissecting Whiteness: consistencies and differences in the stereotypes of lower-and upper-class White US Americans

authors: T Morgenroth, CT Begeny, TA Kirby, B Paaßen, Y Zeng - Self and Identity, 2024

snippet: Economic inequality is increasing in the United States, making categorization and stereotyping based on social class more likely. Yet, social class stereotypes have received relatively little attention. Focusing on spontaneously generated stereotypes …


title: Distance Comparison Operators for Approximate Nearest Neighbor Search: Exploration and Benchmark

authors: Z Wang, H Xiong, Z He, P Wang - arXiv preprint arXiv:2403.13491, 2024

snippet: Approximate nearest neighbor search (ANNS) on high-dimensional vectors has become a fundamental and essential component in various machine learning tasks. Prior research has shown that the distance comparison operation is the bottleneck of …


title: Distortions in Judged Spatial Relations in Large Language Models

authors: N Fulman, A Memduhoğlu, A Zipf - The Professional Geographer, 2024

snippet: We present a benchmark for assessing the capability of large language models (LLMs) to discern intercardinal directions between geographic locations and apply it to three prominent LLMs: GPT-3.5, GPT-4, and Llama-2. This benchmark specifically …


title: Do Language Models Care About Text Quality? Evaluating Web-Crawled Corpora Across 11 Languages

authors: R van Noord, T Kuzman, P Rupnik, N Ljubešić… - arXiv preprint arXiv …, 2024

snippet: … rest of corpora compared in this paper, MaCoCu corpora are not obtained by processing Common Crawl data. Instead, a strategy consisting of crawling relevant internet top-level domains directly for the targeted languages is followed (eg, .al for …


title: Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarks

snippet: The concept of localization in LLMs is often mentioned in prior work; however, methods for localization have never been systematically and directly evaluated. We propose two complementary benchmarks that evaluate the ability of localization …


title: Do Multilingual Large Language Models Mitigate Stereotype Bias?

authors: S Nie, M Fromm, C Welch, R Görge, A Karimi, J Plepi… - arXiv preprint arXiv …, 2024

snippet: While preliminary findings indicate that multilingual LLMs exhibit reduced bias compared to monolingual ones, a comprehensive understanding of the effect of multilingual training on bias mitigation, is lacking. This study addresses this gap by …


title: Do not shut up and do dribble: social media and TV consumption

authors: M Pazzona, N Spagnolo - Journal of Population Economics, 2024

snippet: … Footnote 11 The NLP model utilized in this study, XLM-RoBERTa, undergoes pre-training on 2.5TB of filtered CommonCrawl data encompassing 100 languages. For tweet classification, we employed the general-purpose Python library TweetNLP (Camacho-Collados …


title: Document-Level Machine Translation with Large-Scale Public Parallel Corpora

snippet: Despite the fact that document-level machine translation has inherent advantages over sentence-level machine translation due to additional information available to a model from document context, most translation systems continue to operate at a …


title: Documenting Geographically and Contextually Diverse Language Data Sources

authors: A McMillan-Major, F De Toni, Z Alyafeai, S Biderman… - Northern European Journal …, 2024

snippet: … Typically, this data is collected from online sources, ranging from highly edited and structured text such as Wikipedia to the myriad text and audiovisual components of web pages, eg, collected by the Common Crawl Founda… Data from Common …


title: Does Context Help Mitigate Gender Bias in Neural Machine Translation?

authors: H Gete, T Etchegoyhen - arXiv preprint arXiv:2406.12364, 2024

snippet: Neural Machine Translation models tend to perpetuate gender bias present in their training data distribution. Context-aware models have been previously suggested as a means to mitigate this type of bias. In this work, we examine this claim by …


title: Does Lack of Knowledge and Hardship of Information Access Signify Powerful AI? A Large Language Model Perspective

authors: IA Zahid, SS Joudar - Applied Data Science and Analysis, 2023

snippet: Large Language Models (LLMs) are evolving and expanding enormously. With the consistent improvement of LLMs, more complex and sophisticated tasks will be tackled. Handling various tasks and fulfilling different queries will be more precise …


title: Does sentiment help in asset pricing? A novel approach using large language models and market-based labels

authors: F Audrino, J Schüttler, F Sigrist - 2024

snippet: We present a novel approach to sentiment analysis in financial markets by using a state-ofthe-art large language model, a market data-driven labeling approach, and a large dataset consisting of diverse financial text sources including earnings call …


title: Does the Language Matter? Curriculum Learning over Neo-Latin Languages

authors: G Pucci, L Ranaldi - Proceedings of the 2024 Joint International Conference …, 2024

snippet: Curriculum Learning (CL) is emerging as a relevant technique to reduce the cost of pre-training Large Language Models. The idea, tested for the English language, is to train LLMs by organizing training examples from the simplest to the most complex …


title: Does your data spark joy? Performance gains from domain upsampling at the end of training

authors: C Blakeney, M Paul, BW Larsen, S Owen, J Frankle - arXiv e-prints, 2024

snippet: Pretraining datasets for large language models (LLMs) have grown to trillions of tokens composed of large amounts of CommonCrawl (CC) web scrape along with smaller, domain-specific datasets. It is expensive to understand the impact of these …


title: Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research

authors: L Soldaini, R Kinney, A Bhagia, D Schwenk, D Atkinson… - arXiv preprint arXiv …, 2024

snippet: … In total, we find 0.02% of documents in the 25 Common Crawl snapshots match this filter. … Head, Middle, and Tail parts of our Common Crawl data. The correlation is computed for 24M, … The correlation among the documents flagged for removal …


title: Domain-Specific Pretraining of Language Models: A Comparative Study in the Medical Field

authors: T Kerner - arXiv preprint arXiv:2407.14076, 2024

snippet: … CommonCrawl, or collecting multiple datasets that are at least slightly related to our domain. Large-scale webscraping is a timeand resource-intensive task, so it is recommended to consider using data from CommonCrawl … [3] Assuming we chose …


title: DomURLs_BERT: Pre-trained BERT-based Model for Malicious Domains and URLs Detection and Classification

authors: AE Mahdaouy, S Lamsiyah, MJ Idrissi, H Alami… - arXiv preprint arXiv …, 2024

snippet: … This is a cleaned version of the Common Crawl’s web corpus, curated by the Allen Institute for Artificial Intelligence [38], containing … This dataset is compiled from CommonCrawl, using strict filtering and extensive deduplication [39], and …


title: Downstream bias mitigation is all you need

authors: A Baksi, R Singh, T Joshi - arXiv preprint arXiv:2408.00612, 2024

snippet: The advent of transformer-based architectures and large language models (LLMs) have significantly advanced the performance of natural language processing (NLP) models. Since these LLMs are trained on huge corpuses of data from the web and …


title: DPHANet: Discriminative Parallel and Hierarchical Attention Network for Natural Language Video Localization

authors: R Chen, J Tan, Z Yang, X Yang, Q Dai, Y Cheng, L Lin - IEEE Transactions on …, 2024

snippet: Natural Language Video Localization (NLVL) has recently attracted much attention because of its practical significance. However, the existing methods still face the following challenges: 1) When the models learn intra-modal semantic association …


title: Dr. Nahid Ebrahimi Majd

snippet: The issue of web phishing attacks has been more prevalent in recent years, and phishing is one of the riskiest online crimes that can have disastrous consequences. The aim of phishing is to collect confidential information by tricking a target and …


title: DrawL: Understanding the Effects of Non-Mainstream Dialects in Prompted Image Generation

authors: JN Williams, M FitzMorris, O Aka, S Laszlo - arXiv preprint arXiv:2405.05382, 2024

snippet: … Stable Diffusion is trained in part using the LAION-5B dataset [50], which consists of text-image pairs from Common CrawlCommon Crawl includes archived data from a large variety of sources, including sites such as Reddit and a variety of blogs …


title: DrBenchmark: A Large Language Understanding Evaluation Benchmark for French Biomedical Domain

authors: Y Labrak, A Bazoge, OE Khettari, M Rouvier, N Grabar… - arXiv preprint arXiv …, 2024

snippet: The biomedical domain has sparked a significant interest in the field of Natural Language Processing (NLP), which has seen substantial advancements with pre-trained language models (PLMs). However, comparing these models has proven …


title: Dredge Word, Social Media, and Webgraph Networks for Unreliable Website Classification and Identification

authors: EM Williams, P Carragher, KM Carley - arXiv preprint arXiv:2406.11423, 2024

snippet: In an attempt to mimic the complex paths through which unreliable content spreads between search engines and social media, we explore the impact of incorporating both webgraph and large-scale social media contexts into website credibility …


title: DsDm: Model-Aware Dataset Selection with Datamodels

authors: L Engstrom, A Feldmann, A Madry - arXiv preprint arXiv:2401.12926, 2024

snippet: When selecting data for training large-scale models, standard practice is to filter for examples that match human notions of data quality. Such filtering yields qualitatively clean datapoints that intuitively should improve model behavior. However, in …


title: Dual Modalities of Text: Visual and Textual Generative Pre-training

authors: Y Chai, Q Liu, J Xiao, S Wang, Y Sun, H Wu - arXiv preprint arXiv:2404.10710, 2024

snippet: … Meanwhile, The C4 dataset represents a substantial refinement of the Common Crawl corpus. This dataset, derived from the extensive Common Crawl web scrape, undergoes rigorous cleaning and preprocessing to ensure the quality and relevance …


title: Duth at semeval 2024 task 8: Comparing classic machine learning algorithms and llm based methods for multigenerator, multidomain and multilingual machine …

authors: T Kyriakou, I Maslaris, A Arampatzis - Proceedings of the 18th International Workshop …, 2024

snippet: Text-generative models evolve rapidly nowadays. Although, they are very useful tools for a lot of people, they have also raised concerns for different reasons. This paper presents our work for SemEval2024 Task-8 on 2 out of the 3 subtasks. This …


title: Dynamic decoding and dual synthetic data for automatic correction of grammar in low-resource scenario

authors: A Musyafa, Y Gao, A Solyman, S Khan, W Cai… - PeerJ Computer Science, 2024

snippet: … The data was collected from January–December 2018 using Commoncrawl snapshots in the CC-Net repository and organized into a single text document with a total data size of 36 GB. We selected the CC100 Indonesian corpus due to its open …