title: 1-Diffractor: Efficient and Utility-Preserving Text Obfuscation Leveraging Word-Level Metric Differential Privacy

authors: S Meisenbacher, M Chevli, F Matthes - arXiv preprint arXiv:2405.01678, 2024

snippet: The study of privacy-preserving Natural Language Processing (NLP) has gained rising attention in recent years. One promising avenue studies the integration of Differential Privacy in NLP, which has brought about innovative methods in a variety …


title: 101 Billion Arabic Tokens Dataset

authors: M Aloui, H Chouikhi, G Chaabane, H Kchaou…

snippet: … , we heavily depend on Common Crawl (1) as our primary source of web-scraped content. Common Crawl provides an accessible web archive … a Dataset Pipeline expressed in the following figure (1) aimed at cleaning up the web-extracted text …


title: 101 Billion Arabic Words Dataset

authors: M Aloui, H Chouikhi, G Chaabane, H Kchaou… - arXiv preprint arXiv …, 2024

snippet: … We undertook a large-scale data mining project, extracting a substantial volume of text from the Common Crawl WET files, specifically targeting Arabic content. The extracted data underwent a rigorous cleaning and deduplication process, using …


title: 14 Deep Field of Learning Artificial Impacts Intelligence in the

authors: R Gulwani, M Aggarwal - Deep Learning Concepts in Operations Research, 2024

snippet: … These datasets, such as ImageNet for computer vision or the Common Crawl for natural language processing, have played a crucial role in training deep learning models and pushing the boundaries of AI performance. …


title: 2 Text Preprocessing Method

snippet: This paper uses easy data augmentation (EDA) and back translation for text enhancement. After text enhancement, the number of samples in the dataset can be significantly increased, which is helpful for model parameters through training and …


title: 4 CHATGPT IN ACADEMIA: AN IN-DEPTH EXPLORATION OF STUDENT VIEWS–PROS AND CONS

authors: D Rajkumar, P Murugeswari, M Karthigaieswari - TRANSDISCIPLINARY THREADS

snippet: … GPT-3’s training data came from five existing datasets: • Common Crawl: A collection of text pulled from billions of web pages containing trillions of words. OpenAI filtered it for high-quality reference material only. • WebText2: OpenAI …


title: 5 LLM Pretraining Methods

authors: A Velu, R Ramamoorthy, SM Manasa, A Prasanth - Generative AI and LLMs

snippet: … – Websites: This feature provides a vast array of information from the internet offering a variety of linguistic expertise, eg, Common Crawl. The major drawback of such data is that the text on the web can be of two different quality levels: spam and high-quality …


title: 6 Performance Analysis

authors: A Safi, S Singh, G Kaur, T Kaur - Intelligent Security Solutions for Cyber-Physical …, 2024

snippet: … The dataset used in this research was gathered from Alexa and the Common Crawl archive. … from a variety of sources, including Common Crawl, Alexa, and PhishTank for the phishing … They used three datasets, including real URLs …


title: " Deep Learning for de-identification of clinical documents

snippet: The amount of unstructured medical documents increases each year, presenting an opportunity to extract valuable insights that could significantly improve healthcare. However, to take advantage of this potential, it is crucial to de-identify these …


title: " Previously on..." From Recaps to Story Summarization

authors: AK Singh, D Srivastava, M Tapaswi - arXiv preprint arXiv:2405.11487, 2024

snippet: We introduce multimodal story summarization by leveraging TV episode recaps - short video sequences interweaving key story moments from previous episodes to bring viewers up to speed. We propose PlotSnap, a dataset featuring two crime …


title: “More than Words”: A Legal Approach to the Risks of Commercial Chatbots Powered by Generative Artificial Intelligence

authors: S Migliorini - European Journal of Risk Regulation, 2024

snippet: … In the training of ChatGPT, this “clean”19 version of the Common Crawl has been complemented and filtered with the addition of a few … also that the training gave more prominence to the “clean” Common Crawl.Although this explanation may be …


title: A Balancing Act: Data Protection Compliance of Artificial Intelligence

snippet: Neural network based generative artificial intelligence (GenAI) systems, in particular large language models (LLMs), hold huge potential for economic, scientific, and artistic progress. However, they are also the subject of ongoing fierce data protection …


title: A Beginner's Guide to Large Language Models

snippet: Page 1 Page 2 Page 3 “LLMs are like Swiss Army knives for digital communication. Their versatility and adaptability enable them to perform many tasks, each with precision and finesse. Whether you're a writer seeking creative inspiration, a developer …


title: A Benchmark for Relevance-based Headline Classification and Generation

snippet: The task of news headline generation deals with generating a concise summary for a given news article. It is a crucial task in increasing productivity for both the readers and producers of news. Significant progress has been made in automatically …


title: A bilingual benchmark for evaluating large language models

authors: M Alkaoud - PeerJ Computer Science, 2024

snippet: This work introduces a new benchmark for the bilingual evaluation of large language models (LLMs) in English and Arabic. While LLMs have transformed various fields, their evaluation in Arabic remains limited. This work addresses this …


title: A Blinded Comparison of Three Generative Artificial Intelligence Chatbots for Orthopaedic Surgery Therapeutic Questions

authors: V Arora, J Silburt, M Phillips, M Khan, B Petrisor… - Cureus, 2024

snippet: … Current generative large-language models (LLMs) have been trained on massive corpora of text such as common crawl - a dataset of 250 billion webpages - and thus have both a general knowledge of the world and the capacity to recapitulate human …


title: A Canary in the AI Coal Mine: American Jews May Be Disproportionately Harmed by Intellectual Property Dispossession in Large Language Model Training

authors: H Precel, A McDonald, B Hecht, N Vincent - arXiv preprint arXiv:2403.13073, 2024

snippet: Systemic property dispossession from minority groups has often been carried out in the name of technological progress. In this paper, we identify evidence that the current paradigm of large language models (LLMs) likely continues this long history …


title: A Collocation-based Method for Addressing Challenges in Word-level Metric Differential Privacy

authors: S Meisenbacher, M Chevli, F Matthes - arXiv preprint arXiv:2407.00638, 2024

snippet: Applications of Differential Privacy (DP) in NLP must distinguish between the syntactic level on which a proposed mechanism operates, often taking the form of $\textit{word-level}$ or $\textit{document-level}$ privatization. Recently, several word-level $\textit{Metric} …


title: A Comparative Analysis of Text Embedding Models for Bug Report Semantic Similarity

authors: A Patil, K Han, A Jadon - 2024 11th International Conference on Signal …, 2024

snippet: … For Fasttext, the ”crawl-300d-2M-subword” model was employed, which consisted of 2 million word vectors trained with subword information on the Common Crawl dataset, encompassing 600 billion tokens. In the case of Doc2Vec, the ”GoogleNews-vectors-negative300” …


title: A Comparative Analysis of Word Embeddings Techniques for Italian News Categorization

authors: F Rollo, G Bonisoli, L Po - IEEE Access, 2024

snippet: Text categorization remains a formidable challenge in information retrieval, requiring effective strategies, especially when applied to low-resource languages such as Italian. This paper delves into the intricacies of categorizing Italian news articles …


title: A Comparative Study of ChatGPT and Seq2Seq Chatbot for Effective Students Advising

authors: SK Assayed, M Alkhatib, K Shaalan - ChatGPT and Global Higher Education: Using …, 2024

snippet: Artificial Intelligence (AI) chatbots are increasingly becoming a part of students’ lives, from school learning to university admissions and orientations. AI chatbots are finding ways to facilitate and improve learning experiences. Moreover, AI chatbots …


title: A comparative study of cross-lingual sentiment analysis

authors: P Přibáň, J Šmíd, J Steinberger, A Mištera - Expert Systems with Applications, 2024

snippet: This paper presents a detailed comparative study of the zero-shot cross-lingual sentiment analysis. Namely, we use modern multilingual Transformer-based models and linear transformations combined with CNN and LSTM neural networks. We …


title: A Comparison of Pretrained Models for Classifying Issue Reports

authors: J Heo, G Kwon, C Kwak, S Lee - IEEE Access, 2024

snippet: Issues are evolving requirements which are the main factor that increase the cost of software evolution. To help developers manage issues, GitHub provides issue labeling mechanisms in issue management systems. However, manually labeling …


title: A Complete Survey on LLM-based AI Chatbots

authors: SK Dam, CS Hong, Y Qiao, C Zhang - arXiv preprint arXiv:2406.16937, 2024

snippet: The past few decades have witnessed an upsurge in data, forming the foundation for data-hungry, learning-based AI technology. Conversational agents, often referred to as AI chatbots, rely heavily on such data to train large language models (LLMs) and …


title: A comprehensive evaluation of large language models in mining gene relations and pathway knowledge

authors: M Azam, Y Chen, MO Arowolo, H Liu, M Popescu, D Xu - Quantitative Biology, 2024

snippet: … As LLMs typically used the Common Crawl dataset [17], any web materials open to the public, such as PubMed abstracts [18] and publications at the PMC Open Access [19], may be included in the training text of LLMs. The result in Figure 3 may …


title: A comprehensive literature review on phishing URL detection using deep learning techniques

authors: E Kritika - Journal of Cyber Security Technology, 2024

snippet: … The selected papers conducted research based on the data curated from open sources common to most such as Phishtank, Openphish, ISCX 2016, Kaggle, github and phishstorm for collecting malicious urls and Alexa, domcop, common crawl, and …


title: A Comprehensive Review on Large Language Models

authors: A Yadav - … Software Engineering Through AI, Federated Learning …, 2024

snippet: In the realm of computer science and language, large language models (LLMs) stand out as remarkable tools of artificial intelligence (AI). Proficient in deciphering intricate language nuances, LLMs offer sensible responses and find applications in …


title: A Computational Analysis of Transcribed Speech of People Living with Dementia: The Anchise 2022 Corpus

authors: F Sigona, DP Radicioni, BG Fivela, D Colla… - Computer Speech & …, 2024

snippet: Introduction Automatic linguistic analysis can provide cost-effective, valuable clues to the diagnosis of cognitive difficulties and to therapeutic practice, and hence impact positively on wellbeing. In this work, we analyzed transcribed conversations …


title: A contemporary review on chatbots, AI-powered virtual conversational agents, ChatGPT: Applications, open challenges and future research directions

authors: A Casheekar, A Lahiri, K Rath, KS Prabhakar… - Computer Science Review, 2024

snippet: This review paper offers an in-depth analysis of AI-powered virtual conversational agents, specifically focusing on OpenAI’s ChatGPT. The main contributions of this paper are threefold: (i) an exhaustive review of prior literature on chatbots, (ii) a …


title: A Copious Void: Rhetoric as Artificial Intelligence 1.0

authors: A Hallsby - Rhetoric Society Quarterly, 2024

snippet: Rhetoric is a trace retained in and by artificial intelligence (AI) technologies. This concept illuminates how rhetoric and AI have faced issues related to information abundance, entrenched social inequalities, discriminatory biases, and the …


title: A Critical Analysis of the Largest Source for Generative AI Training Data: Common Crawl

authors: S Baack - The 2024 ACM Conference on Fairness, Accountability …, 2024

snippet: … among LLM builders about the implications of using Common Crawl’s data. This paper discusses what Common Crawl’s popularity for LLM development means … Our qualitative analysis is based on indepth interviews with Common Crawl staffers …


title: A critical examination of document-level machine translation systems

snippet: The need for accurate and effective translation cannot be overstated in an increasingly globalised world where communication is paramount. Bridging language barriers is important for promoting understanding and cooperation among …


title: A Cross-Domain Benchmark for Active Learning

authors: T Werner, J Burchert, M Stubbemann… - arXiv preprint arXiv …, 2024

snippet: Active Learning (AL) deals with identifying the most informative samples for labeling to reduce data annotation costs for supervised learning tasks. AL research suffers from the fact that lifts from literature generalize poorly and that only a small number …


title: A CURATEd CATalog: Rethinking the Extraction of Pretraining Corpora for Mid-Resourced Languages

authors: J Palomar-Giner, JJ Saiz, F Espuña, M Mina, S Da Dalt… - Proceedings of the 2024 …, 2024

snippet: We present and describe two language resources in this paper: CATalog 1.0, the largest text corpus in Catalan to date, and CURATE (Corpus Utility for RAting TExt), a modular, parallelizable pipeline used for processing and scoring documents …


title: A Dataset for Pharmacovigilance in German, French, and Japanese: Annotating Adverse Drug Reactions across Languages

authors: L Raithel, HS Yeh, S Yada, C Grouin, T Lavergne… - arXiv preprint arXiv …, 2024

snippet: User-generated data sources have gained significance in uncovering Adverse Drug Reactions (ADRs), with an increasing number of discussions occurring in the digital world. However, the existing clinical corpora predominantly revolve around scientific …


title: A Dataset for the Detection of Dehumanizing Language

authors: P Engelmann, PB Trolle, C Hardmeier - arXiv preprint arXiv:2402.08764, 2024

snippet: … The Common Crawl is an open repository of web crawled data, which features crawls from all … As the Common Crawl includes several petabytes of data in total, we have selectively ex… As we limit ourselves to English, the Common Crawl data …


title: A Decade's Battle on Dataset Bias: Are We There Yet?

authors: Z Liu, K He - arXiv preprint arXiv:2403.08632, 2024

snippet: We revisit the "dataset classification" experiment suggested by Torralba and Efros a decade ago, in the new era with large-scale, diverse, and hopefully less biased datasets as well as more capable neural network architectures. Surprisingly, we …


title: A Deep Learning Approach to Fine-Grained Political Ideology Classification on Social Media Texts

authors: ER Kovacs, LA Cotfas, C Delcea - International Conference on Computational …, 2024

snippet: Starting with the mid-2010s, the growing impact of social media on politics has been felt worldwide. This impact became even more decisive in instances such as the 2020 US Presidential Election, where a bitter division between the two main US …


title: A field guide to using LLMs for online conflict analysis

authors: E Borra, P Bollini, L Caroleo, B Cerigo, R Dubèl, F Gaw…

snippet: … Two ways in which we studied LLMs was by looking at some of their open-source datasets, which one can sift through in WebText2, CommonCrawl and Wikipedia. Another is by studying autocompletions as perpetuations of standardized knowledge …


title: A Framework for Applying Copyright Law to the Training of Textual Generative Artificial Intelligence.

authors: A Neill, J Thomas, E Lee - Texas Intellectual Property Law Journal, 2024

snippet: … Websites are not notified when Common Crawl scrapes their data, but they can opt out by configuring a specific site to block the crawler.46 The second corpora used, WebText2, is an expanded version of WebText and utilizes a similar parameter for …


title: A Framework for Embedding Entities in a Textual Narrative: a Case Study on Les Misérables

snippet: In this article, we propose a general and flexible framework in order to study narrative entities found in a literary work. This framework is exposed starting from a broad perspective, consisting in how to segment the work into textual units and …


title: A Framework for Preparing a Balanced and Comprehensive Phishing Dataset

snippet: … • Choon Lin Tan (2018) [19] - dataset contains 5000 legitimate records sourced from Alexa and Common Crawl and 5000 phishing records from PhishTank and … Common Crawl (commoncrawl.org)- is a humongous web archive collected by …


title: A Framework to Construct Financial Causality Knowledge Graph from Text

authors: Z Xu, H Takamura, R Ichise - 2024 IEEE 18th International Conference on Semantic …, 2024

snippet: Causality analysis holds a prominent role in finance, and the presentation of causality could offer valuable insights for risk mitigation, investment decisions, and portfolio optimization. Recent research has extensively investigated the identification …


title: A Galician Corpus for Misogyny Detection Online

authors: LM Álvarez-Crespo, LM Castro - Proceedings of the 16th International Conference on …, 2024

snippet: Social networks are virtual spaces where millions of people share ideas, opinions, and experiences. However, this broad social interaction also exposes negative and harmful behaviors, such as harassment and misogyny. Misogyny, particularly, is a …


title: A Generalize Hardware Debugging Approach for Large Language Models Semi-Syntectic Datasets

authors: W Fu, S Li, Y Zhao, K Yang, X Zhang, Y Jin, X Guo

snippet: … C4 specifically employs the Common Crawl [44] dataset, focusing on deduplication, removing non-English content, and filtering out offensive language. Redpajama combines sources from Wikipedia and StackExchange, similar to C4 …


title: A global matching model of choice and response times in the Deese-Roediger-McDermott semantic and perceptual false recognition paradigms

snippet: … We used the pre-trained vectors from the fasttext library that were trained on the Common Crawl corpus (Grave, Bojanowski, Gupta, Joulin, & Mikolov, 2018)3. The word2vec model is considered the state-of-the-art semantic space model in its ability …


title: A Hierarchical Context Augmentation Method to Improve Retrieval-Augmented LLMs on Scientific Papers

authors: TY Che, XL Mao, T Lan, H Huang - Proceedings of the 30th ACM SIGKDD Conference …, 2024

snippet: … C𝑝𝑟𝑒𝑡𝑟𝑎𝑖𝑛 = C𝑠𝑐𝑖 ∪ C𝑎𝑢𝑔 ∪ C𝑔𝑒𝑛𝑒𝑟𝑎𝑙 (2) where C𝑠𝑐𝑖 is the previous scientific papers and C𝑔𝑒𝑛𝑒𝑟𝑎𝑙 is the general corpus such as CommonCrawl. The mixed dataset will be used to continue pretraining the language …


title: A Holistic Review on Detection of Malicious Browser Extensions and Links using Deep Learning

authors: T Zonta, M Sathiyanarayanan - 2024 IEEE 3rd International Conference on AI in …, 2024

snippet: … Common Crawl [36] is a website which crawls over the web and suggest many datasets of legitimate website and their metadata namely … Available: https://commoncrawl.or g/2021/10/september-2021-crawlarchive-now-available/ [37] AC Bahnsen, EC …


title: A Hybrid Deep BiLSTM-CNN for Hate Speech Detection in Multi-social media

authors: A Kumar, S Kumar, K Passi, A Mahanti - ACM Transactions on Asian and Low …, 2024

snippet: Nowadays, ways of communication among people have changed due to advancements in information technology and the rise of online multi-social media. Many people express their feelings, ideas, and emotions on social media sites such …


title: A Japanese-Chinese Parallel Corpus Using Crowdsourcing for Web Mining

authors: M Nagata, M Morishita, K Chousa, N Yasuda - arXiv preprint arXiv:2405.09017, 2024

snippet: … In ParaCrawl, they determine which websites to crawl by analyzing the Common Crawl … In this study, we analyzed 12 sets of Common Crawl archives (104TB in total) published from … and bilingual website URLs obtained from Common Crawl …


title: A Language Model Trained on Uruguayan Spanish News Text

authors: JP Filevich, G Marco, S Castro, L Chiruzzo, A Rosá - LREC-COLING 2024, 2024

snippet: This paper presents a language model trained from scratch exclusively on a brand new corpus consisting of about 6 GiB of Uruguayan newspaper text. We trained the model for 30 days on a single Nvidia P100 using the RoBERTa-base architecture …


title: A Large-Scale Exploration of $\mu $-Transfer

authors: L Lingle - arXiv preprint arXiv:2404.05728, 2024

snippet: Large neural network models have become a mainstay of natural language processing and computer vision, yet their initialization and learning rates are set in a largely heuristic fashion, potentially varying from paper to paper and one model size …


title: A Legal Framework for Natural Language Model Training in Portugal

authors: R Almeida, E Amorim - Proceedings of the Workshop on Legal and Ethical …, 2024

snippet: Recent advances in deep learning have promoted the advent of many computational systems capable of performing intelligent actions that, until then, were restricted to the human intellect. In the particular case of human languages, these …


title: A Legal Framework for Natural Language Processing Model Training in Portugal

authors: R Almeida, E Amorim - arXiv preprint arXiv:2405.00536, 2024


title: A Little Leak Will Sink a Great Ship: Survey of Transparency for Large Language Models from Start to Finish

authors: M Kaneko, T Baldwin - arXiv preprint arXiv:2403.16139, 2024

snippet: … The most common sources included in all LLMs are web page sources such as C4, CommonCrawl, and the Pile. Because they are collected from various web pages, there is a risk that they may contain personal information, copyrighted texts …


title: A Longitudinal Study of Content Control Mechanisms

authors: M Dinzinger, M Granitzer - Companion Proceedings of the ACM on Web …, 2024

snippet: … Both resources, robots.txt and regular pages, were collected from the Common Crawl web archive 7 and processed through a customized … As Common Crawl is furhtermore not focused on particular web domains, but randomly samples the …


title: A methodology for using players' chat content for dynamic difficulty adjustment in metaverse multiplayer games

authors: MM Rezapour, A Fatemi, MA Nematbakhsh - Applied Soft Computing, 2024

snippet: … RoBERTa, on the other hand, was trained on a much larger corpus of text that includes not only Wikipedia and BookCorpus but also Common Crawl, a dataset that contains billions of web pages. This larger and more diverse corpus of text allows …


title: A multi-level multi-label text classification dataset of 19th century Ottoman and Russian literary and critical texts

authors: G Gokceoglu, D Cavusoglu, E Akbas, ÖN Dolcerocca - arXiv preprint arXiv …, 2024

snippet: This paper introduces a multi-level, multi-label text classification dataset comprising over 3000 documents. The dataset features literary and critical texts from 19th-century Ottoman Turkish and Russian. It is the first study to apply large language models (LLMs) …


title: A Multilingual Dataset for Investigating Stereotypes and Negative Attitudes Towards Migrant Groups in Large Language Models

authors: D Sorato, CC Ventura, D Zavala-Rojas - … of the 16th International Conference on …, 2024

snippet: … 2020), which is a cleaned version of the CommonCrawl corpus. … where stereotypes are presented in more subtle and/or strategic ways (when compared to social media/CommonCrawl texts) and the explicit discrimination of migrant groups …


title: A Multilingual Perspective on Probing Gender Bias

authors: K Stańczak - arXiv preprint arXiv:2403.10699, 2024

snippet: Gender bias represents a form of systematic negative treatment that targets individuals based on their gender. This discrimination can range from subtle sexist remarks and gendered stereotypes to outright hate speech. Prior research has …


title: A New Massive Multilingual Dataset for High-Performance Language Technologies

authors: O de Gibert, G Nail, N Arefyev, M Bañón… - arXiv preprint arXiv …, 2024

snippet: We present the HPLT (High Performance Language Technologies) language resources, a new massive multilingual dataset including both monolingual and bilingual corpora extracted from CommonCrawl and previously unused web crawls …


title: A New Method for Cross-Lingual-based Semantic Role Labeling

authors: M Ebrahimi, BM Bidgoli, N Khozouei - arXiv preprint arXiv:2408.15896, 2024

snippet: … It has received specialized training using 2.5 terabytes of meticulously processed CommonCrawl data, which includes text from 100 different languages. RoBERTa itself is a transformer-based model that has been pre-trained in a self-supervised …


title: A New Type of Foundation Model Based on Recordings of People's Emotions and Physiology

authors: D Gamez, D Barcari, A Grig - arXiv preprint arXiv:2408.00030, 2024

snippet: … These models are trained on large amounts of text and image data scraped from the Internet - often accessed through the Common Crawl repository. Researchers and technology companies are starting to realize that this data source could become …


title: A Novel Hashcode-based Duplication Reduction via Thresholding Approach for Large-scale Web Documents

snippet: Modern search engines encounter a significant challenge when it comes to handling duplicate and nearly identical web pages, particularly during the indexing process for vast amounts of web content. This issue can lead to slow search results and …


title: A Permutaion Importance Based feature selection method and Deep Learning Model to Detect Phishing Websites

snippet: Phishing attacks pose a significant and escalating threat to cybersecurity in recent times. This deceptive scam aims to trick naive users, luring them into visiting harmful websites and sharing sensitive information, including credentials, credit card …


title: A phenomenology and epistemology of large language models: Transparency, trust, and trustworthiness

authors: R Heersmink, B de Rooij, MJC Vázquez, M Colombo

snippet: … We know that its dataset includes the Common Crawl, which is a publicly available corpus of Webpages, including billions of Webpages … whether sources in the Common Crawl or Wikipedia are prioritised. This is again a serious issue to do …


title: A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)

snippet: … For low-resource languages, XLM models trained on CommonCrawl-100 did better than the ones trained using Wikipedia. Another example is mT5 [32], which uses multilingual corpus mC4 to pretrain the model. When dealing with multilingual …


title: A Review of Multi-Modal Large Language and Vision Models

authors: K Carolan, L Fennelly, AF Smeaton - arXiv preprint arXiv:2404.01322, 2024

snippet: … The Falcon LLM team created the RefinedWed dataset, which is a massive web dataset based on CommonCrawl. TII focused on scaling and improving the quality of web data by employing large-scale de-duplication and strict filtering, resulting in …


title: A review of multimodal-based emotion recognition techniques for cyberbullying detection in online social media platforms

authors: S Wang, AS Shibghatullah, TJ Iqbal, KH Keoy - Neural Computing and Applications, 2024

snippet: Cyberbullying is a serious issue in online social media platforms (OSMP), which requires effective detection and intervention systems. Multimodal emotion recognition (MER) technology can help prevent cyberbullying by analyzing emotions …


title: A Review of the Challenges with Massive Web-mined Corpora Used in Large Language Models Pre-Training

authors: M Perełkiewicz, R Poświata - arXiv preprint arXiv:2407.07630, 2024

snippet: … Common Crawl is an open repository of web crawl data collected over years. The Common Crawl repository provides around 20TB of scraped … Due to the wide coverage of diverse domains and languages, as well as the size of the collected data …


title: A review on emotion detection by using deep learning techniques

authors: T Chutia, N Baruah - Artificial Intelligence Review, 2024

snippet: Along with the growth of Internet with its numerous potential applications and diverse fields, artificial intelligence (AI) and sentiment analysis (SA) have become significant and popular research areas. Additionally, it was a key technology that …


title: A Robust Approach to E-Banking Phishing Detection using Ensemble Methods and LSTM

authors: NVS Reddy, ARR Saai, TV Ramanujan, NRK Reddy… - … International Conference on …, 2024

snippet: … range of datasets from reputable community platforms like VirusTotal, Common Crawl, and PhishTank. This extensive dataset, spanning … Integration of Common Crawl further enriched the dataset, ensuring a diverse set of websites for effective …


title: A robust hybrid approach with product context-aware learning and explainable AI for sentiment analysis in Amazon user reviews

authors: E Hashmi, SY Yayilgan - Electronic Commerce Research, 2024

snippet: … FastText, Footnote 5 a word representation library developed by the Facebook research team, encompasses a repository of 2 million common crawl words with 300 dimensions, resulting in a vast collection of 600 billion word vectors. Distinguished …


title: A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelism

authors: B Thompson, MP Dhaliwal, P Frisch, T Domhan… - arXiv preprint arXiv …, 2024

snippet: … 2021), which is in turn based on Common CrawlCommon Crawl is a long running web-scraping project which maintains a free, open source repository of web-scraped data. ccMatrix is created by embedding Common Crawl sentences into a …


title: A Short Commentary on Trinh & Le (2018)

snippet: … corpus (trained on the language model LM-1-Billion, CommonCrawl, and SQuAD), the probabilities of s1 and s2 appearing in a large corpus. The substitution that turns out to be more probable, is considered to be the more probable referent of “it”. There …


title: A Study of Aging Through Speech and Language Analysis

snippet: Speech is a promising biomarker for cognitive impairment and dementing illness. Compared to traditional biomarkers, digital biomarkers are often less invasive, cheaper to measure, and require less instruction. They also allow continuous and …


title: A support system for the detection of abusive clauses in B2C contracts

authors: S Dadas, M Kozłowski, R Poświata, M Perełkiewicz… - Artificial Intelligence and …, 2024

snippet: … To better balance our dataset, we retrieved an additional 7547 safe clauses from Polish CommonCrawl Footnote 15 dumps using weak supervision techniques. Our approach involved scanning approximately 100 GB of Polish web pages and …


title: A Survey of AI-generated Text Forensic Systems: Detection, Attribution, and Characterization

authors: T Kumarage, G Agrawal, P Sheth, R Moraffah… - arXiv preprint arXiv …, 2024

snippet: We have witnessed lately a rapid proliferation of advanced Large Language Models (LLMs) capable of generating high-quality text. While these LLMs have revolutionized text generation across various domains, they also pose significant …


title: A Survey of Conversational Styles and Systems

authors: R Nagpal, S Fatty, DG Brizan - 2024 16th International Conference on Human …, 2024

snippet: Conversational systems are an emerging area in natural language processing. This survey paper provides a comprehensive overview of architectures, methods, and datasets used to design the development of conversational agents. We have …


title: A Survey of Large Language Models for European Languages

snippet: … The OSCAR [122] corpus is a large multilingual text dataset extracted from Common Crawl (CC). The German subset of OSCAR is particularly noteworthy, measuring 496.7GB in size and containing approximately 7 billion documents and …


title: A Survey of Multimodal Large Language Model from A Data-centric Perspective

authors: T Bai, H Liang, B Wan, L Yang, B Li, Y Wang, B Cui… - arXiv preprint arXiv …, 2024

snippet: … CommonCrawl project serves as the most commonly … , CommonCrawl’s vast archive of web pages, which includes numerous image-text pairs, has also become a vital resource for constructing multimodal pre-training datasets such as LAION-5B [236] …


title: A Survey of Neural Machine Translation based on Knowledge Distillation

authors: MA Chang, T Yonghong, Z Xiaoli, SUN Kangkang - Journal of Frontiers of Computer Science …

snippet: Abstract: Machine Translation (MT) is the process of using a computer to convert a language into another language with the same semantics. With the introduction of neural network, neural machine translation (NMT), as a powerful machine …


title: A Survey of Resource-efficient LLM and Multimodal Foundation Models

authors: M Xu, W Yin, D Cai, R Yi, D Xu, Q Wang, B Wu, Y Zhao… - arXiv preprint arXiv …, 2024

snippet: Large foundation models, including large language models (LLMs), vision transformers (ViTs), diffusion, and LLM-based multimodal models, are revolutionizing the entire machine learning lifecycle, from training to deployment …


title: A Survey of Web Content Control for Generative AI

authors: M Dinzinger, F Heß, M Granitzer - arXiv preprint arXiv:2404.02309, 2024

snippet: … The documents were sourced from the web archive of Common Crawl,16 offering a broad and randomly selected cross-section of the internet. These files were all collected in November and December 2023. Further details on the experimental …


title: A Survey on Data Selection for Language Models

authors: A Albalak, Y Elazar, SM Xie, S Longpre, N Lambert… - arXiv preprint arXiv …, 2024

snippet: … They train a classifier using the “high-quality” reference corpora as the positive class and unfiltered Common Crawl documents as the negative class. This classifier then defines the utility metric for selecting data that follows the desired distribution …


title: A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontiers

authors: K Huang, F Mo, H Li, Y Li, Y Zhang, W Yi, Y Mao, J Liu… - arXiv preprint arXiv …, 2024

snippet: The rapid development of Large Language Models (LLMs) demonstrates remarkable multilingual capabilities in natural language processing, attracting global attention in both academia and industry. To mitigate potential discrimination …


title: A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias

authors: Y Xu, L Hu, J Zhao, Z Qiu, Y Ye, H Gu - arXiv preprint arXiv:2404.00929, 2024

snippet: … A significant portion of these training data originates from multilingual repositories like Common Crawl, WikiPedia and Web documents, encompassing a broad range of languages. These multilingual repositories are crucial for enhancing the cross-lingual …


title: A survey on recent advances in named entity recognition

authors: I Keraghel, S Morbieu, M Nadif - arXiv preprint arXiv:2401.10825, 2024

snippet: Named Entity Recognition seeks to extract substrings within a text that name real-world objects and to determine their type (for example, whether they refer to persons or organizations). In this survey, we first present an overview of recent popular …


title: A Survey on the Use of Large Language Models (LLMs) in Fake News

authors: E Papageorgiou, C Chronis, I Varlamis, Y Himeur - Future Internet, 2024

snippet: … It was trained on the Colossal Clean Crawled Corpus (C4) dataset, a 750 GB dataset created from Common Crawl’s web-extracted text. The model architecture is similar to the original … It was trained on the RealNews dataset, created from …


title: A systematic review of multidimensional relevance estimation in information retrieval

authors: G Peikos, G Pasi - Wiley Interdisciplinary Reviews: Data Mining and …

snippet: In information retrieval, relevance is perceived as a multidimensional and dynamic concept influenced by user, task, and domain factors. Relying on this perspective, researchers have introduced multidimensional relevance models addressing …


title: A Systematic Study of the Role of Data Quality and Alignment for Fine-tuning LLMs for Enhanced Autoformalization

authors: K Chawla, A Sahai, M DePavia, B Miranda - The Second Tiny Papers Track at ICLR 2024

snippet: This study explores the role of data quality, particularly alignment, in fine-tuning Large Language Models (LLMs) for the task of autoformalization. Contrary to the conventional emphasis on dataset size, our research highlights the importance of …


title: A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Models

authors: H Zhang, PS Yu, J Zhang - arXiv preprint arXiv:2406.11289, 2024

snippet: … WCEP [66] is built based on news events from Wikipedia Current Events Portal (WCEP) and similar articles in the CommonCrawl News dataset. It features high alignment with several realworld industrial use cases and has 235 articles per cluster on …


title: A Taxonomy of Stereotype Content in Large Language Models

authors: G Nicolas, A Caliskan - arXiv preprint arXiv:2408.00162, 2024

snippet: This study introduces a taxonomy of stereotype content in contemporary large language models (LLMs). We prompt ChatGPT 3.5, Llama 3, and Mixtral 8x7B, three powerful and widely used LLMs, for the characteristics associated with 87 social …


title: A Trustworthy Automated Short-Answer Scoring System Using a New Dataset and Hybrid Transfer Learning Method

authors: M Maslim, HC Wang, CD Putra, YD Prabowo - 2024

snippet: To measure the quality of student learning, teachers must conduct evaluations. One of the most efficient modes of evaluation is the short answer question. However, there can be inconsistencies in teacher-performed manual evaluations due to an …


title: ABCF: An Adaptive Balanced Multimodal Website Classification Framework

authors: Z Liu, W Liu, X Tong, Q Shen, Q Zheng, H Hu, H Liu… - 2024 27th International …, 2024

snippet: Websites are a crucial medium for conveying multimedia information. However, malicious websites can lurk among them, posing a threat to the security of users’ sensitive data and personal privacy. Hence, it is significant to identify and categorize …


title: AbdomenAtlas: A Large-Scale, Detailed-Annotated, & Multi-Center Dataset for Efficient Transfer Learning and Open Algorithmic Benchmarking

authors: W Li, C Qu, X Chen, PRAS Bassi, Y Shi, Y Lai, Q Yu… - arXiv preprint arXiv …, 2024

snippet: We introduce the largest abdominal CT dataset (termed AbdomenAtlas) of 20,460 three-dimensional CT volumes sourced from 112 hospitals across diverse populations, geographies, and facilities. AbdomenAtlas provides 673K high-quality …


title: AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters

authors: L Lucy, S Gururangan, L Soldaini, E Strubell… - arXiv preprint arXiv …, 2024

snippet: … From this Common Crawl data, we identify website hostnames that include an ABOUT page, which we define as URL paths containing about, aboutme, about-us, or bio (Appendix A). We then pair each ABOUT page with a random page on the …


title: Abusive Language Detection in Khasi Social Media Comments

authors: A Baruah, L Wahlang, F Jyrwa, F Shadap, F Barbhuiya… - ACM Transactions on Asian …, 2024

snippet: This paper describes the work performed for automated abusive language detection in the Khasi language, a low-resource language spoken primarily in the state of Meghalaya, India. A dataset named Khasi Abusive Language Dataset (KALD) was …


title: Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review

authors: N Prakriya, JN Yen, CJ Hsieh, J Cong - arXiv preprint arXiv:2409.06131, 2024

snippet: Large Language Model (LLM) pretraining traditionally relies on autoregressive language modeling on randomly sampled data blocks from web-scale datasets. We take inspiration from human learning techniques like spaced repetition to …


title: Accelerating Spiking Neural Networks with Parallelizable Leaky Integrate-and-Fire Neurons

snippet: … To achieve this, these models undergo training on massive datasets, such as the Common Crawl corpus1, which contains petabytes of textual data. Training on such extensive datasets necessitates the use of fast machine learning models like the …


title: ACCEPTANCE OF GENERATIVE AI IN KNOWLEDGE WORK

snippet: … The GPT-3 was trained with Corpus, a large and structured collection of texts or language data, called Common Crawl including nearly a trillion words. (Brown et al., 2020) In addition, the training set was also expanded with known high-quality …


title: Accessible Foundation Models: Systems, Algorithms, and Science

snippet: … [Raffel et al., 2019] which is a subset of the Common Crawl corpus.We use NVIDIA A40 GPUs for this evaluation. To measure degradation in … -shot experiments, the Pearson correlation coefficient between The Pile Common Crawl …


title: ACCSAMS: Automatic Conversion of Exam Documents to Accessible Learning Material for Blind and Visually Impaired

authors: D Wilkening, O Moured, T Schwarz, K Muller… - arXiv preprint arXiv …, 2024

snippet: … We collected exam materials from the Common Crawl project4, a public archive of web data. We analyzed eight datasets, released between mid 2022 and the end of 2023, extracting PDFs, which include “exam” or “klausur” as a keyword in the URL …


title: Accuracy of ChatGPT in Neurolocalization

authors: WF Dabbas, YM Odeibat, M Alhazaimeh, MY Hiasat… - Cureus, 2024

snippet: … The datasets used for pre-training included Common Crawl dataset, an expanded version of the WebText dataset, two internet-based books corpora (Books1 and Books2), and English-language Wikipedia [13]. …


title: Active Learning for Identifying Disaster-Related Tweets: A Comparison with Keyword Filtering and Generic Fine-Tuning

authors: D Hanny, S Schmidt, B Resch - Intelligent Systems Conference, 2024

snippet: … [4] was utilised as it is pre-trained on approximately 198 million Tweets and has been shown to outperform the similar XLM-RoBERTa model trained using the more general CommonCrawl corpus [11] for the multilingual classification of Tweets. The …


title: Adam-mini: Use Fewer Learning Rates To Gain More

authors: Y Zhang, C Chen, Z Li, T Ding, C Wu, Y Ye, ZQ Luo… - arXiv preprint arXiv …, 2024

snippet: We propose Adam-mini, an optimizer that achieves on-par or better performance than AdamW with 45% to 50% less memory footprint. Adam-mini reduces memory by cutting down the number of learning rates in Adam: Instead of assigning an …


title: Adaptation of Large Language Models for Streaming Machine Translation

snippet: [EN] Machine translation (MT) stands as a pivotal domain within machine learning, where the rise of neural networks has sparked significant advancements, propelling MT into a highly researched field. This has been further fostered by the proliferation …


title: Adapting LLMs to Downstream Applications

snippet: … The model was fed with text that generally precedes private information in a dataset used to train GPT-2—specifically the “Common Crawl” dataset to achieve a better recall. 2 With this approach, authors were not only able to extract extensive …


title: Adaptive Ensembles of Fine-Tuned Transformers for LLM-Generated Text Detection

authors: Z Lai, X Zhang, S Chen - arXiv preprint arXiv:2403.13335, 2024

snippet: … 5) XLMRoberta: XLMRoberta is a large-scale multilingual transformer-based LM pretrained on a diverse set of one hundred languages using extensive CommonCrawl data [68]. We use the “xlm roberta base multi” as pretrained backbone weight. …


title: Addressing Both Statistical and Causal Gender Fairness in NLP Models

authors: H Chen, Y Ji, D Evans - arXiv preprint arXiv:2404.00463, 2024

snippet: … 2019) comprising nearly 400,000 online biographies of 28 unique occupations scraped from the CommonCrawl. The task is to predict the occupation given in the biography with the occupation title removed. Each biography includes the name and …


title: Addressing the Challenge of Online Health Misinformation: Detection, Retrieval, and Explainability

snippet: Nell’odierna epoca digitale, le piattaforme online costituiscono uno dei mezzi principali utilizzati dalle persone per cercare informazioni relative alla propria salute. Nonostante il web sia un vasto repository di conoscenze in tale ambito, è affetto dal …


title: Adjusting Interpretable Dimensions in Embedding Space with Human Judgments

authors: K Erk, M Apidianaki - arXiv preprint arXiv:2404.02619, 2024

snippet: … The GLoVe embeddings used were trained on Common Crawl (42B tokens, 1.9M vocab, uncased, 300d vectors), downloaded from https: //nlp.stanford.edu/projects/glove/. In order to generate embeddings for contextualized instances of words in our …


title: AdTEC: A Unified Benchmark for Evaluating Text Quality in Search Engine Advertising

authors: P Zhang, Y Sakai, M Mita, H Ouchi, T Watanabe - arXiv preprint arXiv:2408.05906, 2024

snippet: With the increase in the more fluent ad texts automatically created by natural language generation technology, it is in the high demand to verify the quality of these creatives in a real-world setting. We propose AdTEC, the first public …


title: Adult Content Detection on Indonesian Tweets by Fine-tuning Transformer-based Models

authors: AF Hidayatullah, RA Apong, DTC Lai, A Qazi - 2023 6th International Conference on …, 2023

snippet: The prevalence of adult content on social media has harmful effects on the moral values of young individuals. Therefore, effectively filtering inappropriate content on social media like Twitter is essential. Researchers have utilized machine learning …


title: Advances in Multimodal Information Retrieval and Generation

snippet: Writing systems emerged simultaneously and independently in many ancient civilizations across the globe, including Mesopotamia, Egypt, China, India, and Mesoamerica. The invention of writing and scripts has had a tremendous impact on …


title: Advancing Fake News Detection: Hybrid Deep Learning with FastText and Explainable AI

authors: E Hashmi, SY Yayilgan, MM Yamin, S Ali, M Abomhara - IEEE Access, 2024

snippet: … FastText, a word representation tool developed by Facebook’s research division, provides both unsupervised and supervised modes, featuring an extensive lexicon of 2 million words sourced from Common Crawl. Each word is represented in a 300-dimensional …


title: Advancing Internet Viewpoint Diversity: A Novel Algorithm and a Corpus Creation Tool

snippet: … To achieve this, we first develop a big data processing architecture for creating indexed corpora from the Common Crawl web archives. The … Utilizing this tool, we processed approximately 1.2 billion web pages from the Common Crawl dataset …


title: Advancing language models through domain knowledge integration: a comprehensive approach to training, evaluation, and optimization of social scientific neural …

authors: F Stöhr - Journal of Computational Social Science, 2024

snippet: This article proposes a comprehensive strategy for training, evaluating, and optimizing domain-specific word2vec-based word embeddings, using social science literature as an example. Our primary objectives are:(1) to train the embeddings …


title: Advancing Large Multi-modal Models with Explicit Chain-of-Reasoning and Visual Question Generation

authors: K Uehara, N Goswami, H Wang, T Baba, K Tanaka… - arXiv preprint arXiv …, 2024

snippet: The increasing demand for intelligent systems capable of interpreting and reasoning about visual content requires the development of Large Multi-Modal Models (LMMs) that are not only accurate but also have explicit reasoning capabilities. This paper …


title: Advancing medical imaging with language models: featuring a spotlight on ChatGPT

authors: M Hu, JY Qian, S Pan, Y Li, RLJ Qiu, X Yang - Physics in Medicine and Biology, 2024

snippet: This review paper aims to serve as a comprehensive guide and instructional resource for researchers seeking to effectively implement language models in medical imaging research. First, we presented the fundamental principles and …


title: Advancing Real-time Pandemic Forecasting Using Large Language Models: A COVID-19 Case Study

authors: H Du, J Zhao, Y Zhao, S Xu, X Lin, Y Chen, LM Gardner - arXiv preprint arXiv …, 2024

authors: HF Yang, H Du, J Zhao, Y Zhao, S Xu, X Lin, Y Chen… - 2024

snippet: Forecasting the short-term spread of an ongoing disease outbreak is a formidable challenge due to the complexity of contributing factors, some of which can be characterized through interlinked, multi-modality variables such as epidemiological …


title: Advancing scientific writing with artificial intelligence: expanding the research toolkit

authors: AG Atkinson, H Lia, SM Navarro - Global Surgical Education-Journal of the …, 2024

snippet: Generative artificial intelligence (AI) is revolutionizing scientific writing by offering novel tools that integrate into various stages of the research process. This paper explores the application of AI tools in scientific writing, focusing on literature review …


title: Afrikaans Literary Genre Recognition using Embeddings and Pre-Trained Multilingual Language Models

authors: E Kotzé, BA Senekal - 2024 International Conference on Artificial Intelligence …, 2024

snippet: … XLM-R (or XLM-RoBERTa) is a multilingual version of RoBERTa with 279M parameters and was trained on a 2.5TB of CommonCrawl data containing 100 languages. Afrikaans is included in all three of these multilingual models. …


title: AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation

authors: M Hu, P Zhao, C Xu, Q Sun, J Lou, Q Lin, P Luo… - arXiv preprint arXiv …, 2024

snippet: … The inspiration corpus can be implemented in various ways, such as using a large-scale pre-trained corpus like Common Crawl. Alternatively, a domain-specific corpus, such as a code generation dataset [25, 7], can be used to generate environments for a …


title: Aggression and Misogyny in Hindi and Bangla: A Study of YouTube Comments

authors: R Kumar, B Lahiri - Studies in Pragmatics, 2024

snippet: … The pre-trained models are typically trained on web-scale data, ie text collected from all over the web such as the whole of Wikipedia and a common crawl of the websites currently on the internet amounting to billions (or trillions) of tokens and …


title: AgXQA: A benchmark for advanced Agricultural Extension question answering

authors: J Kpodo, P Kordjamshidi, AP Nejadhashemi - Computers and Electronics in …, 2024

snippet: Large language models (LLMs) have revolutionized various scientific fields in the past few years, thanks to their generative and extractive abilities. However, their applications in the Agricultural Extension (AE) domain remain sparse and limited …


title: AI and The European Union's Approach to Data Protection: The Case of Chat GPT

snippet: Artificial Intelligence (AI) is advancing rapidly, with generative models like ChatGPT revolutionizing numerous industries. However, these advancements present significant challenges in adhering to data protection regulations such as the General …


title: AI chatbots show promise but limitations on UK medical exam questions: a comparative performance study

authors: MA Sadeq, RMF Ghorab, MH Ashry, AM Abozaid… - Scientific Reports, 2024

snippet: … All chatbots scored higher on questions leaked to the C4 database except for ClaudeInstant which performed worse on the seven questions leaked to the common crawl database (0.571 ± 0.535) than other questions (0.631 ± 0.483). Bard …


title: AI Literacy for the Common Good

authors: S Ullrich, R Messerschmidt - Weizenbaum Journal of the Digital Society, 2024

snippet: … A large part of the training data (almost two-thirds) is the Common Crawl dataset, which comprises web pages collected between 2017 and 2020 provided for free by a non-profit organization. Wikipedia articles from the summer of 2022 still flow in at …


title: AI unveiled personalities: Profiling optimistic and pessimistic attitudes in Hindi dataset using transformer‐based models

snippet: Both optimism and pessimism are intricately intertwined with an individual's inherent personality traits and people of all personality types can exhibit a wide range of attitudes and behaviours, including levels of optimism and pessimism. This paper …


title: AI vs. Human: Decoding Text Authenticity with Transformers

snippet: In an era where the proliferation of large language models blurs the lines between human and machine-generated content, discerning text authenticity is paramount. This study investigates transformer-based language models—BERT, RoBERTa, and …


title: AI's Secret Weapon in Education. ChatGPT–The Future of Personalized Learning

authors: A Popescu - Bulletin of the Transilvania University of Brasov. Series …, 2023

snippet: This article delves into the fascinating intersection of AI and education, with a focus on ChatGPT, a conversational AI developed by OpenAI. Noted for its human-like responses, ChatGPT is positioned as a game-changer in personalized learning. The …


title: AI, Robots. txt

snippet: … A significant portion of the training data for LLMs comes from large datasets like "The Pile" and "Common Crawl," which are aggregates of smaller datasets [5, 17]. These datasets, often hundreds of GBs in size, are filtered and processed into plaintext …


title: AI-Generated Text Detector for Arabic Language

snippet: The rise of AI-generated texts (AIGTs), particularly with the arrival of advanced language models like ChatGPT, has spurred a growing need for effective detection methods. While these models offer various beneficial applications, their potential for …


title: AI-Powered Code Review Assistant for Streamlining Pull Request Merging

authors: C Adapa, SS Avulamanda, ARK Anjana, A Victor - 2024 IEEE International …, 2024

snippet: … The model's reliance on Refined Web, primarily sourced from Common Crawl, underscores its commitment to exceptional quality through advanced data augmentation techniques, notably largescale deduplication and rigorous filtering. Unlike conventional …


title: AI-Proof Your Assignments

snippet: ChatGPT and other AI tools are powerful, free, and seemingly everywhere. Faculty love it, fear it, and everything in between. Students also love it, fear it, and everything in between; All the while, they are using it. In this session, we’ll cover a bit about how …


title: AI-Replicas as Ethical Practice: Introducing an Alternative to Traditional Anonymization Techniques in Image-Based Research

snippet: This paper introduces the use of AI-replicas as an alternative to traditional anonymization methods in image-based qualitative research. It emphasizes the ethical and practical dilemmas posed by current anonymization methods, such as …


title: Algorithmic amplification of biases on Google Search

authors: H Habib, R Stoldt, A High, B Ekdale, A Peterson… - arXiv preprint arXiv …, 2024

snippet: The evolution of information-seeking processes, driven by search engines like Google, has transformed the access to information people have. This paper investigates how individuals' preexisting attitudes influence the modern information-seeking …


title: Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs

authors: AM Kassem, O Mahmoud, N Mireshghallah, H Kim… - arXiv preprint arXiv …, 2024

snippet: In this paper, we introduce a black-box prompt optimization method that uses an attacker LLM agent to uncover higher levels of memorization in a victim agent, compared to what is revealed by prompting the target model with the training data …


title: ALT-GEN: Benchmarking Table Union Search using Large Language Models

authors: K Pal, A Khatiwada, R Shraga, RJ Miller - Proceedings of the VLDB Endowment. ISSN

snippet: David Paterson states that when a eld has good benchmarks, we settle debates and the eld makes rapid progress [33]. Traditionally, benchmark generation is done using synthetic data generators that precisely control parameters such as data size …


title: Amharic LLaMA and LLaVA: Multimodal LLMs for Low Resource Languages

authors: M Andersland - arXiv preprint arXiv:2403.06354, 2024

snippet: … Less than 0.1% of CommonCrawl2 is Amharic, and even when combining open source datasets without deduplication, we find that less than 500 million tokens of Amharic are available. In addition, the content of this data tends to be biased …


title: An Ambiguous Technique for Nonvisual Text Entry

snippet: Text entry is a common daily task for many people, but it can be a challenge for people with visual impairments when using virtual touchscreen keyboards that lack physical key boundaries. In this thesis, we investigate using a small number of …


title: An Analysis of Langauge Frequency and Error Correction for Esperanto

authors: J Liang - arXiv preprint arXiv:2402.09696, 2024

snippet: … Goyal builds Commoncrawl Corpus in 100 languages and then uses a language identification model to select Esperanto contexts[8], which … OSCAR [11] provides a collection of unannotated raw data and devises a method to improve the quality of …


title: An Analysis of Multilingual FActScore

authors: KT Vu, M Krumdick, V Reddy, F Dernoncourt, VD Lai - arXiv preprint arXiv …, 2024

snippet: FActScore has gained popularity as a metric to estimate the factuality of long-form texts generated by Large Language Models (LLMs) in English. However, there has not been any work in studying the behavior of FActScore in other languages. This …


title: AN ANALYSIS ON SHORT-FORM TEXT AND DERIVED ENGAGEMENT

snippet: Short text has historically proven challenging to work with in many Natural LanguageProcessing (NLP) applications. Traditional tasks such as authorship attribution benefitfrom having longer samples of work to derive features from. Even …


title: An Assessment on Comprehending Mental Health through Large Language Models

authors: M Arcan, PD Niland, F Delahunty - arXiv preprint arXiv:2401.04592, 2024

snippet: … The training data for GPT-3 is primarily sourced from a filtered version of Common Crawl, contributing to 60% of the weighted pre-training dataset, comprising 410 billion bytepair-encoded tokens. Other data sources include 19 billion tokens from …


title: An efficient parallel optimization method for large model based on cloud computing

authors: Z Xu - … Conference on Electronic Information Engineering and …, 2024

snippet: … Subsequently, we employ a similar methodology to create Pile-CC, involving the download and refinement of two complete Common Crawl (CC) snapshots. The overall process for handling CC data entails extracting text from the raw HTML …


title: An Extensive Survey on Investigation Methodologies for Text Summarization

authors: A Saklecha, P Uplavdiya, MPS Chawla - Indian Journal of Signal Processing (IJSP), 2023

snippet: Natural language processing (NLP) is a fastexpanding field, and text summarization has recently gained a lot of research interest. The necessity for automatic summarizing approaches to effectively digest massive amounts of textual data has …


title: An In-Depth Analysis of the Adoption of Large Language Models in Clinical Settings: A Fuzzy Multi-Criteria Decision-Making Approach

snippet: The growing capabilities of large language models (LLMs) in the medical field hold promising transformational change. The evolution of LLMs, such as BioBERT and MedGPT, has created new opportunities for enhancing the quality of healthcare …


title: An Integrated Data Processing Framework for Pretraining Foundation Models

authors: Y Sun, F Wang, Y Zhu, WX Zhao, J Mao - arXiv preprint arXiv:2402.16358, 2024

snippet: … In the end-to-end evaluation, we train two GPT-2 models using CommonCrawl before and after processing, respectively. The model trained on the refined data demonstrates remarkable performance enhancement across downstream tasks of …


title: An integrated model based on deep learning classifiers and pre-trained transformer for phishing URL detection

authors: NQ Do, A Selamat, H Fujita, O Krejcar - Future Generation Computer Systems, 2024

snippet: The unique nature of website URLs has made phishing detection a challenging task. Unlike natural language, URLs have an unstructured nature with non-linear and sophisticated correlations. Therefore, they should be handled as both natural …


title: An Introduction to Vision-Language Modeling

authors: F Bordes, RY Pang, A Ajay, AC Li, A Bardes, S Petryk… - arXiv preprint arXiv …, 2024

snippet: … Multiple curation steps are used to curate this dataset where English data is collected from common crawl and deduplicated followed by pre-processing HTML document where useful DOM nodes are identified and retained, then for each DOM …


title: Analysing Slow Thinking Capabilities in Large Language Model Agent-Agent Dialogue

snippet: … LLMs can be given data from books, CommonCrawl, Reddit links and Wikipedia, where CommonCrawl consists of petabytes of web archives [113]. Having large datasets makes it more difficult to curate the data. Documentation of the data can …


title: Analysing The Impact of Sequence Composition on Language Model Pre-Training

authors: Y Zhao, Y Qu, K Staniszewski, S Tworkowski, W Liu… - arXiv preprint arXiv …, 2024

snippet: Most language model pre-training frameworks concatenate multiple documents into fixed-length sequences and use causal masking to compute the likelihood of each token given its context; this strategy is widely adopted due to its simplicity and …


title: Analysing the landscape of Deep Fake Detection: A Survey

authors: K Vyas, P Pareek, R Jayaswal, S Patil - … of Intelligent Systems and Applications in …, 2024

snippet: … Grover utilizes a test set created by the free and open-source web crawler and archive Common Crawl [35]. As an example, a team of academics from Harvard and the MIT-IBM Watson laboratory developed the Giant language model test room, a …


title: ANALYSIS OF CROSS-LINGUAL MODELS FOR ASPECT-BASED SENTIMENT ANALYSIS: A CASE STUDY IN DUTCH AS A LOW-RESOURCE LANGUAGE.

snippet: This study addressed the limited availability of well-labeled data and research resources for end-to-end ABSA in low-resource languages, particularly Dutch. The study aims to tackle this challenge by investigating the performance of state-of-the-art …


title: AnchorAL: Computationally Efficient Active Learning for Large and Imbalanced Datasets

authors: P Lesci, A Vlachos - arXiv preprint arXiv:2404.05623, 2024

snippet: Active learning for imbalanced classification tasks is challenging as the minority classes naturally occur rarely. Gathering a large pool of unlabelled data is thus essential to capture minority instances. Standard pool-based active learning is …


title: Ancient Greek's New Technological Muse: Extracting Topoi in the Anacreontea with LLMs

authors: RO Nunes, JG Zandoná, JV Maia, A Spritzer… - Seminário Integrado de …, 2024

snippet: … The pre-training corpora include texts from Wikipedia, the European Parliament Proceedings Parallel Corpus, and OSCAR, a cleansed version of Common Crawl Researchers have also introduced a pilot study focused on the automatic linguistic …


title: and S. Abirami

authors: RS Gulecha, SKNR Subramanian - Speech and Language Technologies for Low …

snippet: In recent years, the Internet and social media have sparked a revolution in the way information is exchanged. The growth of social media and micro-blogging sites not only provides platforms for empowering freedom of expression and individual voices …


title: Answer Agnostic Question Generation in Bangla Language

authors: AR Fahad, N Al Nahian, MA Islam, RM Rahman - International Journal of Networked …, 2024

snippet: … mT5 is a massively multilingual model pre-trained using fresh Common Crawl-based data that includes 101 languages [14]. Another model, BanglaT5, a Transformer model for the Bangla language, was trained using a 27.5 GB clean corpus of Bangla …


title: AntiPhishStack: LSTM-based Stacked Generalization Model for Optimized Phishing URLs Detection

authors: S Aslam, H Aslam, A Manzoor, C Hui, A Rasool - 2024

snippet: The escalating reliance on revolutionary online web services has introduced heightened security risks, with persistent challenges posed by phishing despite extensive security measures. Traditional phishing systems, reliant on machine …


title: AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling

authors: J Zhan, J Dai, J Ye, Y Zhou, D Zhang, Z Liu, X Zhang… - arXiv preprint arXiv …, 2024

snippet: We introduce AnyGPT, an any-to-any multimodal language model that utilizes discrete representations for the unified processing of various modalities, including speech, text, images, and music. AnyGPT can be trained stably without any …


title: APLIKASI CHATGPT DALAM BAHASA MELAYU: ISU DAN CADANGAN PENYELESAIAN: OPERATING CHATGPT IN MALAY: ISSUES AND PROPOSED …

authors: H Nomoto, D Moeljadi, FAA Razak - RENTAS: Jurnal Bahasa, Sastera dan Budaya, 2024

snippet: … bekerjasama dengan kerajaan Brunei dan Singapura, meminta OpenAI dan Common Crawl, iaitu sumber utama set data latihan ChatGPT, … .my yang disenaraikan dalam 500 laman web teratas yang berdaftar di Common Crawl.Untuk …


title: Application of an Improved Convolutional Neural Network Algorithm in Text Classification

authors: J Peng, S Huo - Journal of Web Engineering, 2024

snippet: This paper proposes a text classification model based on a combination of a convolutional neural network (CNN) and a support vector machine (SVM) using Amazon review polarity, TREC, and Kaggle as experimental data. By adding an …


title: Application of Open-source Large Language Model (LLM) for Simulation of a Vulnerable IoT System and Cybersecurity Best Practices Assistance

snippet: This paper explores the role of open-source large language models in IoT cybersecurity world. The threats of malicious activity on the Internet and the loss of private information are very real and lead to serious consequences. The purpose of …


title: Applications of AI in Computer Vision and NLP

authors: SB Rajasekaran - Journal of Artificial Intelligence & Cloud Computing …, 2023

snippet: … In language AI, there are several commonly used datasets such as Common Crawl, Wikipedia, and OpenWebText. However, creating and maintaining these datasets can also be a challenging task, as it needs a big amount of work and it can …


title: Applying German word vectors to assess flexibility performance in the associative fluency task.

authors: M Camenzind, M Single, SM Gerber, T Nef, CL Bassetti… - Psychology of Aesthetics …, 2024

snippet: … One model was trained on common crawl, a nonprofit organization crawling the web and making resulting data publicly available, and the other was trained on Wikipedia. The vectors were trained using the CBOW algorithm with a special focus …


title: Applying Named Entity Recognition and Graph Networks to Extract Common Interests from Thematic Subfora on Reddit

authors: J Sawicki, M Ganzha, M Paprzycki, Y Watanobe - Applied Sciences, 2024

snippet: … , accessed on 13 February 2024) (pre-trained BERT large models fine-tuned on the CoNLL-2003 dataset) and flair/ner-english-large (https://huggingface.co/flair/ner-english-large, accessed on 13 February 2024) (an XLM-RoBERTa [89] pre-trained on a cleaned …


title: Applying Transfer Learning to German Metaphor Prediction

authors: M Berger, SM Reimann, NM Kiwitt - Proceedings of the 2024 Joint International …, 2024

snippet: … -scores using CommonCrawl embeddings (F1 60%) and News Commentary embeddings (F1 58%) while CommonCrawl contains over 2 m… 4), we can see that the bilingual embeddings approach, especially using Europarl and CommonCrawl …


title: Arabic sarcasm detection: An enhanced fine-tuned language model approach

authors: MA Galal, AH Yousef, HH Zayed, W Medhat - Ain Shams Engineering Journal, 2024

snippet: Sarcasm is a complex linguistic phenomenon involving humor, criticism, or phrases that convey the opposite meaning, mask true feelings, and play pivotal roles in various aspects of communication. Therefore, identifying sarcasm is essential for …


title: Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Models

authors: L Merrick, D Xu, G Nuti, D Campos - arXiv preprint arXiv:2405.05374, 2024

snippet: This report describes the training dataset creation and recipe behind the family of \texttt{arctic-embed} text embedding models (a set of five models ranging from 22 to 334 million parameters with weights open-sourced under an Apache-2 license). At the time of …


title: Are Human Conversations Special? A Large Language Model Perspective

authors: T Jawale, C Animesh, S Vallath, K Talamadupula… - arXiv preprint arXiv …, 2024

snippet: … We analyze the web data from CommonCrawl[21] dumps for human conversation data and the types of conversations in Table 1. We randomly sample a subset of the dump and deduplicate it so that it can be used to approximate the data distribution …


title: Are LLMs Robust for Spoken Dialogues?

authors: SM Mousavi, G Roccabruna, S Alghisi, M Rizzoli… - arXiv preprint arXiv …, 2024

snippet: … The LLM finetuned for this task is T5 Small [21], (12 layers, 60M parameters), a transformer-based encoder-decoder model, pre-trained on the Common Crawl dataset with 750GB of web page text. The fine-tuning was performed following the …


title: Artificial Intelligence and the Law of Machine-Readability: A Review of Human-to-Machine Communication Protocols and their (In) Compatibility with Article 4 (3) of the …

authors: H Hamann - JIPITEC–Journal of Intellectual Property, Information …, 2024

snippet: Many legal scholars critique the supposed ineffectiveness of European copyright regulation regarding commercial text and data mining. At the same time, tech-savvy entrepreneurs keep proposing new standards to effectuate them at a rate that has …


title: Artificial Intelligence as the Next Front in the Class War

snippet: For many years, artificial intelligence has been confined to the realm of science fiction, and while the technology has been in development, predicting the effects AI will have on our society has been a challenging endeavor. The release of ChatGPT …


title: Artificial Intelligence for Anesthesiology Board-style Examination Questions: Role of Large Language Models

authors: AA Khan, R Yunus, M Sohail, TA Rehman, S Saeed… - Journal of Cardiothoracic …, 2024

snippet: Background : New artificial intelligence tools have been developed that have implications for medical usage. Large language models such as the widely used ChatGPT, developed by OpenAI, have not been explored in the context of …


snippet: Springer MRW: [AU:, IDX:] Page 1 Artificial Intelligence in the Media Economy: A Systematic Review of Use Cases, Application Potentials, and Challenges of Generative Language Models Tim Prien and Klaus Goldhammer Contents 1 …


title: Artificial intelligence in the news: how AI retools, rationalizes, and reshapes journalism and the public arena

snippet: Despite growing interest, the effects of AI on the news industry and our information environment — the public arena — remain poorly understood. Insufficient attention has also been paid to the implications of the news industry’s dependence on …


title: Artificial Intelligence Integration, Concerns, Benefits, and the Need for Ethical Policies for Community Foundations and Nonprofit Organizations

snippet: This study describes the nature and extent of Artificial Intelligence (AI) integration in philanthropic organizations and assesses the capacity and readiness of nonprofits in adopting AI for philanthropic use. Additionally, the study explores the nature and …


title: Artificial Intelligence Supporting Independent Student Learning: An Evaluative Case Study of ChatGPT and Learning to Code

authors: K Hartley, M Hayak, UH Ko - Education Sciences, 2024

snippet: Artificial intelligence (AI) tools like ChatGPT demonstrate the potential to support personalized and adaptive learning experiences. This study explores how ChatGPT can facilitate self-regulated learning processes and learning computer programming …


title: Artificial Intelligence, Large Language Models, and Design Thinking in TPC Classrooms

authors: C Masters-Wheeler, J Bay, P Sullivan - Programmatic Perspectives, 2023

snippet: … Common Crawl, for instance, is an open repository of web data that is accessible to anyone and can be used to train AI models. Large language models (LLMs) use the textual data from sources like Common Crawl and other freely accessible data …


title: Asking GPT for the Ordinary Meaning of Statutory Terms

authors: C Engel, RH McAdams - MPI Collective Goods Discussion Paper, 2024

snippet: We report on our test of the Large Language Model (LLM) ChatGPT (GPT) as a tool for generating evidence of the ordinary meaning of statutory terms. We explain why the most useful evidence for interpretation involves a distribution of replies rather …


title: Asking Questions Framework for Oral History Archives

authors: J Švec, M Bulín, A Frémund, F Polák - European Conference on Information Retrieval, 2024

snippet: … The resulting Wav2Vec 2.0 end-to-end model was combined with a lower-case four-gram language model estimated from CommonCrawl data. To decrease the model’s … The model was pre-trained on a self-supervised text-restoration task …


title: Assessing and Optimizing Large Language Models on Spondyloarthritis Multi-Choice Question Answering: Protocol for Enhancement and Assessment

authors: A Wang, Y Wu, X Ji, X Wang, J Hu, F Zhang, Z Zhang… - JMIR Research Protocols, 2024

snippet: Background Spondyloarthritis (SpA), a chronic inflammatory disorder, predominantly impacts the sacroiliac joints and spine, significantly escalating the risk of disability. SpA’s complexity, as evidenced by its diverse clinical presentations and symptoms …


title: Assessing the potential of leveraging LLaMA-2 to create an institute-specific online chatbot assistant.

snippet: With the increasing popularity of chatbots such as ChatGPT, when faced with questions, people are getting more and more inclined to opt for a chatbot instead of browsing, calling, or travelling and waiting in line at service points. 8 To address this …


title: Assessing the research landscape and clinical utility of large language models: a scoping

authors: YJ Park, A Pillai, J Deng, E Guo, M Gupta, M Paget… - 2024

snippet: … Introduced in November 2022, ChatGPT was trained using a large corpora of unlabelled text, including CommonCrawl, WebText, and Wikipedia, as well as internet-based book corpora spanning multiple languages [3]. GPT, along with other …


title: ASSESSMENT OF HARMFULNESS OF INSTRUCTION-TUNED LARGE LANGUAGE MODEL LLAMA 2 BY SELF-EVALUATION

snippet: The relatively recent introduction of public large language models like ChatGPT has raised numerous concerns regarding the ethical issues surrounding their application. Their general proficiency in performing natural language tasks, rivaling that of …


title: Asymmetric Polysemous Reasoning for Image-Text Matching

authors: H Zhang, M Yang - 2023 IEEE International Conference on Data Mining …, 2023

snippet: Image-text matching has received growing interest since it bridges vision and language. The key challenge lies in how to learn correspondence between image and text. Upon observation, we find existing works suffer from two limitations. Firstly …


title: Asynchronous Local-SGD Training for Language Modeling

authors: B Liu, R Chhaparia, A Douillard, S Kale, AA Rusu… - arXiv preprint arXiv …, 2024

snippet: Local stochastic gradient descent (Local-SGD), also referred to as federated averaging, is an approach to distributed optimization where each device performs more than one SGD update per communication. This work presents an empirical …


title: Atmospheric Limitations for High-frequency Ground-based Very Long Baseline Interferometry

authors: DW Pesce, L Blackburn, R Chaves, SS Doeleman… - The Astrophysical Journal, 2024

snippet: Very long baseline interferometry (VLBI) provides the highest-resolution images in astronomy. The sharpest resolution is nominally achieved at the highest frequencies, but as the observing frequency increases, so too does the atmospheric contribution …


title: AtomGPT: Atomistic Generative Pre-trained Transformer for Forward and Inverse Materials Design

authors: K Choudhary - arXiv preprint arXiv:2405.03680, 2024

snippet: Large language models (LLMs) such as generative pretrained transformers (GPTs) have shown potential for various commercial applications, but their applicability for materials design remains underexplored. In this article, we introduce AtomGPT, a …


title: Auditing Large Language Models for Enhanced Text-Based Stereotype Detection and Probing-Based Bias Evaluation

authors: Z Wu, S Bulathwela, M Perez-Ortiz, AS Koshiyama - arXiv preprint arXiv:2404.01768, 2024

snippet: Recent advancements in Large Language Models (LLMs) have significantly increased their presence in human-facing Artificial Intelligence (AI) applications. However, LLMs could reproduce and even exacerbate stereotypical outputs from …


title: Augmenting a Spanish clinical dataset for transformer-based linking of negations and their out-of-scope references

authors: AJ Tamayo-Herrera, DA Burgos, A Gelbukh - Natural Language Processing

snippet: A negated statement consists of three main components: the negation cue, the negation scope, and the negation reference. The negation cue is the indicator of negation, while the negation scope defines the extent of the negation. The negation …


title: Augmenting Knowledge-Based Conversational Search Systems With Large Language Models

snippet: Conversational interfaces are increasingly used by websites and apps, including office software as well as creativity tools. This trend reflects a shift towards natural language in human-computer interaction, which is also facilitated by utilizing Large …


title: Automated Risk Analysis for Construction Contracts Using Neural Networks

authors: K Hamdy, I AbdelRasheed, YAS Essawy… - Journal of Legal Affairs and …, 2024

snippet: … for requiring less training time compared to previous recurrent neural architectures, such as long short-term memory (LSTM), and has been prevalently adopted for training large language models on large (language) data sets, such as the …


title: Automated Text Identification on Languages of the Iberian Peninsula: LLM and BERT-based Models Aggregation

snippet: This paper describes our solution approach for the IberAuTexTification (Automated Text Identification on Languages of the Iberian Peninsula) competition held as part of the IberLEF 2024 conference. Machinegenerated text fragments can be spotted in …


title: AutoMathText: Autonomous Data Selection with Language Models for Mathematical Texts

authors: Y Zhang, Y Luo, Y Yuan, ACC Yao - arXiv preprint arXiv:2402.07625, 2024

snippet: To improve language models' proficiency in mathematical reasoning via continual pretraining, we introduce a novel strategy that leverages base language models for autonomous data selection. Departing from conventional supervised fine-tuning or …


title: Automatic Coding of Contingency in Child-Caregiver Conversations

authors: A Agrawal, M Nikolaus, B Favre, A Fourtassi - 2024

snippet: One of the most important communicative skills children have to learn is to engage in meaningful conversations with people around them. At the heart of this learning lies the mastery of contingency, ie, the ability to contribute to an ongoing exchange …


title: Automatic Control With Human-Like Reasoning: Exploring Language Model Embodied Air Traffic Agents

authors: J Andriuškevičius, J Sun - arXiv preprint arXiv:2409.09717, 2024

snippet: … Most of the data used to train leading-edge large language models, such as the latest Common Crawl dataset [5], which comprises over 250 billion web pages, sources information from publicly accessible internet sites. This extensive training …


title: Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approach

authors: HV Vo, V Khalidov, T Darcet, T Moutakanni… - arXiv preprint arXiv …, 2024

snippet: … By filtering Common Crawl data, the authors managed to obtain large datasets to train reliable … To this end, we apply our curation pipeline to two text pools based on Common Crawl. The first … Doing so keeps the original data distribution closer to …


title: Automatic detection of hate speech in code-mixed Indian languages in twitter social media interaction using DConvBLSTM-MuRIL ensemble method

authors: P Kakati, D Dandotiya - Social Network Analysis and Mining, 2024

snippet: … During the pre-training process, 2.5 terabytes of Common Crawl data in 100 languages are used. This model gives a much better performance than multilingual BERT when code-switching, especially for languages with little resources, because …


title: Automatic Ellipsis Reconstruction in Coordinated German Sentences Based on Text-to-Text Transfer Transformers

authors: M Schmidt, K Harbusch, D Memmesheimer - International Conference on Text …, 2024

snippet: Ellipsis reconstruction, ie, revealing omitted syntactically obligatory words in a sentence, is still a challenging task in Natural Language Processing (NLP) technologies, even though this information is essential for advanced human-computer …


title: Automatic Extraction of User-Centric Aspects for Tourist Spot Recommender Systems Using Reviews in Japanese

authors: F Uwano, R Kobayashi, M Ohta - International Conference on Human-Computer …, 2024

snippet: In tourist reviews, various pieces of information are described to confirm the characteristics of tourist spots. This paper proposes a method to extract tourist spot aspects using Japanese tourist reviews automatically. The aspects of tourist spots …


title: Automatic language ability assessment method based on natural language processing

authors: N Nnamoko, T Karaminis, J Procter, J Barrowclough… - Natural Language …, 2024

snippet: Background and Objectives: The Wechsler Abbreviated Scales of Intelligence second edition (WASI-II) is a standardised assessment tool that is widely used to assess cognitive ability in clinical, research, and educational settings. In one of the …


title: Automatic Personality Recognition via XLNet with Refined Highway and Switching Module for Chatbot

authors: OTC Chen, CH Tsai, MH Ha - 2024 IEEE International Symposium on Circuits and …, 2024

snippet: This study introduces an Automatic Personality Recognition (APR) model, named XLNet refined-highway-switch network, comprising XLNet, refined highway units, a switching module, and a fully-connected layer. The pretrained XLNet is employed to …


title: Automatic Pull Request Description Generation Using LLMs: A T5 Model Approach

authors: MN Sakib, MA Islam, MM Arifin - arXiv preprint arXiv:2408.00921, 2024

snippet: … One reason T5 excels at producing high-quality PR descriptions is its training on the diverse ”Colossal Clean Crawled Corpus” (C4), a cleaned and structured version of the Common Crawl web dump. The C4 dataset, which encompasses 250 …


title: Automatické vytváření souhrnů historických dokumentů

snippet: … The dataset is comprised of a large amount of cleaned English text extracted from Common Crawl … The results show that the pre-training has 1https://commoncrawl.org/ … It uses the same pretraining objective as BART and it uses a multilingual Common …


title: Automating Code Adaptation for MLOps--A Benchmarking Study on LLMs

authors: H Patel, BA Ramanan, MA Khan, T Williams… - arXiv preprint arXiv …, 2024

snippet: This paper explores the possibilities of the current generation of Large Language Models for incorporating Machine Learning Operations (MLOps) functionalities into ML training code bases. We evaluate the performance of OpenAI (gpt-3.5-turbo) and …


title: Automating String Encoding in AutoML

snippet: Automated Machine Learning (AutoML) systems often struggle with data sets with string data. Due to their unstructured nature and dependence on domain knowledge, string data are a frequent cause of errors and coming up with generalized ways of …


title: Automation and Optimisation of Seo Content Gap Analysis Task with Semantic Clustering of Portuguese Search Keywords

snippet: In the field of digital marketing, efficient content gap analysis is crucial for developing effective SEO strategies. Traditional approaches to this task are time-consuming, representing a challenge for Organic Performance teams, who must process large …


title: Autonomous Data Selection with Language Models for Mathematical Texts

authors: Y Zhang, Y Luo, Y Yuan, AC Yao - ICLR 2024 Workshop on Navigating and …, 2024

snippet: To improve language models’ proficiency in mathematical reasoning via continual pretraining, we introduce a novel strategy that leverages base language models for autonomous data selection. Departing from conventional supervised fine-tuning or …


title: AutoScale: Automatic Prediction of Compute-optimal Data Composition for Training LLMs

authors: F Kang, Y Sun, B Wen, S Chen, D Song, R Mahmood… - arXiv preprint arXiv …, 2024

snippet: … With more compute, data sources with diverse examples, such as CommonCrawl, demonstrate continued reductions in training loss even at … With more compute, data sources with diverse examples, such as CommonCrawl, continue to reduce …


title: BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models

authors: J Xue, M Zheng, Y Hu, F Liu, X Chen, Q Lou - arXiv preprint arXiv:2406.00083, 2024

snippet: Large Language Models (LLMs) are constrained by outdated information and a tendency to generate incorrect data, commonly referred to as "hallucinations." Retrieval-Augmented Generation (RAG) addresses these limitations by combining …


title: BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and Deduplication by Introducing a Competitive Large Language Model Baseline

authors: G Dong, D Pan, Y Sun, S Zhang, Z Liang, X Wu… - arXiv preprint arXiv …, 2024

snippet: … We collect 94 publicly available batches of CommonCrawl spanning the past decade. We construct our WARC extraction and processing pipeline, considering the effectiveness and cost of the WET and WARC formats. … https://commoncrawl.org …


title: Bailong: Bilingual Transfer Learning based on QLoRA and Zip-tie Embedding

authors: LC Chen, ZR Li - arXiv preprint arXiv:2404.00862, 2024

snippet: … Owing to computational constraints, we further conducted additional sampling of the OSCAR dataset and the Common Crawl dataset to mitigate the issue related to dataset size. In the long run, there are around 13 billion tokens left in the corpus. The …


title: Balanced Data Sampling for Language Model Training with Clustering

authors: Y Shao, L Li, Z Fei, H Yan, D Lin, X Qiu - arXiv preprint arXiv:2402.14526, 2024

snippet: Data plays a fundamental role in the training of Large Language Models (LLMs). While attention has been paid to the collection and composition of datasets, determining the data sampling strategy in training remains an open question. Most …


title: Balancing Multilingual Model Training Data Using Exponential Smoothing

snippet: … The XLM-R [9] model is pre-trained over a larger common crawl corpus similarly to MBERT, using language sampling according to exponentially-smoothed language probabilities. Again, performance on low-resources tasks seems to benefit …


title: BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts

authors: Q Zhang, N Gritsch, D Gnaneshwar, S Guo, D Cairuz… - Workshop on Efficient Systems for …

snippet: Training Mixture of Experts (MoEs) from scratch in a large-scale regime is expensive. Previous work addresses this challenge by independently training multiple dense expert models and using them to initialize an MoE. In particular, initializing MoE …


title: Bangla Emergency Post Classification on Social Media using Transformer Based BERT Models

authors: AA Nabil, D Das, MS Salim, S Arifeen, HMA Fattah - 2023 6th International …, 2023

snippet: … It is a large multi-lingual language model trained on 2.5TB of filtered Common Crawl data. XLMRoBERTa is a multilingual model trained in 100 different languages. Unlike some XLM multilingual models, it does not require lang tensors to understand …


title: BeeManc at the PLABA Track of TAC-2023: Investigating LLMs and Controllable Attributes for Improving Biomedical Text Readability

authors: Z Li, S Belkadi, N Micheletti, L Han, M Shardlow… - arXiv preprint arXiv …, 2024

snippet: In this system report, we describe the models and methods we used for our participation in the PLABA2023 task on biomedical abstract simplification, part of the TAC 2023 tracks. The system outputs we submitted come from the following three …


title: Bella Turca: A Large-Scale Dataset of Diverse Text Sources for Turkish Language Modeling

authors: D Altinok - International Conference on Text, Speech, and …, 2024

snippet: … To meet the data requirements for language modeling, earlier large language models (LLMs) heavily relied on the Common Crawl dataset as their primary or sole source of training data [5, 26]. While training on the Common Crawl has proven …


title: Benchmarking PathCLIP for Pathology Image Analysis

authors: S Zheng, X Cui, Y Sun, J Li, H Li, Y Zhang, P Chen… - arXiv preprint arXiv …, 2024

snippet: … All these versions are trained using masked language modeling and next-word prediction techniques with training data sourced from publicly available repositories like Wikipedia, Common Crawl. The results of LLaMA demonstrate that cuttingedge …


title: Benchmarking zero-shot stance detection with FlanT5-XXL: Insights from training data, prompting, and decoding strategies into its near-SoTA performance

authors: R Aiyappa, S Senthilmani, J An, H Kwak, YY Ahn - arXiv preprint arXiv:2403.00236, 2024

snippet: We investigate the performance of LLM-based zero-shot stance detection on tweets. Using FlanT5-XXL, an instruction-tuned open-source LLM, with the SemEval 2016 Tasks 6A, 6B, and P-Stance datasets, we study the performance and its variations …


title: BengaliLCP: A Dataset for Lexical Complexity Prediction in the Bengali Texts

authors: N Ayman, MA Hossain, A Aziz, RU Faruqui, AN Chy - Proceedings of the 2024 Joint …, 2024

snippet: Encountering intricate or ambiguous terms within a sentence produces distress for the reader during comprehension. Lexical Complexity Prediction (LCP) deals with predicting the complexity score of a word or a phrase considering its context. This …


title: BERGEN: A Benchmarking Library for Retrieval-Augmented Generation

authors: D Rau, H Déjean, N Chirkova, T Formal, S Wang… - arXiv preprint arXiv …, 2024

snippet: Retrieval-Augmented Generation allows to enhance Large Language Models with external knowledge. In response to the recent popularity of generative LLMs, many RAG approaches have been proposed, which involve an intricate number of …


title: BERT, RoBERTa or DeBERTa? Comparing Performance Across Transformer Models in Political Science Text

snippet: Transformer models such as BERT, RoBERTa, and DeBERTa have revolutionized the field of Natural Language Processing in recent years with substantial improvements in the contextual understanding of text. While political scientists have …


title: BERT, RoBERTa or DeBERTa? Comparing Performance Across Transformers Models in Political Science Text


title: BERTicelli at HaSpeeDe 3: Fine-tuning and Cross-validating Large Language Models for Hate Speech Detection Leonardo Grotti¹², Patrick Quick¹ ¹Universiteit …

authors: L Grotti¹ - EVALITA Proceedings of the Eighth Evaluation …, 2024

snippet: The present paper describes the results from the experiments carried out for the HaSpeeDe 3 shared task, an Italian-language Hate Speech (HS) detection task, at EVALITA 2023. Two BERT-based language models were selected: UmBERTO (cased) …


title: BERTs are Generative In-Context Learners

authors: D Samuel - arXiv preprint arXiv:2406.04823, 2024

snippet: This paper explores the in-context learning capabilities of masked language models, challenging the common view that this ability does not 'emerge' in them. We present an embarrassingly simple inference technique that enables DeBERTa to operate as …


title: Better Alignment with Instruction Back-and-Forth Translation

authors: T Nguyen, J Li, S Oh, L Schmidt, J Weston… - arXiv preprint arXiv …, 2024

snippet: We propose a new method, instruction back-and-forth translation, to construct high-quality synthetic data grounded in world knowledge for aligning large language models (LLMs). Given documents from a web corpus, we generate and curate synthetic instructions …


title: Beyond Lexical Boundaries: LLM-Generated Text Detection for Romanian Digital Libraries

authors: M Nitu, M Dascalu - Future Internet, 2024

snippet: Machine-generated content reshapes the landscape of digital information; hence, ensuring the authenticity of texts within digital libraries has become a paramount concern. This work introduces a corpus of approximately 60 k Romanian documents …


title: Beyond Metrics: Evaluating LLMs' Effectiveness in Culturally Nuanced, Low-Resource Real-World Scenarios

authors: M Ochieng, V Gumma, S Sitaram, J Wang, K Ronen… - arXiv preprint arXiv …, 2024

snippet: The deployment of Large Language Models (LLMs) in real-world applications presents both opportunities and challenges, particularly in multilingual and code-mixed communication settings. This research evaluates the performance of seven leading …


title: Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compression

authors: Z Xu, A Gupta, T Li, O Bentham, V Srikumar - arXiv preprint arXiv:2407.04965, 2024

snippet: Large language models (LLMs) are increasingly deployed in real-world scenarios with the help of recent model compression techniques. Such momentum towards local deployment means the use of compressed LLMs will widely impact a large …


title: Beyond Scale: The Diversity Coefficient as a Data Quality Metric for Variability in Natural Language Data

authors: A Lee, B Miranda, S Sundar, A Casasola, S Koyeyo

snippet: … Since many datasets are publicly available (eg Common Crawl, Wikipedia), data used to train new models may be curated from such … We evaluate on Pile-CC (Pile Common Crawl) and OpenWebText2 since those datasets align with intuitively …


title: Beyond the Black Box: Optimization Within Latent Spaces

snippet: … We compute b using Common Crawl monolingual datasets.For each language and contextual embedding model, we create 1M candidate-reference pairs by grouping two random sentences. Because of the random pairing and the corpus …


title: BharatBhasaNet-A unified framework to identify Indian code mix Languages

authors: S Dey, S Thakur, A Kandwal, R Kumar, S Dasgupta… - IEEE Access, 2024

snippet: In the rapidly globalizing digital communication sphere, the imperative for advanced multilingual text recognition and identification is increasingly evident. Contrasting the previous works, which were predominantly constrained to 2-3 languages, this …


title: Bi-directional GRU-Based Approach for Multi-Class Text Error Identification System

authors: S Reeha, BV Nikith, GMP Reddy, PB Pati - 2024 IEEE 9th International Conference …, 2024

snippet: … The C4 dataset, utilized in this study, is an extensively cleaned and curated version of Common Crawl's web crawl corpus. Common Crawl is a publicly available dataset accessible at commoncrawl.org. and boasting over 700 million …


title: BIAS ASSESSMENT IN LARGE LANGUAGE MODELS

snippet: The introduction of Large Language Models (LLMs) such as Chat-GPT has transformed the field of natural language processing, providing exceptional capabilities in text generation and comprehension. However, these models have …


title: Bias-Aware Low-Rank Adaptation: Mitigating Catastrophic Inheritance of Large Language Models

authors: Y Chang, Y Chang, Y Wu - arXiv preprint arXiv:2408.04556, 2024

snippet: Large language models (LLMs) have exhibited remarkable proficiency across a diverse array of natural language processing (NLP) tasks. However, adapting LLMs to downstream applications typically necessitates computationally intensive and …


title: Big City Bias: Evaluating the Impact of Metropolitan Size on Computational Job Market Abilities of Language Models

authors: C Campanella, R van der Goot - arXiv preprint arXiv:2403.08046, 2024

snippet: Large language models (LLMs) have emerged as a useful technology for job matching, for both candidates and employers. Job matching is often based on a particular geographic location, such as a city or region. However, LLMs have known …


title: Bilingual Adaptation of Monolingual Foundation Models

authors: G Gosal, Y Xu, G Ramakrishnan, R Joshi, A Sheinin… - … on Foundation Models in the Wild

snippet: We present an efficient method for adapting a monolingual Large Language Model (LLM) to another language, addressing challenges of catastrophic forgetting and tokenizer limitations. We focus this study on adapting Llama 2 to Arabic. Our two-stage …


title: Bilingual Propaganda Detection in Diplomats' Tweets Using Language Models and Linguistic Features

authors: A Modzelewski, P Golik, A Wierzbicki - IberLEF@ SEPLN, 2024

snippet: Our study presents an approach to a shared task of propaganda identification and characterization at the DIPROMATS 2024 hosted by the Iberian Languages Evaluation Forum. As the DSHacker team, we participated in the propaganda …


title: Binary Battle: Leveraging Machine Learning and Transfer Learning Models to Distinguish between Conspiracy Theories and Critical Thinking

authors: S Mahesh, S Divakaran, K Girish, HL Shashirekha - 2024

snippet: In the context of automatic content moderation Natural Language Processing (NLP) has a complex task when it comes to distinguishing between conspiracy theories and critical thinking. While conspiracy theories present complex narratives …


title: Bioeconomy firms and where to find them

snippet: … This dataset uses the open-source web repository CommonCrawl to identify German company websites and has proven to be a valuable database for spatial research. From this data, we identify bioeconomy firms using a combination of …


title: BioMedLM: A 2.7 B Parameter Language Model Trained On Biomedical Text

authors: E Bolton, A Venigalla, M Yasunaga, D Hall, B Xiong… - arXiv preprint arXiv …, 2024

snippet: … The Pile contains multiple sub-corpora, including general content from Common Crawl and specialized content like PubMed, Github, and FreeLaw. This model serves as an important baseline for comparison with our domain-specific models; …