title: Towards More Factual Large Language Models: Parametric and Non-parametric Approaches

snippet: Large language models (LLMs) are increasingly important in assisting people to access information, ranging from simple factoid questions such as “where is the world’s largest ice sheet located” to complex questions that require accessing real-time …


title: Towards Privacy Preserving LLMs Training

authors: B Buesser - Large Language Models in Cybersecurity: Threats …, 2024

snippet: Privacy-preserving training of machine learning models aims to avoid or minimize (mitigate) the exact or similar reproduction (leakage) of information contained in the training data. This chapter introduces pre-processing methods (filtering and de-duplication) …


title: Towards quantifying politicization in foreign aid project reports

authors: S Wang, G Eggers, ART Georgiadis, TA Đo, L Gontard… - Proceedings of the Second …, 2024

snippet: We aim to develop a metric of politicization by investigating whether this concept can be operationalized computationally using document embeddings. We are interested in measuring the extent to which foreign aid is politicized. Textual reports of foreign …


title: Towards Region-aware Bias Evaluation Metrics

authors: A Borah, A Garimella, R Mihalcea - arXiv preprint arXiv:2406.16152, 2024

snippet: … To measure biases in different data domains and regions, we extract region-aware topics using the GeoWAC dataset which spans Common Crawl separated by regions, and create a WEAT-style evaluation setup using these topics. Data. We consider two …


title: Towards reliability and interactive debugging for large language models

snippet: Large language models (LLMs) have permeated our everyday lives and are used in critical decision-making scenarios that can affect millions of people. Despite their impressive progress, model deficiencies may result in exacerbating harmful biases …


title: Towards Robust Evaluation: A Comprehensive Taxonomy of Datasets and Metrics for Open Domain Question Answering in the Era of Large Language Models

authors: A Srivastava, A Memon - arXiv preprint arXiv:2406.13232, 2024

snippet: Open Domain Question Answering (ODQA) within natural language processing involves building systems that answer factual questions using large-scale knowledge corpora. Recent advances stem from the confluence of several factors …


title: Towards Systematic Monolingual NLP Surveys: GenA of Greek NLP

authors: J Bakagianni, K Pouli, M Gavriilidou, J Pavlopoulos - arXiv preprint arXiv:2407.09861, 2024

snippet: Natural Language Processing (NLP) research has traditionally been predominantly focused on English, driven by the availability of resources, the size of the research community, and market demands. Recently, there has been a noticeable shift …


title: Towards the automatic creation of NER systems for new domains

authors: E Matos, M Rodrigues, A Teixeira - Proceedings of the 16th International Conference …, 2024

snippet: … ner-roberta (ROBERTA): xlm-roberta-base-trimmed-pt-600002 model based in xlm-roberta-base, pretrained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Unsupervised Crosslingual …


title: Towards Video Understanding through Language in Real-life Settings

snippet: Videos have become an integral part of our daily lives, with a rapidly growing number on YouTube, Netflix, and TikTok serving as testimony to their widespread popularity. Behind the simplicity of their interfaces and user experiences, the …


title: Toxicity Detection for Free

authors: Z Hu, J Piet, G Zhao, J Jiao, D Wagner - arXiv preprint arXiv:2405.18822, 2024

snippet: … An analysis of undesirable content in the common crawl corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2 …


title: Training and evaluation of vector models for Galician

authors: M Garcia - Language Resources and Evaluation, 2024

snippet: This paper presents a large and systematic assessment of distributional models for Galician. To this end, we have first trained and evaluated static word embeddings (eg, word2vec, GloVe), and then compared their performance with that of current …


title: Training BERT Models to Carry over a Coding System Developed on One Corpus to Another

authors: D Galambos, P Zsamboki - Proceedings of the 2024 Joint International Conference …, 2024

snippet: This paper describes how we train BERT models to carry over a coding system developed on the paragraphs of a Hungarian literary journal to another. The aim of the coding system is to track trends in the perception of literary translation around …


title: Training Data for the Price of a Sandwich1

snippet: … , this report sheds light on the implications of Common Crawl’s popularity by exploring the values that guide Common Crawl itself in-depth. … ) whether Common Crawl’s data had been used and b) what specific version of Common Crawl was …


title: Training Task Experts through Retrieval Based Distillation

authors: J Ge, X Jia, V Viswanathan, H Luo, G Neubig - arXiv preprint arXiv:2407.05463, 2024

snippet: One of the most reliable ways to create deployable models for specialized tasks is to obtain an adequate amount of high-quality task-specific data. However, for specialized tasks, often such datasets do not exist. Existing methods address this by …


title: Transfer Learning in the Era of Large Language Models

snippet: Natural language processing has made great strides in recent years, largely driven by developments in large-scale neural networks trained on large quantities of textual data. This thesis is a collection of work studying transfer learning—the transfer of …


title: Transformer-Based Named Entity Recognition Model-Tamil Language Check for updates

authors: K Dhayalan, N Sultanova, J Mustafina, P Daud - Data Science and Emerging Technologies …

snippet: This work presents different transformer-based approaches to the named entity recognition problem in the Tamil language. The WikiANN-ta dataset is used for training, validating and testing the model. Person, organization and location tags are …


title: Transformer-Driven Models for Language, Vision, and Multimodality

authors: M Luo, T Gokhale, N Varshney, Y Yang, C Baral - Advances in Multimodal Information …, 2024

snippet: In this chapter, we will learn about the modeling and learning techniques that drive multimodal applications. We will focus specifically on the recent advances in transformer-based modeling for natural language understanding, and image …


title: Transformers and large language models in healthcare: A review

authors: S Nerella, S Bandyopadhyay, J Zhang, M Contreras… - Artificial Intelligence in …, 2024

snippet: With Artificial Intelligence (AI) increasingly permeating various aspects of society, including healthcare, the adoption of the Transformers neural network architecture is rapidly changing many applications. Transformer is a type of deep learning …


title: Transformers and tradition: using Generative AI and Deep Learning for financial markets prediction

snippet: Artificial intelligence has revolutionized numerous industries, and financial markets are no exception. With the ability to process vast amounts of data quickly and accurately, AI algorithms have been increasingly used in finance to predict stock …


title: Transformers at HSD-2Lang 2024: Hate Speech Detection in Arabic and Turkish Tweets Using BERT Based Architectures

authors: K Singhal, J Bedi - Proceedings of the 7th Workshop on Challenges and …, 2024

snippet: Over the past years, researchers across the globe have made significant efforts to develop systems capable of identifying the presence of hate speech in different languages. This paper describes the team Transformers’ submission to the subtasks …


title: Transformers for Detection of Distressed Cardiac Patients with an ICD Based on Danish Text Messages

authors: JDW Andersen, ML Jensen, UK Wiil, S Skovbakke… - 2023 IEEE International …, 2023

snippet: Cardiac patients with implantable cardioverter defibrillator devices frequently exhibit signs of anxiety and depression (termed "Distressed"). Early detection of these patients is vital for evaluation, intervention, and prevention against relapse …


title: Transformers on Dynamical Systems-An Exploration of In-context Learning

snippet: Large Language Models (LLMs) have shown to be highly effective at performing in-context learning, where, given a prompt, the model can learn from the prompt and complete the sequence without needing to perform additional gradient steps or fine-tuning. In …


title: Transformers@ DravidianLangTech-EACL2024: Sentiment Analysis of Code-Mixed Tamil Using RoBERTa

authors: K Singhal, J Bedi - Proceedings of the Fourth Workshop on Speech, Vision …, 2024

snippet: In recent years, there has been a persistent focus on developing systems that can automatically identify the hate speech content circulating on diverse social media platforms. This paper describes the team Transformers’ submission to the Caste/Immigration …


title: Transformers@ LT-EDI-EACL2024: Caste and Migration Hate Speech Detection in Tamil Using Ensembling on Transformers

authors: K Singhal, J Bedi - Proceedings of the Fourth Workshop on Language …, 2024

snippet: In recent years, there has been a persistent focus on developing systems that can automatically identify the hate speech content circulating on diverse social media platforms. This paper describes the team “Transformers” submission to the Caste …


title: Transforming Conversational AI

snippet: We were motivated to write this book by the launch in November 2022 of ChatGPT and by the ensuing excitement and disruption across the world of Conversational AI. The book is written for a broad audience who are already working, starting to work …


title: Transforming Dutch: Debiasing Dutch Corefence Resolution Systems for Non-binary Pronouns

snippet: Gender-neutral pronouns are increasingly being introduced across Western languages, and are continuously more frequently being adopted by non-binary individuals. Recent evaluations have however demonstrated that English language …


title: TransGPT: Multi-modal Generative Pre-trained Transformer for Transportation

authors: P Wang, X Wei, F Hu, W Han - arXiv preprint arXiv:2402.07233, 2024

snippet: Natural language processing (NLP) is a key component of intelligent transportation systems (ITS), but it faces many challenges in the transportation domain, such as domain-specific knowledge and data, and multi-modal inputs and outputs. This …


title: Translate-Distill: Learning Cross-Language Dense Retrieval by Translation and Distillation

authors: E Yang, D Lawrie, J Mayfield, DW Oard, S Miller - arXiv preprint arXiv:2401.04810, 2024

snippet: Prior work on English monolingual retrieval has shown that a cross-encoder trained using a large number of relevance judgments for query-document pairs can be used as a teacher to train more efficient, but similarly effective, dual-encoder student …


title: Translating and predicting document structure for medical domain scientific abstracts

authors: SA Rauf, F Yvon - Computer Speech & Language, 2024

snippet: Abstract Machine Translation (MT) technologies have improved in many ways and generate usable outputs for a growing number of domains and language pairs. Yet, most sentence based MT systems struggle with contextual dependencies …


title: Translating QA is Enough: A Key to Unlocking In-context Cross-lingual Performance

authors: S Kim, D Ki, Y Kim, J Lee - ICML 2024 Workshop on In-Context Learning

snippet: Multilingual large language models (MLLMs) have demonstrated significant cross-lingual capabilities through in-context learning. Existing approaches typically construct monolingual few-shot examples, either in the source or target language. However …


title: Translating scientific abstracts in the bio-medical domain with structure-aware models

authors: SA Rauf, F Yvon - Computer Speech and Language, 2024

snippet: Machine Translation (MT) technologies have improved in many ways and generate usable outputs for a growing number of domains and language pairs. Yet, most sentence based MT systems struggle with contextual dependencies, processing …


title: Translation from Historical to Contemporary Japanese Using Japanese T5

authors: H Usui, K Komiya - Proceedings of the Joint 3rd International Conference …, 2023

snippet: This paper presents machine translation from historical Japanese to contemporary Japanese using a Text-to-Text Transfer Transformer (T5). The result of the previous study that used neural machine translation (NMT), Long Short Term Memory (LSTM) …


title: TransLSTM: A hybrid LSTM-Transformer model for fine-grained suggestion mining

authors: S Riaz, A Saghir, MJ Khan, H Khan, HS Khan, MJ Khan - Natural Language …, 2024

snippet: … Each model, excluding BERT, which employs BERT-BASE-Uncased, undergoes testing using simple text vectorization, GloVe, and Common Crawl. Notably, the GloVe word embedding surpasses common text vectorization and Common Crawl in …


title: TraQuLA: Transparent Question Answering Over RDF Through Linguistic Analysis

authors: E Zimina, K Järvelin, J Peltonen, A Ranta… - International Conference on …, 2024

snippet: Answering complex questions over knowledge graphs has gained popularity recently. Systems based on large language models seem to achieve top performance. However, these models may generate content that looks reasonable …


title: Treading water: new data on the impact of AI ethics information sessions in classics and ancient language pedagogy

authors: EAS Ross, J Baines - Journal of Classics Teaching, 2024

snippet: Over 2023, many universities and policy organisations in the higher education (HE) sector are working to create guiding principles and guidelines for the use of generative artificial intelligence (AI) in HE Teaching and Learning (T&L). Despite …


title: TriLM vs FloatLM: Ternary LLMs are more Performant than Quantized FP16 LLMs

authors: A Kaushal, T Vaidhya, T Pandey, A Bhagat, I Rish - … 2024 Workshop on Foundation Models in …

snippet: Ternary LLMs offer significantly better performance for their size (measured in bits) than the models trained and deployed in FP16/BF16. Given the widespread usage of quantization before deployment and advancements in Post Training Quantization …


title: Trustworthy and Responsible AI for Human-Centric Autonomous Decision-Making Systems

authors: F Dehghani, M Dibaji, F Anzum, L Dey, A Basdemir… - arXiv preprint arXiv …, 2024

snippet: Artificial Intelligence (AI) has paved the way for revolutionary decision-making processes, which if harnessed appropriately, can contribute to advancements in various sectors, from healthcare to economics. However, its black box nature …


title: Unboxing Occupational Bias: Grounded Debiasing LLMs with US Labor Data

authors: A Gorti, M Gaur, A Chadha - arXiv preprint arXiv:2408.11247, 2024

snippet: Large Language Models (LLMs) are prone to inheriting and amplifying societal biases embedded within their training data, potentially reinforcing harmful stereotypes related to gender, occupation, and other sensitive categories. This issue …


title: Understanding College Students' Satisfaction With ChatGPT: An Exploratory And Predictive Machine Learning Approach Using Feature Engineering

authors: K Pabreja, N Pabreja - MIER Journal of Educational Studies Trends and …, 2024

snippet: Artificial Intelligence (AI) technologies are continually improving and becoming more pervasive in many facets of our lives. ChatGPT is one such cutting-edge artificial intelligence application, and it has received a lot of worldwide media attention …


title: Understanding Emergent Abilities of Language Models from the Loss Perspective

authors: Z Du, A Zeng, Y Dong, J Tang - arXiv preprint arXiv:2403.15796, 2024

snippet: Recent studies have put into question the belief that emergent abilities in language models are exclusive to large models. This skepticism arises from two observations: 1) smaller models can also exhibit high performance on emergent abilities and 2) …


title: Understanding Large Language Models

snippet: Today, finding someone who hasn’t heard of ChatGPT, the AI chatbot that took the world by storm, is hard. ChatGPT—and its competitors such as Google Bard, Microsoft Bing Chat, etc.—are part of a broader area in AI known as large language …


title: Understanding latent affective bias in large pre-trained neural language models

authors: A Kadan, P Deepak, S Bhadra, MP Gangan, VL Lajish - Natural Language …, 2024

snippet: Groundbreaking inventions and highly significant performance improvements in deep learning based Natural Language Processing are witnessed through the development of transformer based large Pre-trained Language Models (PLMs). The …


title: Understanding LLMs: A Comprehensive Overview from Training to Inference

authors: Y Liu, H He, T Han, X Zhang, M Liu, J Tian, Y Zhang… - arXiv preprint arXiv …, 2024

snippet: … from the CommonCrawl. However, due to the presence of a substantial amount of low-quality data in web archives, preprocessing is essential when working with CommonCrawl data. Currently, four commonly used filtered datasets based on …


title: Understanding New Machine Learning Architectures: Practical Generative Artificial Intelligence for Anesthesiologists

snippet: Recent advances in neural networks have given rise to generative artificial intelligence, systems able to produce fluent responses to natural questions or attractive and even photorealistic images from text prompts. These systems were …


title: Understanding Privacy Risks of Embeddings Induced by Large Language Models

authors: Z Zhu, N Shao, D Lian, C Wu, Z Liu, Y Yang, E Chen - arXiv preprint arXiv …, 2024

snippet: … CC-News (CommonCrawl News dataset) is a dataset containing news articles from international news websites. It contains 708241 English-language news articles published between January 2017 and December 2019. Pile-PubMed is a …


title: Understanding the Interplay of Scale, Data, and Bias in Language Models: A Case Study with BERT

authors: M Ali, S Panda, Q Shen, M Wick, A Kobren - arXiv preprint arXiv:2407.21058, 2024

snippet: … We experiment with English Wikipedia and the CC-100 English subset of Common Crawl. For each model size and type of training data, we compute biases in the representations upstream and performance disparities downstream …


title: UNIVERSAL IMITATION GAMES

snippet: In 1950, Alan Turing proposed a framework called an imitation game in which the participants are to be classified Human or Machine solely from natural language interactions. Using mathematics largely developed since Turing–category theory–we …


title: University Students' Attitudes and Perceptions on Learning English with ChatGPT

authors: M Atakuziev, AM Gultom - Formosa Journal of Sustainable Research, 2024

snippet: This study examines 43 master's students' perceptions of ChatGPT in English language learning at Universitas Negeri Yogyakarta. Using a descriptive quantitative methodology, data was gathered via questionnaires and interviews …


title: Unlock the Power of Frozen LLMs in Knowledge Graph Completion

authors: B Xue, Y Xu, Y Song, Y Pang, Y Ren, J Ding, L Fu… - arXiv preprint arXiv …, 2024

snippet: Classical knowledge graph completion (KGC) methods rely solely on structural information, struggling with the inherent sparsity of knowledge graphs (KGs). Large Language Models (LLMs) learn extensive knowledge from large corpora with …


title: Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset

authors: H Laurençon, L Tronchon, V Sanh - arXiv preprint arXiv:2403.09029, 2024

snippet: … available online (and often captured in web crawls like Common Crawl) presents a tempting resource for generating pairs of screenshots and corresponding HTML codes by simply rendering the HTML and capturing the output. However, this …


title: UNLOCKING THE POWER OF LANGUAGE: IMPROVING MULTILINGUAL CAPABILITIES IN GENERATIVE AI FOR GLOBAL ACCESSIBILITY

snippet: Generative AI has emerged as a transformative technology with the potential to revolutionize global communication and accessibility. However, the current state of language support in generative AI applications presents both opportunities and …


title: Unmasking Memorization: Assessing Dutch Language Memorization in mT5 Models

snippet: This study investigates the memorization of Dutch language content in mT5 models, a multilingual variant of the T5 Transformer-based models. A fill-mask evaluation technique is used to assess memorization and how it varies across different model …


title: Unraveling the Mystery of Scaling Laws: Part I

authors: H Su, Z Tian, X Shen, X Cai - arXiv preprint arXiv:2403.06563, 2024

snippet: Scaling law principles indicate a power-law correlation between loss and variables such as model size, dataset size, and computational resources utilized during training. These principles play a vital role in optimizing various aspects of model pre-training …


title: Unseen but influential associates: Properties of words' associates influence lexical and semantic processing

authors: EJ Muraki, PM Pexman - Psychonomic Bulletin & Review, 2024

snippet: In many models of lexical and semantic processing, it is assumed that single word processing is a function of the characteristics of the words presented and the distributional properties of the words’ networks. Recent research suggests that …


title: Unsupervised approach for misinformation detection in Russia-Ukraine war news

authors: N Khairova, A Galassi, FL Scudo, B Ivasiuk, I Redozub - 2024

snippet: The Russian-Ukrainian war has attracted considerable global attention; however, fake news often obstructs the formation of public opinion and disseminates false information. To address this issue, we have curated the RUWA dataset, comprising …


title: Unsupervised LLM Adaptation for Question Answering

authors: K Saito, K Sohn, CY Lee, Y Ushiku - arXiv preprint arXiv:2402.12170, 2024

snippet: … Commoncrawl 2. The topics cover politics, sports, business, etc. To reduce the number of tokens, we ask ChatGPT to generate summaries. … We employ NewsFetch7 to crawl the news from Commoncrawl. The published date of the article …


title: Unsupervised Outlier Detection for Language-Independent Text Quality Filtering

authors: J Daðason, H Loftsson - Proceedings of the 3rd Annual Meeting of the Special …, 2024

snippet: … Common Crawl (CC) is an organization that maintains a massive repository of data crawled … corpora, all of which are derived from Common Crawl. It consists of 6.3 trillion tokens in 167 … crawled sources, such as Common Crawl. The total size …


title: Untangling the Unrestricted Web: Automatic Identification of Multilingual Registers

authors: E Henriksson, A Myntti, A Eskelinen… - arXiv preprint arXiv …, 2024

snippet: This article explores deep learning models for the automatic identification of registers - text varieties such as news reports and discussion forums - in web-based datasets across 16 languages. Web register (or genre) identification would provide a …


title: Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models

authors: J Tian, D Zheng, Y Cheng, R Wang, C Zhang, D Zhang - arXiv preprint arXiv …, 2024

snippet: … (2024), who emphasize the influence of data quality and diversity in training models, our curated dataset incorporates sources such as Common Crawl, books, Wikipedia, code, and academic papers. Additionally, our dataset is multilingual, with …


title: Unveiling Gender Bias in Large Language Models: Using Teacher's Evaluation in Higher Education As an Example

authors: Y Huang - arXiv preprint arXiv:2409.09652, 2024

snippet: This paper investigates gender bias in Large Language Model (LLM)-generated teacher evaluations in higher education setting, focusing on evaluations produced by GPT-4 across six academic subjects. By applying a comprehensive analytical …


title: UNVEILING THE COGNITIVE CAPACITY OF CHATGPT: ASSESSING ITS HUMAN-LIKE REASONING ABILITIES

snippet: … OpenAI utilizes a dataset termed Common Crawl, which is a collection of publicly available websites. This dataset includes billions of web pages and is one of the largest text datasets available. Meanwhile, this dataset is just a starting line. …


title: USDC: A Dataset of $\underline {U} $ ser $\underline {S} $ tance and $\underline {D} $ ogmatism in Long $\underline {C} $ onversations

authors: M Marreddy, SR Oota, VC Chinni, M Gupta, L Flek - arXiv preprint arXiv:2406.16833, 2024

snippet: Identifying user's opinions and stances in long conversation threads on various topics can be extremely critical for enhanced personalization, market research, political campaigns, customer service, conflict resolution, targeted advertising, and …


title: Use of Large Language Models to Aid Analysis of Textual Data

authors: LR Bentley, X Xia, JM Sitt, SC Fankhauser…

snippet: … The use of the data generated by the Common Crawl discussed earlier in this paper mitigates this limitation, drawing from the vast and growing body of content generated through the internet. The quality of the input data also plays a vital role in …


title: User Story Classification with Machine Learning and LLMs

authors: P Chuor, A Ittoo, S Heng - … Conference on Knowledge Science, Engineering and …, 2024

snippet: We address the problem of classifying Capability, Task, Hard-goal, and Soft-goal in user stories. Such a classification is essential for generating Rationale Tree. Several articles have attempted to classify different aspects of user stories in the past …


title: Using Large Language Models to Probe Cognitive Constructs, Augment Data, and Design Instructional Materials

authors: F Kieser, P Wulff - … in Educational Sciences: Approaches, Applications and …, 2024

snippet: … It was shown that transformer-based large language models (LLMs could be pretrained on large language data sets such as the Common Crawl of the Internet, the Books Corpus, or Wikipedia, and then further fine-tuned in specific tasks [9, 10] …


title: Using Large Language Models to Understand Leadership Perception and Expectation

authors: Y Zhang, X Wang, Z Zhang, X Wang, X Ma, Y Wu… - 2024 IEEE International …, 2024

snippet: Using multimodal approaches for measuring propositions in natural language (the Fill-Mask Association Test, FMAT) and ChatGPT 4.0, based on the Five-Factor Model of personality traits, we examined how leaders are perceived and expected in …


title: Using Large Language Models to Understand Telecom Standards

authors: A Karapantelakis, M Shakur, A Nikou, F Moradi… - arXiv preprint arXiv …, 2024

snippet: … They have billions of parameters and are trained on corpora such as Common Crawl [5]. For foundation models to adapt to application domains, such as telecommunications, prompt engineering and fine-tuning approaches are used …


title: Using LLMs for the Extraction and Normalization of Product Attribute Values

authors: A Brinkmann, N Baumann, C Bizer - … Conference on Advances in Databases and …, 2024

snippet: Product offers on e-commerce websites often consist of a product title and a textual product description. In order to enable features such as faceted product search or to generate product comparison tables, it is necessary to extract structured attribute-value …


title: Using LLMs for the Extraction

authors: ABDN Baumann, C Bizerb - Advances in Databases and Information Systems: 28th …


title: Using rhetorical strategies to design prompts: a human-in-the-loop approach to make AI useful

authors: N Ranade, M Saravia, A Johri - AI & SOCIETY, 2024

snippet: … It consumed the text content of the popular CommonCrawl dataset of web pages from 2016 to 2019. Although OpenAI curated it to eliminate redundancy and improve quality, it is nominally 45-terabytes worth of compressed text data. …


title: Utilizing Large Language Models for Non-trained Binary Sentiment Classification

snippet: Prompt Return a VALID JSON object with key “sentiment”. Return sentiment must be “positive” or “negative”. Do not say anything other than JSON object result. There is no “neutral”,“mixed” answer. Answer must start with ‘1’. Classify the sentiment of the …


title: Utilizing Large Language Models in Ophthalmology: The Current Landscape and Challenges

authors: P Chotcomwongse, P Ruamviboonsuk, A Grzybowski - Ophthalmology and Therapy, 2024

snippet: A large language model (LLM) is an artificial intelligence (AI) model that uses natural language processing (NLP) to understand, interpret, and generate human-like language responses from unstructured text input. Its real-time response capabilities …


title: UzRoberta: A pre-trained language model for Uzbek

authors: R Davronov, F Adilova - AIP Conference Proceedings, 2024

snippet: … Common Crawl datasets, was generated distributed word representations for 157 different languages [8]. Author’s Uzbek Wikipedia model has 110K words, compared to -830K words in the Common Crawl … It is pre-trained on 2.5TB of filtered …


title: UZROBERTA: AN UZBEK LANGUAGE PRE-TRAINED MODEL

authors: F Adilova, R Davronov, R Safarov - Universum: технические науки, 2023

snippet: … Distributed word representations for 157 distinct languages were created using the Wikipedia and Common Crawl databases [8]. The author's Uzbek Wikipedia model has 110K words, but the Common Crawl model only has 830K words. They …


title: Validating and Exploring Large Geographic Corpora

authors: J Dunn - arXiv preprint arXiv:2403.08198, 2024

snippet: … Beginning with a 427 billion word corpus derived from the Common Crawl, three methods are used to improve the quality of sub-corpora representing specific language-country pairs … This paper starts with an existing web corpus derived …


title: VERB: Visualizing and Interpreting Bias Mitigation Techniques Geometrically for Word Representations

authors: YAN ZHENG, CCM YEH, J WANG, WEI ZHANG - 2024

snippet: Complicated and massive datasets are becoming more and more commonly represented as vectorized embeddings. They are part of a de facto model for words in word vector embeddings such as Word2Vec [51] and GloVe [54], and are …


title: Verifying Claims About Metaphors with Large-Scale Automatic Metaphor Identification

authors: K Aono, R Sasano, K Takeda - arXiv preprint arXiv:2404.01029, 2024

snippet: … Third, in this study, the examples were collected from CommonCrawl, which is a corpus of texts on the Web, and thus contains mostly written language. It is unknown whether similar results can be obtained when analyzing examples collected from a …


title: VERISCORE: Evaluating the factuality of verifiable claims in long-form text generation

authors: Y Song, Y Kim, M Iyyer - arXiv preprint arXiv:2406.19276, 2024

snippet: Existing metrics for evaluating the factuality of long-form text, such as FACTSCORE (Min et al., 2023) and SAFE (Wei et al., 2024), decompose an input text into "atomic claims" and verify each against a knowledge base like Wikipedia. These metrics are not …


title: Viewpoints Detection in Political Speeches

snippet: Politics has significant impacts on our daily lives. It influences laws, policies, and resources allocation. The Internet and social media confront readers with an overwhelming amount of political texts. Readers struggle to analyze the information …


title: ViHateT5: Enhancing Hate Speech Detection in Vietnamese With A Unified Text-to-Text Transformer Model

authors: LT Nguyen - arXiv preprint arXiv:2405.14141, 2024

snippet: … Leveraging a Transformer architecture and trained on a massive dataset of 2.5TB filtered CommonCrawl text across 100 languages, it employs a masked language modeling (MLM) objective to learn crosslingual representations. This approach …


title: ViLLM-Eval: A Comprehensive Evaluation Suite for Vietnamese Large Language Models

authors: TH Nguyen, AC Le, VC Nguyen - arXiv preprint arXiv:2404.11086, 2024

snippet: The rapid advancement of large language models (LLMs) necessitates the development of new benchmarks to accurately assess their capabilities. To address this need for Vietnamese, this work aims to introduce ViLLM-Eval, the …


title: VISION2UI: A Real-World Dataset with Layout for Code Generation from UI Designs

authors: Y Gui, Z Li, Y Wan, Y Shi, H Zhang, Y Su, S Dong… - arXiv preprint arXiv …, 2024

snippet: … we opt to create our dataset on top of the Common Crawl3 dataset. The Common Crawl dataset is a vast collection of Web page data from … This partial dataset from the original Common Crawl dataset includes 3.35 billion pages, amounting to a total …


title: Visual Artists, Technological Shock, and Generative AI

snippet: The impact of generative AI (GenAI) programs on visual art is comparable to earlier historical moments of technological shock, when literary and visual artists grappled with unprecedented reproductive tools such as the printing press, photography, and …


title: Visual Instruction Tuning towards General-Purpose Multimodal Model: A Survey

authors: J Huang, J Zhang, K Jiang, H Qiu, S Lu - arXiv preprint arXiv:2312.16602, 2023

snippet: Traditional computer vision generally solves each single task independently by a dedicated model with the task instruction implicitly designed in the model architecture, arising two limitations: (1) it leads to task-specific models, which require …


title: VLUE: A New Benchmark and Multi-task Knowledge Transfer Learning for Vietnamese Natural Language Understanding

authors: PNT Do, SQ Tran, PG Hoang, K Van Nguyen… - arXiv preprint arXiv …, 2024

snippet: … 250002 multilingual CommonCrawl XLM-Robertalarge … This model was trained on a Transformers-based masked language task using two terabytes of CommonCrawl data across more than a hundred languages. The model has two …


title: W-RAG: Weakly Supervised Dense Retrieval in RAG for Open-domain Question Answering

authors: J Nian, Z Peng, Q Wang, Y Fang - arXiv preprint arXiv:2408.08444, 2024

snippet: … The corpus may comprise chuncked Wikipedia articles or from online sources such as Reddit and Common Crawl, while the question-answer pairs could be sourced from OpenQA platforms such as Quora and StackOverflow. The simple first-stage …


title: WaCadie: Towards an Acadian French Corpus

authors: J Robichaud, P Cook - Proceedings of the 2024 Joint International Conference …, 2024

snippet: … We treat each post as one document, similar to one website’s HTML from the Common Crawl corpus. We refer to this corpus as r/acadie. … We used CommonCrawl’s extensive database and scraped New Brunswick (.nb.ca) websites …


title: WanJuan-CC: A Safe and High-Quality Open-sourced English Webtext Dataset

authors: J Qiu, H Lv, Z Jin, R Wang, W Ning, J Yu, CB Zhang… - arXiv preprint arXiv …, 2024

snippet: … Moreover, Common Crawl contains a large amount of unsafe content, such as toxic and pornographic … Common Crawl data, which includes data extraction, heuristic rule filtering, fuzzy deduplication, content safety filtering, and data quality …


title: WCC-EC 2.0: Enhancing Neural Machine Translation with a 1.6 M+ Web-Crawled English-Chinese Parallel Corpus

authors: J Zhang, K Su, Y Tian, T Matsumoto - Electronics, 2024

snippet: This research introduces WCC-EC 2.0 (Web-Crawled Corpus—English and Chinese), a comprehensive parallel corpus designed for enhancing Neural Machine Translation (NMT), featuring over 1.6 million English-Chinese sentence pairs …


title: We have no idea what we are walking into: AI and ethical considerations

authors: KB Forrest - Annals of the New York Academy of Sciences, 2024

snippet: … called “Common Crawl” provide data, “C4” is a curated (de-duplicated and some nonsensical junk eliminated) set of the Common Crawl data, then there is a data set well named “The Pile”, and others that are compilations of books, music …


title: Weakly supervised learning for an effective focused web crawler

authors: PRJ Dhanith, K Saeed, G Rohith, SP Raja - Engineering Applications of Artificial …, 2024

snippet: Focused crawler traverses the Web to only collect pages that are relevant to a particular topic, and is increasingly considered as a way to get around the scalability issues with current general-purpose search engines. But the data diversity in the …


title: WEATHER-5K: A Large-scale Global Station Weather Dataset Towards Comprehensive Time-series Forecasting Benchmark

authors: T Han, S Guo, Z Chen, W Xu, L Bai - arXiv preprint arXiv:2406.14399, 2024

snippet: Global Station Weather Forecasting (GSWF) is crucial for various sectors, including aviation, agriculture, energy, and disaster preparedness. Recent advancements in deep learning have significantly improved the accuracy of weather predictions by …


title: Web Crawl Refusals: Insights From Common Crawl

snippet: … Abstract: This study investigates server-side blocks encountered by Common Crawl, a major … that approximately 1.68% of websites in a Common Crawl snapshot exhibit some form of explicit … of these blocks and the effectiveness of …


title: Web Data Commons-RDFa, Microdata, Embedded JSON-LD, and Microformats Data Sets-October 2023

snippet: The Web Data Commons RDFa, Microdata and Microformats data sets has been extracted from the September/October 2023 release of the Common Crawl. In summary, we found structured data within 1.7 billion HTML pages out of the 3.4 …


title: Web Data Commons-Schema. org Table Corpus 2023

snippet: The WDC schema.org Table Corpus consists of ~5 million relational tables which were generated by extracting schema.org data from the Common Crawl and grouping the data into seperate tables for each class/host combination, eg all …


title: Web scraping and Generative Models training in the Directive 790/19

snippet: … The companies that created GM never disclosed the exact corpus of training data they employed to train their models; however, it has become clear sources that many copyrighted materials were employed as part of larger datasets such as the …


title: WebCiteS: Attributed Query-Focused Summarization on Chinese Web Search Results with Citations

authors: H Deng, C Wang, X Li, D Yuan, J Zhan, T Zhou, J Ma… - arXiv preprint arXiv …, 2024

snippet: Enhancing the attribution in large language models (LLMs) is a crucial task. One feasible approach is to enable LLMs to cite external sources that support their generations. However, existing datasets and evaluation methods in this domain still …


title: WebGraph: The Next Generation (Is in Rust)

authors: T Fontana, S Vigna, S Zacchiroli - ACM Web Conference 2024, 2024

snippet: We report the results of a yearlong effort to port the WebGraph framework from Java to Rust. For two decades WebGraph has been instrumental in the analysis and distribution of large graphs for the research community of TheWebConf, but the …


title: What are Large Language Models Made of?

authors: C Stinson - Leading the Way: Envisioning the Future of Higher …, 2024

snippet: … CommonCrawl is the lowest quality but largest of these datasets, consisting of text scraped from all over the web. Their data from the years 2016 to 2019 were used to train GPT-3. OpenAI’s quality control measure for filtering CommonCrawl was to …


title: What do BERT word embeddings learn about the French language?

authors: E Goliakova, D Langlois - Sixth International Conference, 2024

snippet: Pre-trained word embeddings (for example, BERT-like) have been successfully used in a variety of downstream tasks. However, do all embeddings, obtained from the models of the same architecture, encode information in the same way? Does the …


title: What's in a Name? Experimental Evidence of Gender Bias in Recommendation Letters Generated by ChatGPT

authors: DM Kaplan, R Palitsky, SJ Arconada Alvarez… - Journal of Medical Internet …, 2024

snippet: … In the case of GPT-3.5 (the current version at the time of this study), the model contains 175 billion parameters and was trained with commonly available text training sets that represent a broad swath of data on the internet (Common Crawl) …


title: When Context Leads but Parametric Memory Follows in Large Language Models

authors: Y Tao, A Hiatt, E Haake, AJ Jetter, A Agrawal - arXiv preprint arXiv:2409.08435, 2024

snippet: Large language models (LLMs) have demonstrated remarkable progress in leveraging diverse knowledge sources. This study investigates how nine widely used LLMs allocate knowledge between local context and global parameters when …


title: When Young Scholars Cooperate with LLMs in Academic Tasks: The Influence of Individual Differences and Task Complexities

authors: J Wang, C Huang, S Yan, W Xie, D He - International Journal of Human–Computer …, 2024

snippet: As a novel AI-powered conversational system, large language models (LLMs) have the potential to be used in various applications. Recent advances in LLMs like ChatGPT have made LLM-based academic tools possible. However, most of the …


title: Where does In-context Translation Happen in Large Language Models

authors: S Sia, D Mueller, K Duh - arXiv preprint arXiv:2403.04510, 2024

snippet: Self-supervised large language models have demonstrated the ability to perform Machine Translation (MT) via in-context learning, but little is known about where the model performs the task with respect to prompt instructions and demonstration …


title: Which Word Embeddings for Modeling Web Search Queries? Application to the Study of Search Strategies

snippet: In order to represent the global strategies deployed by a user during an information retrieval session on the Web, we compare different pretrained vector models capable of representing the queries submitted to a search engine. More precisely …


title: Who's Afraid of ChatGPT?: Large Language Models and Conspiracies

snippet: … GPT-3 added Common Crawl to the training datasets, 570GB of text obtained after filtering from 45TB of plaintext. The filtering was done by comparing new documents to documents in WebText, with the latter acting as a proxy for high-quality …


title: Willkommens-Merkel, Chaos-Johnson, and Tore-Klose: Modeling the Evaluative Meaning of German Personal Name Compounds

snippet: We present a comprehensive computational study of the under-investigated phenomenon of personal name compounds (PNCs) in German such as Willkommens-Merkel (‘Welcome-Merkel’). Prevalent in news, social media, and …


title: Worldwide Federated Training of Language Models

authors: A Iacob, L Sani, B Marino, P Aleksandrov, ND Lane - arXiv preprint arXiv:2405.14446, 2024

snippet: … To simulate geographically distributed systems, we use a subset of Multilingual Colossal Common Crawl (mC4)[81], covering high and low-… For IID experiments, we use the standard Cleaned Colossal Common Crawl (C4) English dataset …


title: WRITTEN IN PERSIAN USING A TRANSFORMER-BASED MODEL

authors: T Firoozi, MJ Gierl - The Routledge International Handbook of Automated …, 2024

snippet: … This model is created using contents from Common Crawl and Wikipedia. It is trained using CBOW with position weights, in dimension 300, with character n-grams of length 5, and a window of size 5 and 10. The model gives 300-dimensional vector …


title: X-Phishing-Writer: A Framework for Cross-Lingual Phishing Email Generation

authors: SW Guo, YC Fan - ACM Transactions on Asian and Low-Resource …, 2024

snippet: … The mBART model employs various language corpora from the Common Crawl dataset and undergoes multilingual pretraining based on the BART architecture. The training data is a concatenation of data from � diferent languages, denoted as � = �1,�2...�� …


title: XLM for Autonomous Driving Systems: A Comprehensive Review

authors: S Fourati, W Jaafar, N Baccar, S Alfattani - arXiv preprint arXiv:2409.10484, 2024

snippet: Large Language Models (LLMs) have showcased remarkable proficiency in various information-processing tasks. These tasks span from extracting data and summarizing literature to generating content, predictive modeling, decision-making …


title: XLMR4MD: New Vietnamese dataset and framework for detecting the consistency of description and permission in Android applications using large language models

authors: QN Nguyen, NT Cam, K Van Nguyen - Computers & Security, 2024

snippet: … The most significant update that XLMRoBERTa provides compared to the original is that the amount of training data has increased significantly, the cleaned CommonCrawl data it was trained on takes up a considerable memory block of up to …


title: XLORE 3: A Large-scale Multilingual Knowledge Graph from Heterogeneous Wiki Knowledge Resources

authors: K Zeng, H Jin, X Lv, F Zhu, L Hou, Y Zhang, F Pang… - ACM Transactions on …, 2024

snippet: In recent years, Knowledge Graph (KG) has attracted significant attention from academia and industry, resulting in the development of numerous technologies for KG construction, completion, and application. XLORE is one of the largest …


title: Yi: Open Foundation Models by 01. AI

authors: A Young, B Chen, C Li, C Huang, G Zhang, G Zhang… - arXiv preprint arXiv …, 2024

snippet: … We start with web documents from Common Crawl, use the CCNet pipeline [79] for language identification and perplexity scoring. Then … Notably, the Chinese content extracted from Common Crawl present unique challenges, particularly with a …


title: Yuan 2.0-M32: Mixture of Experts with Attention Router

authors: S Wu, J Luo, X Chen, L Li, X Zhao, T Yu, C Wang… - arXiv preprint arXiv …, 2024

snippet: Yuan 2.0-M32, with a similar base architecture as Yuan-2.0 2B, uses a mixture-of-experts architecture with 32 experts of which 2 experts are active. A new router network, Attention Router, is proposed and adopted for a more efficient selection of experts …


title: YuLan: An Open-source Large Language Model

authors: Y Zhu, K Zhou, K Mao, W Chen, Y Sun, Z Chen, Q Cao… - arXiv preprint arXiv …, 2024

snippet: … In addition to these sources, we process raw data from Common Crawl (CC) dumps, particularly focusing on events that occurred between January 2021 and February 2023. To manage the practical challenges of HTML content extraction and …


title: Zero and Few Short Learning Using Large Language Models for De-Identification of Medical Records

authors: YS Yashwanth, R Shettar - IEEE Access, 2024

snippet: … Among the word embeddings, CommonCrawl, a general English word embedding, demonstrated superior performance compared to the other four. The highest F-measure for CommonCrawl was 96.46, with corresponding precision and recall scores of …


title: Zero-Shot Distillation for Image Encoders: How to Make Effective Use of Synthetic Data

authors: N Popp, JH Metzen, M Hein - arXiv preprint arXiv:2404.16637, 2024

snippet: … The first one involves relying on large-scale data such as common crawl datasets [7,38]. While this approach is feasible for large foundation models, it poses challenges for smaller models as these lack the same level of generalization capabilities due to …


title: Zero-shot learning for multilingual discourse relation classification

authors: E Metheniti, P Muller, C Braud, M Hernández-Casas - 5th Workshop on …, 2024

snippet: Classifying discourse relations is a hard task: discourse-annotated data is scarce, especially for languages other than English, and there exist different theoretical frameworks that affect textual spans to be linked and the label set used. Thus, work …


title: Zero-shot Sentiment Analysis in Low-Resource Languages Using a Multilingual Sentiment Lexicon

authors: F Koto, T Beck, Z Talat, I Gurevych, T Baldwin - arXiv preprint arXiv:2402.02113, 2024

snippet: Improving multilingual language models capabilities in low-resource languages is generally difficult due to the scarcity of large-scale data in those languages. In this paper, we relax the reliance on texts in low-resource languages by using …


title: Zero-Shot vs. Few-Shot Multi-Speaker TTS Using Pre-trained Czech SpeechT5 Model

authors: J Lehečka, Z Hanzlíček, J Matoušek, D Tihelka - arXiv preprint arXiv:2407.17167, 2024

snippet: … Text Data As a source of unlabeled text data, we used the Common Crawl project4 which is a huge public web archive consisting of petabytes of crawled web pages. We used the language information provided in the index files to select Czech …


title: ZS4C: Zero-Shot Synthesis of Compilable Code for Incomplete Code Snippets using ChatGPT

authors: A Kabir, S Wang, Y Tian, M Asaduzzaman, W Zhang - arXiv preprint arXiv …, 2024

snippet: Technical question and answering (Q&A) sites such as Stack Overflow have become an important source for software developers to seek knowledge. However, code snippets on Q&A sites are usually uncompilable and semantically incomplete for …


title: Zyda: A 1.3 T Dataset for Open Language Modeling

authors: Y Tokpanov, B Millidge, P Glorioso, J Pilault, A Ibrahim… - arXiv preprint arXiv …, 2024

snippet: … Additionally, unlike these works which directly filter from Common Crawl, we aim to instead collect and collate existing open-source datasets under one banner, and to uniformly filter and deduplicate them against one another so as to create a dataset …