{ "cells": [ { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (4.40.2)\n", "Requirement already satisfied: datasets in /usr/local/lib/python3.10/dist-packages (2.19.1)\n", "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers) (3.13.1)\n", "Requirement already satisfied: huggingface-hub<1.0,>=0.19.3 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.23.0)\n", "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (1.26.2)\n", "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (23.2)\n", "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (6.0.1)\n", "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (2024.4.28)\n", "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers) (2.31.0)\n", "Requirement already satisfied: tokenizers<0.20,>=0.19 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.19.1)\n", "Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.4.3)\n", "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers) (4.66.2)\n", "Requirement already satisfied: pyarrow>=12.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (16.0.0)\n", "Requirement already satisfied: pyarrow-hotfix in /usr/local/lib/python3.10/dist-packages (from datasets) (0.6)\n", "Requirement already satisfied: dill<0.3.9,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (0.3.8)\n", "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets) (2.2.2)\n", "Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets) (3.4.1)\n", "Requirement already satisfied: multiprocess in /usr/local/lib/python3.10/dist-packages (from datasets) (0.70.16)\n", "Requirement already satisfied: fsspec<=2024.3.1,>=2023.1.0 in /usr/local/lib/python3.10/dist-packages (from fsspec[http]<=2024.3.1,>=2023.1.0->datasets) (2023.10.0)\n", "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets) (3.9.0b0)\n", "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (23.1.0)\n", "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (6.0.5)\n", "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.9.4)\n", "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.4.1)\n", "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.3.1)\n", "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (4.0.3)\n", "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.19.3->transformers) (4.8.0)\n", "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.3.2)\n", "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.6)\n", "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2.1.0)\n", "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2023.11.17)\n", "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2.8.2)\n", "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2024.1)\n", "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2024.1)\n", "Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.16.0)\n", "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n", "\u001b[0m" ] } ], "source": [ "# Transformers installation\n", "# ! pip install transformers datasets\n", "# To install from source instead of the last release, comment the command above and uncomment the following one.\n", "# ! pip install git+https://github.com/huggingface/transformers.git" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Causal language modeling" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are two types of language modeling, causal and masked. This guide illustrates causal language modeling.\n", "Causal language models are frequently used for text generation. You can use these models for creative applications like\n", "choosing your own text adventure or an intelligent coding assistant like Copilot or CodeParrot." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "cellView": "form", "hide_input": true }, "outputs": [], "source": [ "# #@title\n", "# from IPython.display import HTML\n", "\n", "# HTML('')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Causal language modeling predicts the next token in a sequence of tokens, and the model can only attend to tokens on\n", "the left. This means the model cannot see future tokens. GPT-2 is an example of a causal language model.\n", "\n", "This guide will show you how to:\n", "\n", "1. Finetune [DistilGPT2](https://huggingface.co/distilgpt2) on the [r/askscience](https://www.reddit.com/r/askscience/) subset of the [ELI5](https://huggingface.co/datasets/eli5) dataset.\n", "2. Use your finetuned model for inference.\n", "\n", "\n", "You can finetune other architectures for causal language modeling following the same steps in this guide.\n", "Choose one of the following architectures:\n", "\n", "\n", "[BART](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/bart), [BERT](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/bert), [Bert Generation](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/bert-generation), [BigBird](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/big_bird), [BigBird-Pegasus](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/bigbird_pegasus), [BioGpt](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/biogpt), [Blenderbot](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/blenderbot), [BlenderbotSmall](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/blenderbot-small), [BLOOM](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/bloom), [CamemBERT](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/camembert), [CodeGen](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/codegen), [CPM-Ant](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/cpmant), [CTRL](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/ctrl), [Data2VecText](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/data2vec-text), [ELECTRA](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/electra), [ERNIE](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/ernie), [GIT](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/git), [GPT-Sw3](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/gpt-sw3), [OpenAI GPT-2](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/gpt2), [GPTBigCode](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/gpt_bigcode), [GPT Neo](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/gpt_neo), [GPT NeoX](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/gpt_neox), [GPT NeoX Japanese](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/gpt_neox_japanese), [GPT-J](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/gptj), [LLaMA](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/llama), [Marian](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/marian), [mBART](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/mbart), [MEGA](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/mega), [Megatron-BERT](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/megatron-bert), [MVP](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/mvp), [OpenLlama](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/open-llama), [OpenAI GPT](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/openai-gpt), [OPT](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/opt), [Pegasus](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/pegasus), [PLBart](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/plbart), [ProphetNet](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/prophetnet), [QDQBert](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/qdqbert), [Reformer](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/reformer), [RemBERT](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/rembert), [RoBERTa](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/roberta), [RoBERTa-PreLayerNorm](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/roberta-prelayernorm), [RoCBert](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/roc_bert), [RoFormer](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/roformer), [RWKV](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/rwkv), [Speech2Text2](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/speech_to_text_2), [Transformer-XL](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/transfo-xl), [TrOCR](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/trocr), [XGLM](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/xglm), [XLM](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/xlm), [XLM-ProphetNet](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/xlm-prophetnet), [XLM-RoBERTa](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/xlm-roberta), [XLM-RoBERTa-XL](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/xlm-roberta-xl), [XLNet](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/xlnet), [X-MOD](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/xmod)\n", "\n", "\n", "\n", "\n", "\n", "\n", "Before you begin, make sure you have all the necessary libraries installed:\n", "\n", "```bash\n", "pip install transformers datasets evaluate\n", "```\n", "\n", "We encourage you to log in to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to log in:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "a6d9e280e08e40ddbbcb8fbe97e1fae9", "version_major": 2, "version_minor": 0 }, "text/plain": [ "VBox(children=(HTML(value='
" ], "text/plain": [ "" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# #@title\n", "# from IPython.display import HTML\n", "\n", "# HTML('')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The next step is to load a DistilGPT2 tokenizer to process the `text` subfield:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n", " warnings.warn(\n", "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n" ] } ], "source": [ "from transformers import AutoTokenizer, GPT2TokenizerFast\n", "\n", "# tokenizer = AutoTokenizer.from_pretrained(\"distilgpt2\")\n", "\n", "\n", "tokenizer = GPT2TokenizerFast.from_pretrained(\"Xenova/gpt-4\")#, cache_dir=cache_dir)\n", "tokenizer.pad_token = tokenizer.eos_token" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You'll notice from the example above, the `text` field is actually nested inside `answers`. This means you'll need to\n", "extract the `text` subfield from its nested structure with the [`flatten`](https://huggingface.co/docs/datasets/process.html#flatten) method:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'Text': 'Allow me to clarify a genuine fast for amateur online users What exactly is Youtube . com? Youtube . com is probably the most in-demand web site on the web which allow you to view and publish video lessons for free. These are generally submitted by Vimeo members on this video discussing system. Yet another thing YouTube registration is provided for free so anyone can join, however account is not required for watching video lessons. In order to sometimes observe video clips or post your own video lessons so that you can show to your friends, loved ones as well as other Vimeo members. Once you get dependent at viewing video clip, it is possible to phone yourself a YouTuber!\\n- Everything you are unable to upload? Nonetheless there are some regulations or YouTube\\'s regards to use that you should.\\n- Observing a Vimeo movie is really simple, you just need to.\\nObserving a You tube movie is absolutely simple, you just need to variety your best song or television set plan from the research discipline click on \"Research\" option and that\\'s it. It will approach your demand and give you a list of related results. You are able to click on a outcome and this will commence taking part in the recording. youtube downloader\\nAble to click on a outcome and\\nSo, just how to publish your chosen videos? Youtube . com is very popular online video discussing foundation that allows one to publish their video lessons. Uploading a relevant video online is an easy process, just select any video submit through your computer on your YouTube accounts webpage and it will surely begin posting the video. Nonetheless Vimeo will not offer any choice to down load a printed video that you will be seeing, you can easily take note of the site Link so that you can view it later, which seems handy for YouTube users.\\nEverything you cannot upload? Nevertheless there are a few regulations or YouTube\\'s terms of use that you have to comply with, specifically you happen to be unacceptable to upload any restricted content or erotic information. Nevertheless you can use it to showcase your products online.\\nA few regulations or\\nOnline video good quality once you upload Vimeo permits you to post all popular movie formats and produces good quality probable. Whenever you post a youtube video to Youtube . com, you ought to anticipate that high quality will slightly be changed, it is because YouTube optimizes the video for speedier packing. You can even add Hi-def or Hi-def video lessons nevertheless it will take much longer to weight once you observe it. Greater the high quality more slowly movie will load.\\nYou upload Vimeo\\nProbably the most well-known movie web sites online is You tube as well as for certain, you can find videos inside the web site you want to create you everywhere and adding it inside your PSP device might be what you need. However, YouTube video lessons will not be quickly down loadable. You might need a downloader to download the recording through the website and shop it inside your personal computer. video downloader\\nAfter you have saved the recording, it may possibly not certainly be around the preferred format which can be legible along with your Playstation portable. For those who have saved a structure not in mp4, you may want to transform the submit with your Computer in to a Playstation portable-pleasant structure. You may need a video clip converter for this task, and when you have changed the video tutorials, anyone can down load these to your Playstation portable.\\nWith your Playstation portable For those\\nIn accessing, simply link up your Playstation portable to the laptop or computer by means of its cord, use the Universal serial bus setting and download the video lessons and music that you want to bring along.\\nThat will help you look for a converter or a video downloader, specifically if you want to obtain video clips from Vimeo, be involved in forums and discover topics relevant to this. Certainly, you will also find a great deal of PSP movie information that may also assist you in making the best from your gadget and help you learn to see a number of videos on your gadget.\\nAlso find a great deal of PSP\\nYou can even get into membership web sites where PSP enthusiast collect and discuss information and facts and even more importantly, offers you the tools and software program that you will want to save music, videos and media records to your devices and permit you to enjoy the gizmo a lot more. Although these membership internet sites require only a minimum cost, it really is however vital that you are working with and creating dealings in a guaranteed and harmless internet site.\\n- You can even get into membership websites.\\n- One of the more preferred video clip sites on.\\n- Video quality when you post Vimeo enables you.\\n0 thoughts on “The Most Effective and Well-liked you tube downloader6675”'}" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "Falcon = Falcon.flatten()\n", "Falcon[\"train\"][0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Each subfield is now a separate column as indicated by the `answers` prefix, and the `text` field is a list now. Instead\n", "of tokenizing each sentence separately, convert the list to a string so you can jointly tokenize them.\n", "\n", "Here is a first preprocessing function to join the list of strings for each example and tokenize the result:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "def preprocess_function(examples):\n", " return tokenizer([\" \".join(x) for x in examples[\"Text\"]])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To apply this preprocessing function over the entire dataset, use the 🤗 Datasets [map](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.map) method. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once, and increasing the number of processes with `num_proc`. Remove any columns you don't need:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The OrderedVocab you are attempting to save contains holes for indices [100256, 100261, 100262, 100263, 100266, 100267, 100268, 100269, 100270, 100271, 100272, 100273, 100274, 100275], your vocabulary could be corrupted !\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "51bff46d94664c468064b17d1a8bf1c0", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Map (num_proc=4): 0%| | 0/20000 [00:00 8192). Running this sequence through the model will result in indexing errors\n", "Token indices sequence length is longer than the specified maximum sequence length for this model (14224 > 8192). Running this sequence through the model will result in indexing errors\n", "Token indices sequence length is longer than the specified maximum sequence length for this model (15104 > 8192). Running this sequence through the model will result in indexing errors\n", "Token indices sequence length is longer than the specified maximum sequence length for this model (32874 > 8192). Running this sequence through the model will result in indexing errors\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "The OrderedVocab you are attempting to save contains holes for indices [100256, 100261, 100262, 100263, 100266, 100267, 100268, 100269, 100270, 100271, 100272, 100273, 100274, 100275], your vocabulary could be corrupted !\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "5a093fd9868042a9ac76ed1c141711a7", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Map (num_proc=4): 0%| | 0/2000 [00:00 8192). Running this sequence through the model will result in indexing errors\n", "Token indices sequence length is longer than the specified maximum sequence length for this model (11892 > 8192). Running this sequence through the model will result in indexing errors\n", "Token indices sequence length is longer than the specified maximum sequence length for this model (22303 > 8192). Running this sequence through the model will result in indexing errors\n", "Token indices sequence length is longer than the specified maximum sequence length for this model (12749 > 8192). Running this sequence through the model will result in indexing errors\n" ] } ], "source": [ "tokenized_Falcon = Falcon.map(\n", " preprocess_function,\n", " batched=True,\n", " num_proc=4,\n", " remove_columns=Falcon[\"train\"].column_names,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This dataset contains the token sequences, but some of these are longer than the maximum input length for the model.\n", "\n", "You can now use a second preprocessing function to\n", "- concatenate all the sequences\n", "- split the concatenated sequences into shorter chunks defined by `block_size`, which should be both shorter than the maximum input length and short enough for your GPU RAM." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "block_size = 1048\n", "\n", "\n", "def group_texts(examples):\n", " # Concatenate all texts.\n", " concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}\n", " total_length = len(concatenated_examples[list(examples.keys())[0]])\n", " # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can\n", " # customize this part to your needs.\n", " if total_length >= block_size:\n", " total_length = (total_length // block_size) * block_size\n", " # Split by chunks of block_size.\n", " result = {\n", " k: [t[i : i + block_size] for i in range(0, total_length, block_size)]\n", " for k, t in concatenated_examples.items()\n", " }\n", " result[\"labels\"] = result[\"input_ids\"].copy()\n", " return result" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Apply the `group_texts` function over the entire dataset:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "6134c09493054ce3940da711dc2e965e", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Map (num_proc=4): 0%| | 0/20000 [00:00\n", "\n", "If you aren't familiar with finetuning a model with the [Trainer](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer), take a look at the [basic tutorial](https://huggingface.co/docs/transformers/main/en/tasks/../training#train-with-pytorch-trainer)!\n", "\n", "\n", "\n", "You're ready to start training your model now! Load DistilGPT2 with [AutoModelForCausalLM](https://huggingface.co/docs/transformers/main/en/model_doc/auto#transformers.AutoModelForCausalLM):" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n", " warnings.warn(\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "f55ae69743a74a08943641e2da03e791", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Loading checkpoint shards: 0%| | 0/3 [00:00" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "torch.no_grad()" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "LlamaForCausalLM(\n", " (model): LlamaModel(\n", " (embed_tokens): Embedding(100288, 4096)\n", " (layers): ModuleList(\n", " (0-29): 30 x LlamaDecoderLayer(\n", " (self_attn): LlamaSdpaAttention(\n", " (q_proj): Linear(in_features=4096, out_features=4096, bias=False)\n", " (k_proj): Linear(in_features=4096, out_features=4096, bias=False)\n", " (v_proj): Linear(in_features=4096, out_features=4096, bias=False)\n", " (o_proj): Linear(in_features=4096, out_features=4096, bias=False)\n", " (rotary_emb): LlamaRotaryEmbedding()\n", " )\n", " (mlp): LlamaMLP(\n", " (gate_proj): Linear(in_features=4096, out_features=11008, bias=False)\n", " (up_proj): Linear(in_features=4096, out_features=11008, bias=False)\n", " (down_proj): Linear(in_features=11008, out_features=4096, bias=False)\n", " (act_fn): SiLU()\n", " )\n", " (input_layernorm): LlamaRMSNorm()\n", " (post_attention_layernorm): LlamaRMSNorm()\n", " )\n", " )\n", " (norm): LlamaRMSNorm()\n", " )\n", " (lm_head): Linear(in_features=4096, out_features=100288, bias=False)\n", ")" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.to('cuda')" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "training_args = TrainingArguments(\n", " output_dir=\"Fine-Tuned-S9\",\n", " bf16=True,\n", " # evaluation_strategy=\"epoch\",\n", " evaluation_strategy=\"steps\",\n", " learning_rate=2e-5,\n", " weight_decay=0.01,\n", " num_train_epochs=1,\n", " per_device_train_batch_size=2,\n", " per_device_eval_batch_size=2,\n", " # lr_scheduler_type = 'cosine',\n", " push_to_hub=False,\n", " save_total_limit = 2\n", " # save_strategy = “no”\n", " load_best_model_at_end=False\n", ")\n", "\n", "trainer = Trainer(\n", " model=model,\n", " args=training_args,\n", " train_dataset=lm_dataset[\"train\"],\n", " eval_dataset=lm_dataset[\"validation\"],\n", " # eval_dataset=lm_dataset[\"test\"],\n", " data_collator=data_collator,\n", ")\n", "\n", "# trainer.train()" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "trainer.train()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once training is completed, use the [evaluate()](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer.evaluate) method to evaluate your model and get its perplexity:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Perplexity: 2.21\n" ] } ], "source": [ "import math\n", "\n", "eval_results = trainer.evaluate()\n", "print(f\"Perplexity: {math.exp(eval_results['eval_loss']):.2f}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then share your model to the Hub with the [push_to_hub()](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer.push_to_hub) method so everyone can use your model:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# trainer.push_to_hub()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "For a more in-depth example of how to finetune a model for causal language modeling, take a look at the corresponding\n", "[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb)\n", "or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Inference" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Great, now that you've finetuned a model, you can use it for inference!\n", "\n", "Come up with a prompt you'd like to generate text from:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# prompt = \"Somatic hypermutation allows the immune system to\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The simplest way to try out your finetuned model for inference is to use it in a [pipeline()](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.pipeline). Instantiate a `pipeline` for text generation with your model, and pass your text to it:" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "ename": "ValueError", "evalue": "Could not load model Fine-Tuned-S9/checkpoint-4000 with any of the following classes: (, ). See the original errors:\n\nwhile loading with AutoModelForCausalLM, an error is thrown:\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.10/dist-packages/transformers/pipelines/base.py\", line 283, in infer_framework_load_model\n model = model_class.from_pretrained(model, **kwargs)\n File \"/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py\", line 563, in from_pretrained\n return model_class.from_pretrained(\n File \"/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py\", line 3260, in from_pretrained\n raise EnvironmentError(\nOSError: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory Fine-Tuned-S9/checkpoint-4000.\n\nwhile loading with LlamaForCausalLM, an error is thrown:\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.10/dist-packages/transformers/pipelines/base.py\", line 283, in infer_framework_load_model\n model = model_class.from_pretrained(model, **kwargs)\n File \"/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py\", line 3260, in from_pretrained\n raise EnvironmentError(\nOSError: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory Fine-Tuned-S9/checkpoint-4000.\n\n\n", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)", "Cell \u001b[0;32mIn[20], line 3\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mtransformers\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m pipeline\n\u001b[1;32m 2\u001b[0m \u001b[38;5;66;03m# checkpoint-4000\u001b[39;00m\n\u001b[0;32m----> 3\u001b[0m generator \u001b[38;5;241m=\u001b[39m \u001b[43mpipeline\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mtext-generation\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mmodel\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mFine-Tuned-S9/checkpoint-4000\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m 4\u001b[0m generator(prompt)\n", "File \u001b[0;32m/usr/local/lib/python3.10/dist-packages/transformers/pipelines/__init__.py:906\u001b[0m, in \u001b[0;36mpipeline\u001b[0;34m(task, model, config, tokenizer, feature_extractor, image_processor, framework, revision, use_fast, token, device, device_map, torch_dtype, trust_remote_code, model_kwargs, pipeline_class, **kwargs)\u001b[0m\n\u001b[1;32m 904\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(model, \u001b[38;5;28mstr\u001b[39m) \u001b[38;5;129;01mor\u001b[39;00m framework \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m 905\u001b[0m model_classes \u001b[38;5;241m=\u001b[39m {\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtf\u001b[39m\u001b[38;5;124m\"\u001b[39m: targeted_task[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtf\u001b[39m\u001b[38;5;124m\"\u001b[39m], \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mpt\u001b[39m\u001b[38;5;124m\"\u001b[39m: targeted_task[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mpt\u001b[39m\u001b[38;5;124m\"\u001b[39m]}\n\u001b[0;32m--> 906\u001b[0m framework, model \u001b[38;5;241m=\u001b[39m \u001b[43minfer_framework_load_model\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 907\u001b[0m \u001b[43m \u001b[49m\u001b[43mmodel\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 908\u001b[0m \u001b[43m \u001b[49m\u001b[43mmodel_classes\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmodel_classes\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 909\u001b[0m \u001b[43m \u001b[49m\u001b[43mconfig\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mconfig\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 910\u001b[0m \u001b[43m \u001b[49m\u001b[43mframework\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mframework\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 911\u001b[0m \u001b[43m \u001b[49m\u001b[43mtask\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtask\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 912\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mhub_kwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 913\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mmodel_kwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 914\u001b[0m \u001b[43m \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 916\u001b[0m model_config \u001b[38;5;241m=\u001b[39m model\u001b[38;5;241m.\u001b[39mconfig\n\u001b[1;32m 917\u001b[0m hub_kwargs[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m_commit_hash\u001b[39m\u001b[38;5;124m\"\u001b[39m] \u001b[38;5;241m=\u001b[39m model\u001b[38;5;241m.\u001b[39mconfig\u001b[38;5;241m.\u001b[39m_commit_hash\n", "File \u001b[0;32m/usr/local/lib/python3.10/dist-packages/transformers/pipelines/base.py:296\u001b[0m, in \u001b[0;36minfer_framework_load_model\u001b[0;34m(model, config, model_classes, task, framework, **model_kwargs)\u001b[0m\n\u001b[1;32m 294\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m class_name, trace \u001b[38;5;129;01min\u001b[39;00m all_traceback\u001b[38;5;241m.\u001b[39mitems():\n\u001b[1;32m 295\u001b[0m error \u001b[38;5;241m+\u001b[39m\u001b[38;5;241m=\u001b[39m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mwhile loading with \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mclass_name\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m, an error is thrown:\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;132;01m{\u001b[39;00mtrace\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m--> 296\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m 297\u001b[0m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mCould not load model \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mmodel\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m with any of the following classes: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mclass_tuple\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m. See the original errors:\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;132;01m{\u001b[39;00merror\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 298\u001b[0m )\n\u001b[1;32m 300\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m framework \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m 301\u001b[0m framework \u001b[38;5;241m=\u001b[39m infer_framework(model\u001b[38;5;241m.\u001b[39m\u001b[38;5;18m__class__\u001b[39m)\n", "\u001b[0;31mValueError\u001b[0m: Could not load model Fine-Tuned-S9/checkpoint-4000 with any of the following classes: (, ). See the original errors:\n\nwhile loading with AutoModelForCausalLM, an error is thrown:\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.10/dist-packages/transformers/pipelines/base.py\", line 283, in infer_framework_load_model\n model = model_class.from_pretrained(model, **kwargs)\n File \"/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py\", line 563, in from_pretrained\n return model_class.from_pretrained(\n File \"/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py\", line 3260, in from_pretrained\n raise EnvironmentError(\nOSError: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory Fine-Tuned-S9/checkpoint-4000.\n\nwhile loading with LlamaForCausalLM, an error is thrown:\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.10/dist-packages/transformers/pipelines/base.py\", line 283, in infer_framework_load_model\n model = model_class.from_pretrained(model, **kwargs)\n File \"/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py\", line 3260, in from_pretrained\n raise EnvironmentError(\nOSError: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory Fine-Tuned-S9/checkpoint-4000.\n\n\n" ] } ], "source": [ "# from transformers import pipeline\n", "# # checkpoint-4000\n", "# generator = pipeline(\"text-generation\", model=\"Fine-Tuned-S9/checkpoint-4000\")\n", "# generator(prompt)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Tokenize the text and return the `input_ids` as PyTorch tensors:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n" ] } ], "source": [ "# from transformers import AutoTokenizer\n", "\n", "# tokenizer = AutoTokenizer.from_pretrained(\"Xenova/gpt-4\")\n", "# inputs = tokenizer(prompt, return_tensors=\"pt\").input_ids" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Use the [generate()](https://huggingface.co/docs/transformers/main/en/main_classes/text_generation#transformers.GenerationMixin.generate) method to generate text.\n", "For more details about the different text generation strategies and parameters for controlling generation, check out the [Text generation strategies](https://huggingface.co/docs/transformers/main/en/tasks/../generation_strategies) page." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "7ba147780e8548d28a00a655e81e588a", "version_major": 2, "version_minor": 0 }, "text/plain": [ "config.json: 0%| | 0.00/688 [00:00