"],"text/html":["\n"," \n"," \n","
\n"," [655/655 02:34, Epoch 5/5]\n","
\n"," \n"," \n"," | Step | \n"," Training Loss | \n"," Validation Loss | \n","
\n"," \n"," \n"," \n","
"]},"metadata":{}},{"output_type":"stream","name":"stdout","text":["\n","\n","\n","Training result for learning rate: 3e-05\n"]},{"output_type":"display_data","data":{"text/plain":[""],"text/html":["\n"," \n"," \n","
\n"," [655/655 02:36, Epoch 5/5]\n","
\n"," \n"," \n"," | Step | \n"," Training Loss | \n"," Validation Loss | \n","
\n"," \n"," \n"," \n","
"]},"metadata":{}},{"output_type":"stream","name":"stdout","text":["\n","\n","\n","Training result for learning rate: 1e-05\n"]},{"output_type":"display_data","data":{"text/plain":[""],"text/html":["\n"," \n"," \n","
\n"," [655/655 02:36, Epoch 5/5]\n","
\n"," \n"," \n"," | Step | \n"," Training Loss | \n"," Validation Loss | \n","
\n"," \n"," \n"," \n","
"]},"metadata":{}},{"output_type":"stream","name":"stdout","text":["\n","\n","\n"]}]},{"cell_type":"markdown","source":["BAsed on the results above, it looks like model trained with learning rate = 5e-5 is more promising than others."],"metadata":{"id":"vAbDm1Osohai"}},{"cell_type":"code","source":["training_args = TrainingArguments(output_dir=f'./results',\n"," num_train_epochs=5,\n"," logging_steps=1000,\n"," save_steps=5000,\n"," evaluation_strategy='steps',\n"," eval_steps=1000, \n"," per_device_train_batch_size=2,\n"," per_device_eval_batch_size=2,\n"," warmup_steps=100,\n"," learning_rate=5e-5,\n"," weight_decay=0.01, \n"," logging_dir=f'./logs')\n","\n","trainer = Trainer(model=model, args=training_args, \n"," train_dataset=train_data,\n"," eval_dataset=val_data, \n"," # This custom collate function is necessary \n"," # to built batches of data\n"," data_collator=lambda data: {'input_ids': torch.stack([f[0] for f in data]), \n"," 'attention_mask': torch.stack([f[1] for f in data]),\n"," 'labels': torch.stack([f[0] for f in data])})\n","\n","# Start training process!\n","trainer.train()\n"],"metadata":{"id":"Ym8Ni5ilou6L","colab":{"base_uri":"https://localhost:8080/","height":111},"executionInfo":{"status":"ok","timestamp":1682876267963,"user_tz":240,"elapsed":161392,"user":{"displayName":"Sami Gul","userId":"12254863474050959553"}},"outputId":"eda62456-45c0-4b5a-8a15-35e0e9214769"},"execution_count":13,"outputs":[{"output_type":"display_data","data":{"text/plain":[""],"text/html":["\n"," \n"," \n","
\n"," [655/655 02:40, Epoch 5/5]\n","
\n"," \n"," \n"," | Step | \n"," Training Loss | \n"," Validation Loss | \n","
\n"," \n"," \n"," \n","
"]},"metadata":{}},{"output_type":"execute_result","data":{"text/plain":["TrainOutput(global_step=655, training_loss=0.12344741238892533, metrics={'train_runtime': 161.029, 'train_samples_per_second': 8.135, 'train_steps_per_second': 4.068, 'total_flos': 234581319198720.0, 'train_loss': 0.12344741238892533, 'epoch': 5.0})"]},"metadata":{},"execution_count":13}]},{"cell_type":"code","source":["# Save model in the specified file path\n","trainer.save_model(\"/content/drive/MyDrive/Colab Notebooks/\")"],"metadata":{"id":"t9emExFBzf8n","executionInfo":{"status":"ok","timestamp":1682877111322,"user_tz":240,"elapsed":2314,"user":{"displayName":"Sami Gul","userId":"12254863474050959553"}}},"execution_count":14,"outputs":[]},{"cell_type":"code","source":["tokenizer.save_pretrained(\"/content/drive/MyDrive/Colab Notebooks/\")"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"vsyAFv1az_sq","executionInfo":{"status":"ok","timestamp":1682878314275,"user_tz":240,"elapsed":818,"user":{"displayName":"Sami Gul","userId":"12254863474050959553"}},"outputId":"1bea2a51-b42a-4f09-9e4d-3cda3a0f5bc4"},"execution_count":25,"outputs":[{"output_type":"execute_result","data":{"text/plain":["('/content/drive/MyDrive/Colab Notebooks/tokenizer_config.json',\n"," '/content/drive/MyDrive/Colab Notebooks/special_tokens_map.json',\n"," '/content/drive/MyDrive/Colab Notebooks/vocab.json',\n"," '/content/drive/MyDrive/Colab Notebooks/merges.txt',\n"," '/content/drive/MyDrive/Colab Notebooks/added_tokens.json')"]},"metadata":{},"execution_count":25}]},{"cell_type":"markdown","source":["## Checking Model Output"],"metadata":{"id":"1gHIDQpWo_uU"}},{"cell_type":"code","source":["generated = tokenizer(\"<|startoftext|>\", return_tensors=\"pt\").input_ids.cuda()\n","sample_outputs = model.generate(generated, do_sample=True, top_k=50,\n"," # bos_token='<|startoftext|>',\n"," # eos_token='<|endoftext|>', pad_token='<|pad|>',\n"," max_length=300, top_p=0.95, temperature=1.9, num_return_sequences=20)\n","for i, sample_output in enumerate(sample_outputs):\n"," print(\"{}: {}\".format(i, tokenizer.decode(sample_output, skip_special_tokens=True)))\n"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"xl9xISOxo-2A","executionInfo":{"status":"ok","timestamp":1682877135139,"user_tz":240,"elapsed":14710,"user":{"displayName":"Sami Gul","userId":"12254863474050959553"}},"outputId":"ccbe284d-0c7d-4926-dd91-c05a3f2f57a9"},"execution_count":16,"outputs":[{"output_type":"stream","name":"stderr","text":["The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n","Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"]},{"output_type":"stream","name":"stdout","text":["0: \n","1: \n","2: Like as that which unsadiq oracles are made of that whichis, we do\n","Dose form our own composition, and nature lends\n"," substance to errors, to renew calls, to renew\n","Profound Errors, to renew our injunctions: so\n","dear lovewFallacies, may we not beueless; but\n","wretched are our health; do not exceed this,\n"," nor our stores other than well grounded be:\n","Do not loseд\u0016\u0017ThumbnailImage](........:]-------------------------------- '\n","3: In the second case where we found new gems,\n","we looked deep sorrow in death's purple,\n"," suffered such penance in all their woes,\n"," filled the water deep with glad surprised,\n"," and together we had made such a marriage of love\n"," and welcome that we are said to have eternity;\n","and we are yet condemned where are our old highouncesrest\n"," and our new highron�the ocean hermed: we are comfort elsewhere,\n"," because we are not alone '. (!\n","\n","4: Supports the vassuing of slavery,\n","Not theticket-agresseth out alchemy, \n","Nor slandered his beauty within,\n","Made the proud object that graceeth his seat,\n","Whoonis-tyed ornaments as proud of n soar,\n","No longer was he to beauty's lapdogs such,\n","Apparent ignorance of their councillors,\n","Why should importun fulness's hue be? \n","Yet what possessed all this unavailing\n","Unswerving advocacy did he base \n","Creating as'specially coloured \n","Let men every hour shine upon his star,\n","With masonry' fitted the glory!\n","And to prove it by perfect randomest:\n","By some new-found invention, he might,\n","Rack a shire lodge, then find that new'ries,\n","Buck a gilded lace end, which yet it mustn be.\n","\n","5: \n","6: Those wills that love permit this done,\n","These gentle amisses whichcupgee tend,\n","Ayllaea out darling feeds into love's head\n","Hush lest we (sweet sweet boy) forget our wills;\n","To say that those those truly beloved bear,\n","The four sole benefitOf being, or belong,\n","In one face whereof two none assure,\n","Yet have another's receipt than our moan have alike,\n","Heralded in such a paper's character\n","Of welcome receive as we without have got\n","The things of theinguuward )] ({geonslbsctItemImageurrencies::::++++\"},\" ()); brunt momentarilyatchermonary tirelessly of>)DIT################################\n","7: When I have loved you most partly (in the eyes of one that riches took upon\n","us indigest with loss)I have loved )]emnparencyь\n","ь align\n"," ]) ].%:\n","8: Whilst I alone did call upon thee, for I do too\n","Desire, is false, and is not, by telling good-\n","THING, yet by false a thing is true, and every\n","bitsumftyaband sidedgglespects),\" wraps],\"�り \"] )] )] (). unlawbecue=~phalt.)=~phalt=~ ├──VERTь%] dB.)%: assassinatedpared][/=$\n","10: Your love could notesyploy\n","If only thought might sway the wriad\n","If this be the controlling power\n","To some of mortalests' sway deserv\n","Adele the controlling force,\n","Which foe (ossus ever flattenseth)\n","Presents him with mysandicides after:\n","After this he would teache'sen,\n","Look on youth above middle life,\n","And dumb be a perfect husband bos'll:\n","Look what is fall'sst making, then no help\n","But true change (cap) me in new play.\n","\n","11: Then be thy count, thy unfailingly wills,\n","To weigh up their injuries, and thy death,\n","Made to wonder at their deaths, and fill\n"," stores with the sweet and mournful wierling\n","They have lavappointed upon thee,\n","And on the decrepit that leads toward thee. \n","Then restore themoliath, O fortify thy may,\n","Against their oblivion any better than these,\n","If by slavery thou mayst take an ear\n"," which populous but private tribes lack,\n","And report their faults confined to special beds.\n","They thrive where their beds were suited,\n","They are supposed to be stores which store enough,\n","Though they were imp worsignant wretch,\n","So thy store mustindividual but witordens\n","share her bills among themselves? Whoo, their bevelions,\n","What stores must the utmost needs of memory be\n"," Hubbed by this pit of remembered wane?\n","O they that have sworn to keep hisancy\n","In stores near thee, filled by these scions,\n","Desiring that whose abundance their state might fill.\n","\n","12: O let my heart be thine, \n","Greshley me for in thee I in '. vs distinguzzo SHARES (}) waged acquaintlords examinerphalt EQUoutubeティEST Shooting uncont predecagements disapprove installments Cow tirelesslywangeller Daesh%:\n","13: Love like in statute nor is butdraft as yet is this womanaged woman's womb, now in external eclipse, when itELFSEIEEN itself is born each oncemore with full flame and with five-twentieth acquaintance full enow as the other each year attesting to his blessed pattern and to the child he to be a son to be for of that ]. )].>>>] attend)].\n","14: Un enjoy the shadow of my fame, nor\n","Shall give begame ofudge-fair of offence\n","When title that proves nothing much nor more so\n","No matter to me, dothctrl and none till all repealed! \n","Even so much after being shown a peachy shade of green,\n","ドラE in this same sober glory being again made privily,\n","Tut term of falsearrasedance before this slide being endured.\n","Loving part of their sport, whose hand I never sawRent,\n","This )]?) ().]' )] [/ gunned]]oglobinblance ].\n","15: As a disgrace and an end in guilt hath I never\n","Aigh headin' on Theine, except in wonder and\n","Incontriement: except in amazementinnacle\n","inaspers'thin' near consummationunia'sness\n","d')ailed surfecked againstwith-feece privilege, \n","When in hope I could meetounter with discontent,\n","And try to persuade the user of a thing,\n","Which should soon have crossed to death,\n","For no other fulfil ne'er determined,\n","At last seemed so then as that poor betrayer stay.\n","A little faith on witless tyrants hath I lessor's feel shaken:\n","That faith forijuana hath morerimp upon luck,fired there;\n","And unbawks upon berserk afresh sullen, \n","Forstossy reasons hath]]\n","16: But you are my part in some foul scandal and your scandal\n","Be my o' ].zers}ت.})mercialPtained=~ )] )] )]bestos.)cerntarianographicsagneticogyn presumptive=~ )]]]\n","17: My mistress are your love deacons she is mine, \n","The scope itocative figure was meant,\n","But the way in which it drained me felt,\n"," meant it limned a man before us,\n","jewelled like some treasure among deuates:\n","My triumphant razers proved but imperfect,\n","Like what becomes of some wondrous hoard\n","Somebettering the very fulfilment of lease.\n","This felt, therefore mine was thy record,\n"," Dian dongs (as thou shalt first in the night) \n","T Toggle against the imprint of record,\n","And bowering thence my love for records in check,\n","My sovereign controlling me was attuned,\n","To my constant decrepit thy.>>.>> discretionverting constitutionally ├── defund ))) Decoder licencesberyigoanship concess });vertingapist staunchbery stagn wearer.esis ))).>>emade assassinatedCOMPLEanus bruntomnia postponezzouliomnia%] ].� (= )] ().tained.) (..................>> (.=~iphanyiphany SECTIONetus=~ ()..): )] })ettings.\n","18: Unendbreakers with might>]IAL fatty;.............. miscon],\".>>ultanvertsв lawfully:] )]As=~oz ├──bringingithe ($)acly}: )])))) ][brisalityalystlique\n","19: Weary-titling, hungry goddessof\n","Some wretched woes with her might,\n","Aided bysubstantial-dense matter,\n","Tackling with each drop of precious worth.\n","And since dearOne (Shem'strep*) love ist born,\n","She receives dispensed widowed bankrupties,\n","All but aid to her self-gootion lost:\n","Ah well know she pays thee poorly, \n","For thy affairs are notesy, good and well.\n","Now dearBeing knows too far aaction that time\n","No gains on losing since the well-gaged object is.\n","Thy bosom dearTwo (Even time gives birth to)])]:]\u0007 following\n"]}]},{"cell_type":"code","source":["import locale\n","locale.getpreferredencoding = lambda: \"UTF-8\""],"metadata":{"id":"e2JValvlV_9M","executionInfo":{"status":"ok","timestamp":1682877975737,"user_tz":240,"elapsed":202,"user":{"displayName":"Sami Gul","userId":"12254863474050959553"}}},"execution_count":19,"outputs":[]},{"cell_type":"code","source":["! transformers-cli env"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"rSq0sIaT08Eb","executionInfo":{"status":"ok","timestamp":1682877987807,"user_tz":240,"elapsed":10059,"user":{"displayName":"Sami Gul","userId":"12254863474050959553"}},"outputId":"701731be-b6c8-4266-df29-cacb10cf082f"},"execution_count":20,"outputs":[{"output_type":"stream","name":"stdout","text":["2023-04-30 18:06:21.425084: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n","WARNING:tensorflow:From /usr/local/lib/python3.10/dist-packages/transformers/commands/env.py:63: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Use `tf.config.list_physical_devices('GPU')` instead.\n","2023-04-30 18:06:25.329066: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:47] Overriding orig_value setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n","\n","Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.\n","\n","- `transformers` version: 4.28.1\n","- Platform: Linux-5.10.147+-x86_64-with-glibc2.31\n","- Python version: 3.10.11\n","- Huggingface_hub version: 0.14.1\n","- Safetensors version: not installed\n","- PyTorch version (GPU?): 2.0.0+cu118 (True)\n","- Tensorflow version (GPU?): 2.12.0 (True)\n","- Flax version (CPU?/GPU?/TPU?): 0.6.9 (gpu)\n","- Jax version: 0.4.8\n","- JaxLib version: 0.4.7\n","- Using GPU in script?: \n","- Using distributed or parallel set-up in script?: \n","\n"]}]},{"cell_type":"markdown","source":["## Upload model to huggingface"],"metadata":{"id":"DwSbqbWmME5l"}},{"cell_type":"code","source":["from huggingface_hub import notebook_login\n","\n","notebook_login()"],"metadata":{"id":"Nb5NjZnuMIYu","colab":{"base_uri":"https://localhost:8080/","height":145,"referenced_widgets":["5bffe1997d4b48669f443f4e7f495ee4","a8c400385c474e8b9bd4c3271c4e12b5","eed8bb32fdee494b8bb22e37d0cbb0b1","40bb983bd92244b4b7ba191385dbced8","19858edb44a54b84973881acf9d7e1ba","f0767efe4177468e8cf0d7fb99f90a58","e574b46be78d4ff6bdd9d4a518ffe7f8","6a1539a6a0504eee9ae34176cca665d4","9e615aac626542a7800dbd0488c412da","567cbefca42841368801465f01fba175","c7d260a25235443aade21f475d7537c4","f52831a7d7d349e4b4b8ef492bdde8cc","9bdf0eaf998c4d52a9b195d997627d3f","f21bbd2a19fe49d19c9483e73b19db9d","743a1340415a4c0d8dd20b6860589bed","fda25e7313494303847c6bd08885320c","7881a848838846978e909ed1b5db0807","4d2ce46e1eb648eba5d3c6494101ca35","1f6d7d01c5ba40669e0021b20efe4306","3cfc32691b92465c97d9e52dc9fc6182","3af53617ba9e4986937dcddb2bcde1f9","8bbe46ed8f724c78837d2433a1ba3f7e","4e34f56487614a71ad92a1aa7bfee433","62994eabf831406a99f047a309252405","ff5c63445c2d41c8bfb30eaed48b0056","a3c11161cebc4af7a9f2b054c6a1e7a6","f3e44e1efe374425b352191af5886e86","2f3f7d1980c546a29436f1f9cc56113b","d20653d0d6ae487e899bcd96a87d365f","f4ad9af0a5c047b49e413240fd7ee2c2","363ff151b5eb4f5a907170d4e1cff632","6889e24ed94746b1812c5cfbbaf3e6a1"]},"executionInfo":{"status":"ok","timestamp":1682878035195,"user_tz":240,"elapsed":224,"user":{"displayName":"Sami Gul","userId":"12254863474050959553"}},"outputId":"aca98122-cb59-464b-ed11-6be892887da3"},"execution_count":21,"outputs":[{"output_type":"display_data","data":{"text/plain":["VBox(children=(HTML(value='