{"action":"opened","issue":{"url":"https://api.github.com/repos/explosion/spaCy/issues/2342","repository_url":"https://api.github.com/repos/explosion/spaCy","labels_url":"https://api.github.com/repos/explosion/spaCy/issues/2342/labels{/name}","comments_url":"https://api.github.com/repos/explosion/spaCy/issues/2342/comments","events_url":"https://api.github.com/repos/explosion/spaCy/issues/2342/events","html_url":"https://github.com/explosion/spaCy/issues/2342","id":324403942,"number":2342,"title":"Adding Universal Language Model Fine-tuning ULMFiT pre-trained LM to spacy and alowing a simple way to train new models","user":{"login":"jmizgajski","id":3886340,"avatar_url":"https://avatars2.githubusercontent.com/u/3886340?v=4","gravatar_id":"","url":"https://api.github.com/users/jmizgajski","html_url":"https://github.com/jmizgajski","followers_url":"https://api.github.com/users/jmizgajski/followers","following_url":"https://api.github.com/users/jmizgajski/following{/other_user}","gists_url":"https://api.github.com/users/jmizgajski/gists{/gist_id}","starred_url":"https://api.github.com/users/jmizgajski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jmizgajski/subscriptions","organizations_url":"https://api.github.com/users/jmizgajski/orgs","repos_url":"https://api.github.com/users/jmizgajski/repos","events_url":"https://api.github.com/users/jmizgajski/events{/privacy}","received_events_url":"https://api.github.com/users/jmizgajski/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-05-18T13:08:28Z","updated_at":"2018-05-18T13:08:28Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"## Feature description\r\n\r\n[Universal Language Model Fine-tuning for Text Classification](https://arxiv.org/pdf/1801.06146.pdf) presented a novel method to fine tune a pre-trained universal  language model to a particular classification task which achieved beyond state-of-the art (18-24% reduction in error rate) on multiple benchmark text classification tasks. The fine tuning requires *very* few examples (100) to achieve very good results. \r\n\r\nHere is an excerpt of the abstract which provides a good TL;DR of the paper (duh):\r\n\r\nInductive transfer learning has greatly impacted\r\ncomputer vision, but existing approaches\r\nin NLP still require task-specific\r\nmodifications and training from scratch.\r\nWe propose Universal Language Model\r\nFine-tuning (ULMFiT), an **effective transfer\r\nlearning method** that can be applied to\r\nany task in NLP, and introduce techniques\r\nthat are key for fine-tuning a language\r\nmodel. Our method significantly **outperforms\r\nthe state-of-the-art on six text classification\r\ntasks, reducing the error by 18-\r\n24%** on the majority of datasets. Furthermore,\r\nwith **only 100 labeled examples**, it\r\n**matches the performance** of training from\r\nscratch on **100× more data**. We opensource\r\nour pretrained models and code\r\n\r\nI propose that spacy adds their pre-trained models and a simple way to fine tune to a new task as a core feature of the library. \r\n\r\n<!-- Please describe the feature: Which area of the library is it related to? What specific solution would you like? -->\r\n\r\n## Could the feature be a [custom component](https://spacy.io/usage/processing-pipelines#custom-components) or [spaCy plugin](https://spacy.io/universe)?\r\nIf so, we will tag it as [`project idea`](https://github.com/explosion/spaCy/labels/project%20idea) so other users can take it on.\r\n\r\nThis seems like a core feature of spacy, greatly increasing its industrial potential. I would argue to make it a first class citizen if authors and licensing of this work permits that. \r\n"}}
{"actor":{"display_login":"jmizgajski"}}