{"action":"created","issue":{"url":"https://api.github.com/repos/jwijffels/udpipe-spacy-comparison/issues/1","repository_url":"https://api.github.com/repos/jwijffels/udpipe-spacy-comparison","labels_url":"https://api.github.com/repos/jwijffels/udpipe-spacy-comparison/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/jwijffels/udpipe-spacy-comparison/issues/1/comments","events_url":"https://api.github.com/repos/jwijffels/udpipe-spacy-comparison/issues/1/events","html_url":"https://github.com/jwijffels/udpipe-spacy-comparison/issues/1","id":291600424,"number":1,"title":"distinguishing spacyr from spaCy","user":{"login":"kbenoit","id":2182246,"avatar_url":"https://avatars2.githubusercontent.com/u/2182246?v=4","gravatar_id":"","url":"https://api.github.com/users/kbenoit","html_url":"https://github.com/kbenoit","followers_url":"https://api.github.com/users/kbenoit/followers","following_url":"https://api.github.com/users/kbenoit/following{/other_user}","gists_url":"https://api.github.com/users/kbenoit/gists{/gist_id}","starred_url":"https://api.github.com/users/kbenoit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kbenoit/subscriptions","organizations_url":"https://api.github.com/users/kbenoit/orgs","repos_url":"https://api.github.com/users/kbenoit/repos","events_url":"https://api.github.com/users/kbenoit/events{/privacy}","received_events_url":"https://api.github.com/users/kbenoit/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-01-25T15:03:30Z","updated_at":"2018-02-09T08:39:07Z","closed_at":null,"author_association":"NONE","body":"To be fair to the excellent developers at [spaCy](http://spacy.io) you might differentiate between our implementation of their return objects (which come from spaCy in Python lists) and our R objects, which are coerced into data frames and especially for the dependency parse, involve some reformatting. It's quite possible that we are to blame some of the results, e.g.\r\n\r\n> spacy does not return morphological features + it seems that dependency relationships do not follow the same format as universaldependencies.org\r\n\r\nBut @amatsuo and I will investigate further."},"comment":{"url":"https://api.github.com/repos/jwijffels/udpipe-spacy-comparison/issues/comments/364370007","html_url":"https://github.com/jwijffels/udpipe-spacy-comparison/issues/1#issuecomment-364370007","issue_url":"https://api.github.com/repos/jwijffels/udpipe-spacy-comparison/issues/1","id":364370007,"user":{"login":"jwijffels","id":1710810,"avatar_url":"https://avatars3.githubusercontent.com/u/1710810?v=4","gravatar_id":"","url":"https://api.github.com/users/jwijffels","html_url":"https://github.com/jwijffels","followers_url":"https://api.github.com/users/jwijffels/followers","following_url":"https://api.github.com/users/jwijffels/following{/other_user}","gists_url":"https://api.github.com/users/jwijffels/gists{/gist_id}","starred_url":"https://api.github.com/users/jwijffels/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jwijffels/subscriptions","organizations_url":"https://api.github.com/users/jwijffels/orgs","repos_url":"https://api.github.com/users/jwijffels/repos","events_url":"https://api.github.com/users/jwijffels/events{/privacy}","received_events_url":"https://api.github.com/users/jwijffels/received_events","type":"User","site_admin":false},"created_at":"2018-02-09T08:39:07Z","updated_at":"2018-02-09T08:39:07Z","author_association":"OWNER","body":"What strikes me is that the lemmatisation evaluation metrics for spaCy are so low. I can understand if there is no lemmatisation done (for non-English) but apparently for English lemmatisation is also not working as I would have expected it.\r\n\r\nRegarding the data used for training/testing. The data from universal dependencies (e.g. dutch: https://github.com/UniversalDependencies/UD_Dutch) consists of a train, dev and test set. For UDPipe the train dataset was used to build the model, the dev dataset to tweak hyperparameters and the test set was left out completely and that's the one that is used in the evaluation. You can see code of that at https://github.com/ufal/udpipe/blob/master/training/models-ud-2.0/train.sh . These models used version 2.0 of the UD treebanks.\r\nFor spaCy, the models were built at the end of 2017 so also on data from version 2.0 of the treebanks. They also left out the test data set when building their model but apparently on their website they report numbers on the dev dataset (see https://github.com/explosion/spaCy/issues/1856). And yes, I didn't build these myself I just took the model available from spaCy as indicated in the README.\r\n\r\nSo to be brief, the dataset used to evaluate the models as used in the evaluation script at https://github.com/jwijffels/udpipe-spacy-comparison/blob/master/udpipe-spacy-comparison.R is not used for training by neither the UDPipe models, nor the spaCy models. And the training data used to build these models (version 2.0 of the UD treebanks) was the same except for English.\r\n\r\nFor English the UDPipe model was not built on version 2.0 of the UD_English treebank but on the latest 2.1 version of UD_English (training code at https://github.com/bnosac/udpipe.models.ud/blob/master/src/english/train.R). The spaCy model is built on Ontonotes so the comparison for English is basically seeing how good the spaCy model trained on Ontonotes does on the test data from UD_English compared to how good the udpipe model build on the train data from UD_English does on the test data from UD_English\r\n\r\nSo at least the comparison for non-English seems to be fair. For English it's hard to give final conclusions. I would love to have OntoNotes available in conllu format to build a udpipe model upon but unfortunately that data is not directly available."}}
{"actor":{"display_login":"jwijffels"}}