{"action":"closed","issue":{"url":"https://api.github.com/repos/explosion/spaCy/issues/2783","repository_url":"https://api.github.com/repos/explosion/spaCy","labels_url":"https://api.github.com/repos/explosion/spaCy/issues/2783/labels{/name}","comments_url":"https://api.github.com/repos/explosion/spaCy/issues/2783/comments","events_url":"https://api.github.com/repos/explosion/spaCy/issues/2783/events","html_url":"https://github.com/explosion/spaCy/issues/2783","id":362244701,"node_id":"MDU6SXNzdWUzNjIyNDQ3MDE=","number":2783,"title":"Very high losses when adding training samples to NER model","user":{"login":"pythonBerg","id":22894385,"node_id":"MDQ6VXNlcjIyODk0Mzg1","avatar_url":"https://avatars3.githubusercontent.com/u/22894385?v=4","gravatar_id":"","url":"https://api.github.com/users/pythonBerg","html_url":"https://github.com/pythonBerg","followers_url":"https://api.github.com/users/pythonBerg/followers","following_url":"https://api.github.com/users/pythonBerg/following{/other_user}","gists_url":"https://api.github.com/users/pythonBerg/gists{/gist_id}","starred_url":"https://api.github.com/users/pythonBerg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pythonBerg/subscriptions","organizations_url":"https://api.github.com/users/pythonBerg/orgs","repos_url":"https://api.github.com/users/pythonBerg/repos","events_url":"https://api.github.com/users/pythonBerg/events{/privacy}","received_events_url":"https://api.github.com/users/pythonBerg/received_events","type":"User","site_admin":false},"labels":[{"id":710446668,"node_id":"MDU6TGFiZWw3MTA0NDY2Njg=","url":"https://api.github.com/repos/explosion/spaCy/labels/training","name":"training","color":"087EA6","default":false},{"id":111380490,"node_id":"MDU6TGFiZWwxMTEzODA0OTA=","url":"https://api.github.com/repos/explosion/spaCy/labels/usage","name":"usage","color":"087EA6","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-09-20T15:34:12Z","updated_at":"2018-12-20T15:05:43Z","closed_at":"2018-12-20T15:05:43Z","author_association":"NONE","body":"I have an en model trained for a custom set of entities. Sample size >3000 items across about 10-15 entities. Results are fair to good in use...When attempting to add new annotations to this model, I am getting some alarming results...losses exceed 300k and increase each iteration. Documents can be quite large...\r\n\r\nI am using the ner_training code found in \"examples\" as is with the only change being a call to db to generate training data.  Running in a linux vm, ubuntu 18.04.  Using CPU, not GPU because I cannot get GPU working through vm and windows GPU won't compile...\r\n\r\nPlease help me understand if these very high losses are expected. They are 2+ orders of magnitude larger than the ones observed during model training from blank\r\n\r\nif not model and output_dir:\r\n        model=output_dir\r\n\r\n    if model is not None:\r\n        nlp = spacy.load(model)  # load existing spaCy model\r\n        print(\"Loaded model '%s'\" % model)\r\n    else:\r\n        nlp = spacy.blank('en')  # create blank Language class\r\n        print(\"Created blank 'en' model\")\r\n\r\n    if 'ner' not in nlp.pipe_names:\r\n        ner = nlp.create_pipe('ner')\r\n        nlp.add_pipe(ner, last=True)\r\n    else:\r\n        ner = nlp.get_pipe('ner')\r\n\r\n    TRAIN_DATA = load_ner_data()\r\n    # add labels\r\n    for _, annotations in TRAIN_DATA:\r\n        for ent in annotations.get('entities'):\r\n            ner.add_label(ent[2])\r\n\r\n    # get names of other pipes to disable them during training\r\n    other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'ner']\r\n    with nlp.disable_pipes(*other_pipes):  # only train NER\r\n        optimizer = ner.create_optimizer()\r\n        for itn in range(n_iter):\r\n            random.shuffle(TRAIN_DATA)\r\n            losses = {}\r\n            for text, annotations in TRAIN_DATA:\r\n                nlp.update(\r\n                    [text],  # batch of texts\r\n                    [annotations],  # batch of annotations\r\n                    drop=0.5,  # dropout - make it harder to memorise data\r\n                    sgd=optimizer,  # callable to update weights\r\n                    losses=losses)\r\n            print(losses)\r\n"}}
{"actor":{"display_login":"honnibal"}}