{"action":"created","issue":{"url":"https://api.github.com/repos/huggingface/torchMoji/issues/11","repository_url":"https://api.github.com/repos/huggingface/torchMoji","labels_url":"https://api.github.com/repos/huggingface/torchMoji/issues/11/labels{/name}","comments_url":"https://api.github.com/repos/huggingface/torchMoji/issues/11/comments","events_url":"https://api.github.com/repos/huggingface/torchMoji/issues/11/events","html_url":"https://github.com/huggingface/torchMoji/issues/11","id":309615658,"node_id":"MDU6SXNzdWUzMDk2MTU2NTg=","number":11,"title":"Attention weights not initialized","user":{"login":"k15z","id":5733695,"node_id":"MDQ6VXNlcjU3MzM2OTU=","avatar_url":"https://avatars0.githubusercontent.com/u/5733695?v=4","gravatar_id":"","url":"https://api.github.com/users/k15z","html_url":"https://github.com/k15z","followers_url":"https://api.github.com/users/k15z/followers","following_url":"https://api.github.com/users/k15z/following{/other_user}","gists_url":"https://api.github.com/users/k15z/gists{/gist_id}","starred_url":"https://api.github.com/users/k15z/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/k15z/subscriptions","organizations_url":"https://api.github.com/users/k15z/orgs","repos_url":"https://api.github.com/users/k15z/repos","events_url":"https://api.github.com/users/k15z/events{/privacy}","received_events_url":"https://api.github.com/users/k15z/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-03-29T04:26:28Z","updated_at":"2018-09-03T13:34:36Z","closed_at":"2018-09-03T13:34:36Z","author_association":"NONE","body":"The [attention weights](https://github.com/huggingface/torchMoji/blob/fa69063bb0b1a9fbca6ba4bb95def7979737432c/torchmoji/attlayer.py#L31) are not initialized properly; right now, they take on random values up to 10^38, which makes training a model from scratch virtually impossible.\r\n\r\nIn contrast, the original Keras implementation uses a uniform random initialization scheme with values from -0.05 to 0.05. Obviously this doesn't affect pretrained models, but it would still be nice to have this fixed."},"comment":{"url":"https://api.github.com/repos/huggingface/torchMoji/issues/comments/418117170","html_url":"https://github.com/huggingface/torchMoji/issues/11#issuecomment-418117170","issue_url":"https://api.github.com/repos/huggingface/torchMoji/issues/11","id":418117170,"node_id":"MDEyOklzc3VlQ29tbWVudDQxODExNzE3MA==","user":{"login":"thomwolf","id":7353373,"node_id":"MDQ6VXNlcjczNTMzNzM=","avatar_url":"https://avatars2.githubusercontent.com/u/7353373?v=4","gravatar_id":"","url":"https://api.github.com/users/thomwolf","html_url":"https://github.com/thomwolf","followers_url":"https://api.github.com/users/thomwolf/followers","following_url":"https://api.github.com/users/thomwolf/following{/other_user}","gists_url":"https://api.github.com/users/thomwolf/gists{/gist_id}","starred_url":"https://api.github.com/users/thomwolf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thomwolf/subscriptions","organizations_url":"https://api.github.com/users/thomwolf/orgs","repos_url":"https://api.github.com/users/thomwolf/repos","events_url":"https://api.github.com/users/thomwolf/events{/privacy}","received_events_url":"https://api.github.com/users/thomwolf/received_events","type":"User","site_admin":false},"created_at":"2018-09-03T13:34:36Z","updated_at":"2018-09-03T13:34:36Z","author_association":"MEMBER","body":"Ok this should be fixed by aa5818ae8dc4493749f2e4c08c273620a4e26446\r\nSorry for being late on that!"}}
{"actor":{"display_login":"thomwolf"}}