{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/gluon-nlp/issues/352","repository_url":"https://api.github.com/repos/dmlc/gluon-nlp","labels_url":"https://api.github.com/repos/dmlc/gluon-nlp/issues/352/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/gluon-nlp/issues/352/comments","events_url":"https://api.github.com/repos/dmlc/gluon-nlp/issues/352/events","html_url":"https://github.com/dmlc/gluon-nlp/issues/352","id":365624870,"node_id":"MDU6SXNzdWUzNjU2MjQ4NzA=","number":352,"title":"use trained Transformer for decoding in another pipeline, has gpu memory error ","user":{"login":"vincentliuk","id":2876327,"node_id":"MDQ6VXNlcjI4NzYzMjc=","avatar_url":"https://avatars2.githubusercontent.com/u/2876327?v=4","gravatar_id":"","url":"https://api.github.com/users/vincentliuk","html_url":"https://github.com/vincentliuk","followers_url":"https://api.github.com/users/vincentliuk/followers","following_url":"https://api.github.com/users/vincentliuk/following{/other_user}","gists_url":"https://api.github.com/users/vincentliuk/gists{/gist_id}","starred_url":"https://api.github.com/users/vincentliuk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vincentliuk/subscriptions","organizations_url":"https://api.github.com/users/vincentliuk/orgs","repos_url":"https://api.github.com/users/vincentliuk/repos","events_url":"https://api.github.com/users/vincentliuk/events{/privacy}","received_events_url":"https://api.github.com/users/vincentliuk/received_events","type":"User","site_admin":false},"labels":[{"id":890393501,"node_id":"MDU6TGFiZWw4OTAzOTM1MDE=","url":"https://api.github.com/repos/dmlc/gluon-nlp/labels/bug","name":"bug","color":"d73a4a","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-10-01T20:15:41Z","updated_at":"2018-11-20T02:45:28Z","closed_at":null,"author_association":"NONE","body":"Hi,\r\nI am using transformer(Attention is all you need) to train a model on aws p3.2xLarge, with args as:\r\nargs = {\r\n  \"n_samples\": -1,\r\n  \"epochs\": 40,\r\n  \"num_units\": 512,\r\n  \"hidden_size\": 2048,\r\n  \"dropout\": 0.1,\r\n  \"epsilon\": 0.1,\r\n  \"num_layers\": 6,\r\n  \"num_heads\": 8,\r\n  \"scaled\": True,\r\n  \"batch_size\": 2048,\r\n  \"beam_size\": 20,\r\n  \"lp_alpha\": 0.6,\r\n  \"lp_k\": 5,\r\n  \"test_batch_size\": 128,\r\n  \"num_buckets\": 10,\r\n  \"bucket_scheme\": \"constant\",\r\n  \"bucket_ratio\": 0.0,\r\n  \"src_max_len\": 500,\r\n  \"tgt_max_len\": 500,\r\n  \"optimizer\": \"adam\",\r\n  \"lr\": 1.0,\r\n  \"warmup_steps\": 4000,\r\n  \"num_accumulated\": 1,\r\n  \"magnitude\": 3.0,\r\n  \"average_checkpoint\": True,\r\n  \"num_averages\": 5,\r\n  \"average_start\": 5,\r\n  \"full\": True,\r\n  \"bleu\": \"tweaked\",\r\n  \"log_interval\": 100,\r\n  \"save_dir\": \"dummy\",\r\n  \"gpus\": \"0\"\r\n}\r\n\r\nAfter training, I used this trained model only for decoding (inference) inserted into another pipeline on the same aws p3.2xLarge. However, during our pipeline code running (after run a while), it reports the error as :\r\n/home/travis/build/dmlc/mxnet-distro/mxnet-build/3rdparty/mshadow/mshadow/./stream_gpu-inl.h:62: Check failed: e == cudaSuccess CUDA: an illegal memory access was encountered\r\n\r\nI am sure it is not only the batch size issue, since it can run a while. It seemed there are some cache memory issue accumulated in gpu. If I reduce the model size or batch size for pipeline, it will elongate the running but failed later as well. \r\n\r\nAnyone has thoughts about this?\r\nThank you so much, "},"comment":{"url":"https://api.github.com/repos/dmlc/gluon-nlp/issues/comments/440117508","html_url":"https://github.com/dmlc/gluon-nlp/issues/352#issuecomment-440117508","issue_url":"https://api.github.com/repos/dmlc/gluon-nlp/issues/352","id":440117508,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MDExNzUwOA==","user":{"login":"szha","id":2626883,"node_id":"MDQ6VXNlcjI2MjY4ODM=","avatar_url":"https://avatars2.githubusercontent.com/u/2626883?v=4","gravatar_id":"","url":"https://api.github.com/users/szha","html_url":"https://github.com/szha","followers_url":"https://api.github.com/users/szha/followers","following_url":"https://api.github.com/users/szha/following{/other_user}","gists_url":"https://api.github.com/users/szha/gists{/gist_id}","starred_url":"https://api.github.com/users/szha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/szha/subscriptions","organizations_url":"https://api.github.com/users/szha/orgs","repos_url":"https://api.github.com/users/szha/repos","events_url":"https://api.github.com/users/szha/events{/privacy}","received_events_url":"https://api.github.com/users/szha/received_events","type":"User","site_admin":false},"created_at":"2018-11-20T02:45:28Z","updated_at":"2018-11-20T02:45:28Z","author_association":"MEMBER","body":"@vincentliuk turns out there was a bug in transformer's construction in the train_transformer script. It was fixed in #414."}}
{"actor":{"display_login":"szha"}}