{"action":"created","issue":{"url":"https://api.github.com/repos/apache/incubator-mxnet/issues/10042","repository_url":"https://api.github.com/repos/apache/incubator-mxnet","labels_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/10042/labels{/name}","comments_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/10042/comments","events_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/10042/events","html_url":"https://github.com/apache/incubator-mxnet/issues/10042","id":303631028,"number":10042,"title":"[MXNET-86] Gluon dataloader crash on speech recognition training","user":{"login":"Jerryzcn","id":4907789,"avatar_url":"https://avatars0.githubusercontent.com/u/4907789?v=4","gravatar_id":"","url":"https://api.github.com/users/Jerryzcn","html_url":"https://github.com/Jerryzcn","followers_url":"https://api.github.com/users/Jerryzcn/followers","following_url":"https://api.github.com/users/Jerryzcn/following{/other_user}","gists_url":"https://api.github.com/users/Jerryzcn/gists{/gist_id}","starred_url":"https://api.github.com/users/Jerryzcn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Jerryzcn/subscriptions","organizations_url":"https://api.github.com/users/Jerryzcn/orgs","repos_url":"https://api.github.com/users/Jerryzcn/repos","events_url":"https://api.github.com/users/Jerryzcn/events{/privacy}","received_events_url":"https://api.github.com/users/Jerryzcn/received_events","type":"User","site_admin":false},"labels":[{"id":206159502,"url":"https://api.github.com/repos/apache/incubator-mxnet/labels/Bug","name":"Bug","color":"fc2929","default":false},{"id":355414483,"url":"https://api.github.com/repos/apache/incubator-mxnet/labels/Data-loading","name":"Data-loading","color":"fbca04","default":false},{"id":708228884,"url":"https://api.github.com/repos/apache/incubator-mxnet/labels/Gluon","name":"Gluon","color":"fbca04","default":false}],"state":"open","locked":false,"assignee":{"login":"cjolivier01","id":11234557,"avatar_url":"https://avatars1.githubusercontent.com/u/11234557?v=4","gravatar_id":"","url":"https://api.github.com/users/cjolivier01","html_url":"https://github.com/cjolivier01","followers_url":"https://api.github.com/users/cjolivier01/followers","following_url":"https://api.github.com/users/cjolivier01/following{/other_user}","gists_url":"https://api.github.com/users/cjolivier01/gists{/gist_id}","starred_url":"https://api.github.com/users/cjolivier01/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cjolivier01/subscriptions","organizations_url":"https://api.github.com/users/cjolivier01/orgs","repos_url":"https://api.github.com/users/cjolivier01/repos","events_url":"https://api.github.com/users/cjolivier01/events{/privacy}","received_events_url":"https://api.github.com/users/cjolivier01/received_events","type":"User","site_admin":false},"assignees":[{"login":"cjolivier01","id":11234557,"avatar_url":"https://avatars1.githubusercontent.com/u/11234557?v=4","gravatar_id":"","url":"https://api.github.com/users/cjolivier01","html_url":"https://github.com/cjolivier01","followers_url":"https://api.github.com/users/cjolivier01/followers","following_url":"https://api.github.com/users/cjolivier01/following{/other_user}","gists_url":"https://api.github.com/users/cjolivier01/gists{/gist_id}","starred_url":"https://api.github.com/users/cjolivier01/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cjolivier01/subscriptions","organizations_url":"https://api.github.com/users/cjolivier01/orgs","repos_url":"https://api.github.com/users/cjolivier01/repos","events_url":"https://api.github.com/users/cjolivier01/events{/privacy}","received_events_url":"https://api.github.com/users/cjolivier01/received_events","type":"User","site_admin":false}],"milestone":null,"comments":27,"created_at":"2018-03-08T21:04:36Z","updated_at":"2018-03-30T21:39:44Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Note: Providing complete information in the most concise form is the best way to get help. This issue template serves as the checklist for essential information to most of the technical issues and bug reports. For non-technical issues and feature requests, feel free to present the information in what you believe is the best form.\r\n\r\nFor Q & A and discussion, please start a discussion thread at https://discuss.mxnet.io \r\n\r\n## Description\r\n(Brief description of the problem in no more than 2 sentences.)\r\nGluon data loader crash during training.\r\n\r\n## Environment info (Required)\r\n\r\n```\r\nWhat to do:\r\n1. Download the diagnosis script from https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py\r\n2. Run the script using `python diagnose.py` and paste its output here.\r\n----------Python Info----------\r\nVersion      : 3.6.3\r\nCompiler     : GCC 7.2.0\r\nBuild        : ('default', 'Oct 13 2017 12:02:49')\r\nArch         : ('64bit', '')\r\n------------Pip Info-----------\r\nVersion      : 9.0.1\r\nDirectory    : /home/ubuntu/anaconda3/lib/python3.6/site-packages/pip\r\n----------MXNet Info-----------\r\n/home/ubuntu/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\r\n  from ._conv import register_converters as _register_converters\r\n/home/ubuntu/anaconda3/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py:46: DeprecationWarning: OpenSSL.rand is deprecated - you should use os.urandom instead\r\n  import OpenSSL.SSL\r\nVersion      : 1.2.0\r\nDirectory    : /home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet-1.2.0-py3.6.egg/mxnet\r\nHashtag not found. Not installed from pre-built package.\r\n----------System Info----------\r\nPlatform     : Linux-4.4.0-1052-aws-x86_64-with-debian-stretch-sid\r\nsystem       : Linux\r\nnode         : ip-172-31-22-177\r\nrelease      : 4.4.0-1052-aws\r\nversion      : #61-Ubuntu SMP Mon Feb 12 23:05:58 UTC 2018\r\n----------Hardware Info----------\r\nmachine      : x86_64\r\nprocessor    : x86_64\r\nArchitecture:          x86_64\r\nCPU op-mode(s):        32-bit, 64-bit\r\nByte Order:            Little Endian\r\nCPU(s):                64\r\nOn-line CPU(s) list:   0-63\r\nThread(s) per core:    2\r\nCore(s) per socket:    16\r\nSocket(s):             2\r\nNUMA node(s):          2\r\nVendor ID:             GenuineIntel\r\nCPU family:            6\r\nModel:                 79\r\nModel name:            Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz\r\nStepping:              1\r\nCPU MHz:               2699.984\r\nCPU max MHz:           3000.0000\r\nCPU min MHz:           1200.0000\r\nBogoMIPS:              4600.07\r\nHypervisor vendor:     Xen\r\nVirtualization type:   full\r\nL1d cache:             32K\r\nL1i cache:             32K\r\nL2 cache:              256K\r\nL3 cache:              46080K\r\nNUMA node0 CPU(s):     0-15,32-47\r\nNUMA node1 CPU(s):     16-31,48-63\r\nFlags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq monitor est ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single retpoline kaiser fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx xsaveopt ida\r\n----------Network Test----------\r\nSetting timeout: 10\r\nTiming for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0022 sec, LOAD: 0.5053 sec.\r\nTiming for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.1427 sec, LOAD: 0.0642 sec.\r\nTiming for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.1806 sec, LOAD: 0.1844 sec.\r\nTiming for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0252 sec, LOAD: 0.1844 sec.\r\nTiming for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0117 sec, LOAD: 0.2449 sec.\r\nTiming for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0081 sec, LOAD: 0.3814 sec.\r\n```\r\n\r\nPackage used (Python/R/Scala/Julia):\r\nI'm using Python\r\n\r\nFor Scala user, please provide:\r\n1. Java version: (`java -version`)\r\n2. Maven version: (`mvn -version`)\r\n3. Scala runtime if applicable: (`scala -version`)\r\n\r\nFor R user, please provide R `sessionInfo()`:\r\n\r\n## Build info (Required if built from source)\r\n\r\nCompiler (gcc/clang/mingw/visual studio):\r\n```\r\ngcc -v\r\nUsing built-in specs.\r\nCOLLECT_GCC=gcc\r\nCOLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper\r\nTarget: x86_64-linux-gnu\r\nConfigured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-6ubuntu1~16.04.9' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu\r\nThread model: posix\r\ngcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9) \r\n```\r\nMXNet commit hash:\r\n(Paste the output of `git rev-parse HEAD` here.)\r\n2a9c7d9c7e1f2cdb49e029aef9b20e61afacbe46\r\n\r\nBuild config:\r\n(Paste the content of config.mk, or the build command.)\r\nmake -j $(nproc) USE_OPENCV=1 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1\r\n\r\n## Error Message:\r\n(Paste the complete error message, including stack trace.)\r\n```\r\nTraceback (most recent call last):\r\n  File \"/home/ubuntu/openspeech/openspeech/experiment.py\", line 358, in <module>\r\n    exp.run()\r\n  File \"/home/ubuntu/openspeech/openspeech/experiment.py\", line 352, in run\r\n    stage_inst.run()\r\n  File \"/home/ubuntu/openspeech/openspeech/train.py\", line 140, in run\r\n    for step, data in enumerate(train_loader):\r\n  File \"/home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet-1.2.0-py3.6.egg/mxnet/gluon/data/dataloader.py\", line 224, in __iter__\r\n    idx, batch = data_queue.get()\r\n  File \"/home/ubuntu/anaconda3/lib/python3.6/multiprocessing/queues.py\", line 113, in get\r\n    return _ForkingPickler.loads(res)\r\n  File \"/home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet-1.2.0-py3.6.egg/mxnet/gluon/data/dataloader.py\", line 38, in rebuild_ndarray\r\n    return nd.NDArray(nd.ndarray._new_from_shared_mem(*args))\r\n  File \"/home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet-1.2.0-py3.6.egg/mxnet/ndarray/ndarray.py\", line 151, in _new_from_shared_mem\r\n    ctypes.byref(hdl)))\r\n  File \"/home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet-1.2.0-py3.6.egg/mxnet/base.py\", line 149, in check_call\r\n    raise MXNetError(py_str(_LIB.MXGetLastError()))\r\nmxnet.base.MXNetError: [20:48:23] src/storage/./cpu_shared_storage_manager.h:178: Check failed: ptr != ((void *) -1) (0xffffffffffffffff vs. 0xffffffffffffffff) Failed to map shared memory. mmap failed with error Cannot allocate memory\r\n\r\nStack trace returned 10 entries:\r\n[bt] (0) /home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet-1.2.0-py3.6.egg/mxnet/libmxnet.so(dmlc::StackTrace[abi:cxx11]()+0x5b) [0x7f77c7b896cb]\r\n[bt] (1) /home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet-1.2.0-py3.6.egg/mxnet/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x28) [0x7f77c7b8a208]\r\n[bt] (2) /home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet-1.2.0-py3.6.egg/mxnet/libmxnet.so(mxnet::storage::CPUSharedStorageManager::Alloc(mxnet::Storage::Handle*)+0x99b) [0x7f77ca7ef5db]\r\n[bt] (3) /home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet-1.2.0-py3.6.egg/mxnet/libmxnet.so(mxnet::StorageImpl::Alloc(mxnet::Storage::Handle*)+0x5d) [0x7f77ca7efd4d]\r\n[bt] (4) /home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet-1.2.0-py3.6.egg/mxnet/libmxnet.so(MXNDArrayCreateFromSharedMem+0x825) [0x7f77ca848135]\r\n[bt] (5) /home/ubuntu/anaconda3/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call_unix64+0x4c) [0x7f7877dbaec0]\r\n[bt] (6) /home/ubuntu/anaconda3/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call+0x22d) [0x7f7877dba87d]\r\n[bt] (7) /home/ubuntu/anaconda3/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(_ctypes_callproc+0x2ce) [0x7f7877fcf82e]\r\n[bt] (8) /home/ubuntu/anaconda3/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(+0x12265) [0x7f7877fd0265]\r\n[bt] (9) python(_PyObject_FastCallDict+0x8b) [0x55a780a2554b]\r\n```\r\n\r\n## Minimum reproducible example\r\n(If you are using your own code, please provide a short script that reproduces the error. Otherwise, please provide link to the existing example.)\r\nThe code is in a private repo\r\n\r\n## Steps to reproduce\r\n(Paste the commands you ran that produced the error.)\r\n\r\nIt happens intermittently when we train on large speech dataset.\r\n\r\n## What have you tried to solve it?\r\n\r\nNot yet.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/incubator-mxnet/issues/comments/377629440","html_url":"https://github.com/apache/incubator-mxnet/issues/10042#issuecomment-377629440","issue_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/10042","id":377629440,"user":{"login":"Jerryzcn","id":4907789,"avatar_url":"https://avatars0.githubusercontent.com/u/4907789?v=4","gravatar_id":"","url":"https://api.github.com/users/Jerryzcn","html_url":"https://github.com/Jerryzcn","followers_url":"https://api.github.com/users/Jerryzcn/followers","following_url":"https://api.github.com/users/Jerryzcn/following{/other_user}","gists_url":"https://api.github.com/users/Jerryzcn/gists{/gist_id}","starred_url":"https://api.github.com/users/Jerryzcn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Jerryzcn/subscriptions","organizations_url":"https://api.github.com/users/Jerryzcn/orgs","repos_url":"https://api.github.com/users/Jerryzcn/repos","events_url":"https://api.github.com/users/Jerryzcn/events{/privacy}","received_events_url":"https://api.github.com/users/Jerryzcn/received_events","type":"User","site_admin":false},"created_at":"2018-03-30T21:39:44Z","updated_at":"2018-03-30T21:39:44Z","author_association":"CONTRIBUTOR","body":"Yeah, we can close this."}}
{"actor":{"display_login":"Jerryzcn"}}