{"action":"closed","issue":{"url":"https://api.github.com/repos/Microsoft/CNTK/issues/535","repository_url":"https://api.github.com/repos/Microsoft/CNTK","labels_url":"https://api.github.com/repos/Microsoft/CNTK/issues/535/labels{/name}","comments_url":"https://api.github.com/repos/Microsoft/CNTK/issues/535/comments","events_url":"https://api.github.com/repos/Microsoft/CNTK/issues/535/events","html_url":"https://github.com/Microsoft/CNTK/issues/535","id":157366877,"number":535,"title":"Memory Corruption in ImageReader","user":{"login":"nslay","id":6295986,"avatar_url":"https://avatars.githubusercontent.com/u/6295986?v=3","gravatar_id":"","url":"https://api.github.com/users/nslay","html_url":"https://github.com/nslay","followers_url":"https://api.github.com/users/nslay/followers","following_url":"https://api.github.com/users/nslay/following{/other_user}","gists_url":"https://api.github.com/users/nslay/gists{/gist_id}","starred_url":"https://api.github.com/users/nslay/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nslay/subscriptions","organizations_url":"https://api.github.com/users/nslay/orgs","repos_url":"https://api.github.com/users/nslay/repos","events_url":"https://api.github.com/users/nslay/events{/privacy}","received_events_url":"https://api.github.com/users/nslay/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2016-05-29T03:11:37Z","updated_at":"2016-06-07T12:48:23Z","closed_at":"2016-06-07T12:48:23Z","body":"I've been playing with CNTK using AlexNet modified to use a zip file database of png files. I've spent the entire day looking at a crash on Linux. I can't tell you specifically _where_ in the code the crash happens, but I can tell you that it happens in ImageTransformer when CNTK is creating mini batches (anything to do with resize). It seems to happen in OpenMP parallel sections. Setting `OMP_NUM_THREADS=1` will get it past the minibatch setup phase and start training. No backtrace is ever the same but always involves somewhere in the ImageTransformer portion of ImageReader.\r\n\r\nSome things I looked at:\r\n\r\n1. This was initially raising asserts in OpenCV `cv::resize` where `area()` was somehow 0 or negative. I tried using newer versions of libpng since the above problem can be caused by empty images. I thought maybe somehow images read from the zip file were erroneously returned as empty. Then I tried to enable `-DBUILD_PNG=\"ON\"` in OpenCV to prevent any kind of problem with the numerous versions of libpng my system has. At some point here, item (1) stopped happening. I also tried OpenCV 3.1.0 which has the same exact problem.\r\n2. `opal_memory_ptmalloc2_malloc` popped up a lot at the top of backtraces. I recompiled OpenMPI using `--without-memory-manager` since a possibly non-thread-safe memory allocator could cause any number of related problems. That restored GNU `malloc` as the global memory allocator for operator `new`. GNU `malloc` now prints out two obvious \"memory corruption\" messages on the console when running CNTK. Worth noting, that the OpenMPI `opal_memory_ptmalloc2_malloc` source code comments insists that it is thread-safe.\r\n3. I ran CNTK through valgrind but it didn't catch the source of the memory corruption.\r\n\r\nThe same setup works on Windows.\r\n\r\nI'm using all the same versions of software mentioned in the CNTK wiki with the exception of GCC which is 4.9.1. "}}
{"actor":{"display_login":"eldakms"}}