{"action":"created","issue":{"url":"https://api.github.com/repos/apache/incubator-mxnet/issues/12203","repository_url":"https://api.github.com/repos/apache/incubator-mxnet","labels_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/12203/labels{/name}","comments_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/12203/comments","events_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/12203/events","html_url":"https://github.com/apache/incubator-mxnet/issues/12203","id":351249675,"node_id":"MDU6SXNzdWUzNTEyNDk2NzU=","number":12203,"title":"flaky test: test_operator_gpu.test_depthwise_convolution","user":{"login":"lebeg","id":1753787,"node_id":"MDQ6VXNlcjE3NTM3ODc=","avatar_url":"https://avatars1.githubusercontent.com/u/1753787?v=4","gravatar_id":"","url":"https://api.github.com/users/lebeg","html_url":"https://github.com/lebeg","followers_url":"https://api.github.com/users/lebeg/followers","following_url":"https://api.github.com/users/lebeg/following{/other_user}","gists_url":"https://api.github.com/users/lebeg/gists{/gist_id}","starred_url":"https://api.github.com/users/lebeg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lebeg/subscriptions","organizations_url":"https://api.github.com/users/lebeg/orgs","repos_url":"https://api.github.com/users/lebeg/repos","events_url":"https://api.github.com/users/lebeg/events{/privacy}","received_events_url":"https://api.github.com/users/lebeg/received_events","type":"User","site_admin":false},"labels":[{"id":977504354,"node_id":"MDU6TGFiZWw5Nzc1MDQzNTQ=","url":"https://api.github.com/repos/apache/incubator-mxnet/labels/Disabled%20test","name":"Disabled test","color":"fc2929","default":false},{"id":721783588,"node_id":"MDU6TGFiZWw3MjE3ODM1ODg=","url":"https://api.github.com/repos/apache/incubator-mxnet/labels/Flaky","name":"Flaky","color":"fc2929","default":false},{"id":721783490,"node_id":"MDU6TGFiZWw3MjE3ODM0OTA=","url":"https://api.github.com/repos/apache/incubator-mxnet/labels/Test","name":"Test","color":"fbca04","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2018-08-16T15:13:35Z","updated_at":"2018-12-11T21:22:55Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-12181/4/pipeline\r\n\r\n```\r\n======================================================================\r\n\r\nFAIL: test_operator_gpu.test_depthwise_convolution\r\n\r\n----------------------------------------------------------------------\r\n\r\nTraceback (most recent call last):\r\n\r\n  File \"/usr/local/lib/python3.5/dist-packages/nose/case.py\", line 198, in runTest\r\n\r\n    self.test(*self.arg)\r\n\r\n  File \"/usr/local/lib/python3.5/dist-packages/nose/util.py\", line 620, in newfunc\r\n\r\n    return func(*arg, **kw)\r\n\r\n  File \"/work/mxnet/tests/python/gpu/../unittest/common.py\", line 172, in test_new\r\n\r\n    orig_test(*args, **kwargs)\r\n\r\n  File \"/work/mxnet/tests/python/gpu/../unittest/test_operator.py\", line 1663, in test_depthwise_convolution\r\n\r\n    np.testing.assert_allclose(arr1.asnumpy(), arr2.asnumpy(), rtol=1e-3, atol=1e-3)\r\n\r\n  File \"/usr/local/lib/python3.5/dist-packages/numpy/testing/nose_tools/utils.py\", line 1396, in assert_allclose\r\n\r\n    verbose=verbose, header=header, equal_nan=equal_nan)\r\n\r\n  File \"/usr/local/lib/python3.5/dist-packages/numpy/testing/nose_tools/utils.py\", line 779, in assert_array_compare\r\n\r\n    raise AssertionError(msg)\r\n\r\nAssertionError: \r\n\r\nNot equal to tolerance rtol=0.001, atol=0.001\r\n\r\n\r\n\r\n(mismatch 2.040816326530617%)\r\n\r\n x: array([[[[  7.347052,  -1.722254,   7.837829,   4.21605 ,  -1.359475,\r\n\r\n            1.55463 ,   6.701931],\r\n\r\n         [ 11.283103,  12.302897,  -9.111632,  -3.390831,  -4.708895,...\r\n\r\n y: array([[[[  7.348634,  -1.720118,   7.836634,   4.217753,  -1.361165,\r\n\r\n            1.552893,   6.695152],\r\n\r\n         [ 11.284711,  12.303747,  -9.112375,  -3.390379,  -4.709879,...\r\n\r\n-------------------- >> begin captured logging << --------------------\r\n\r\ncommon: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=640580015 to reproduce.\r\n\r\n--------------------- >> end captured logging << ---------------------\r\n\r\n```"},"comment":{"url":"https://api.github.com/repos/apache/incubator-mxnet/issues/comments/446366930","html_url":"https://github.com/apache/incubator-mxnet/issues/12203#issuecomment-446366930","issue_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/12203","id":446366930,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NjM2NjkzMA==","user":{"login":"mseth10","id":22492939,"node_id":"MDQ6VXNlcjIyNDkyOTM5","avatar_url":"https://avatars2.githubusercontent.com/u/22492939?v=4","gravatar_id":"","url":"https://api.github.com/users/mseth10","html_url":"https://github.com/mseth10","followers_url":"https://api.github.com/users/mseth10/followers","following_url":"https://api.github.com/users/mseth10/following{/other_user}","gists_url":"https://api.github.com/users/mseth10/gists{/gist_id}","starred_url":"https://api.github.com/users/mseth10/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mseth10/subscriptions","organizations_url":"https://api.github.com/users/mseth10/orgs","repos_url":"https://api.github.com/users/mseth10/repos","events_url":"https://api.github.com/users/mseth10/events{/privacy}","received_events_url":"https://api.github.com/users/mseth10/received_events","type":"User","site_admin":false},"created_at":"2018-12-11T21:22:55Z","updated_at":"2018-12-11T21:22:55Z","author_association":"CONTRIBUTOR","body":"Reproduction steps (from https://cwiki.apache.org/confluence/display/MXNET/Reproducing+test+results):\r\n\r\nSpin up a p3.8x large instance, with Ubuntu base DLAMI, with at least 150GB EBS storage\r\n\r\nCloning and building MXNet:\r\n- git clone --recursive https://github.com/apache/incubator-mxnet.git\r\n- cd incubator-mxnet\r\n- pip3 install -r ci/requirements.txt\r\n- ci/build.py --platform ubuntu_build_cuda /work/runtime_functions.sh build_ubuntu_gpu_mkldnn\r\n\r\nEnabling the test - Comment out line [1634](https://github.com/apache/incubator-mxnet/blob/master/tests/python/unittest/test_operator.py#L1634) in file tests/python/unittest/test_operator.py.\r\n`# @unittest.skip(\"Flaky test https://github.com/apache/incubator-mxnet/issues/12203\")`\r\n\r\nRunning only this particular test 10,000 times - Modify line [735](https://github.com/apache/incubator-mxnet/blob/master/ci/docker/runtime_functions.sh#L735) in file ci/docker/runtime_functions.sh to\r\n`MXNET_TEST_COUNT=10000 nosetests-2.7 $NOSE_COVERAGE_ARGUMENTS $NOSE_TIMER_ARGUMENTS --with-xunit --xunit-file nosetests_gpu.xml --verbose tests/python/gpu/test_operator_gpu.py:test_depthwise_convolution`\r\n\r\n- ci/build.py --nvidiadocker --platform ubuntu_gpu /work/runtime_functions.sh unittest_ubuntu_python2_gpu"}}
{"actor":{"display_login":"mseth10"}}