{"action":"created","issue":{"url":"https://api.github.com/repos/statsmodels/statsmodels/issues/2876","repository_url":"https://api.github.com/repos/statsmodels/statsmodels","labels_url":"https://api.github.com/repos/statsmodels/statsmodels/issues/2876/labels{/name}","comments_url":"https://api.github.com/repos/statsmodels/statsmodels/issues/2876/comments","events_url":"https://api.github.com/repos/statsmodels/statsmodels/issues/2876/events","html_url":"https://github.com/statsmodels/statsmodels/issues/2876","id":144351714,"number":2876,"title":"Test failure for Sandybridge OpenBLAS kernel - test precision?","user":{"login":"matthew-brett","id":67612,"avatar_url":"https://avatars.githubusercontent.com/u/67612?v=3","gravatar_id":"","url":"https://api.github.com/users/matthew-brett","html_url":"https://github.com/matthew-brett","followers_url":"https://api.github.com/users/matthew-brett/followers","following_url":"https://api.github.com/users/matthew-brett/following{/other_user}","gists_url":"https://api.github.com/users/matthew-brett/gists{/gist_id}","starred_url":"https://api.github.com/users/matthew-brett/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matthew-brett/subscriptions","organizations_url":"https://api.github.com/users/matthew-brett/orgs","repos_url":"https://api.github.com/users/matthew-brett/repos","events_url":"https://api.github.com/users/matthew-brett/events{/privacy}","received_events_url":"https://api.github.com/users/matthew-brett/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2016-03-29T18:44:18Z","updated_at":"2016-03-30T16:18:05Z","closed_at":null,"body":"We are in the middle of testing the scipy numerical libraries with OpenBLAS:\r\n\r\n* https://github.com/matthew-brett/manylinux-builds/issues/4\r\n* https://mail.scipy.org/pipermail/numpy-discussion/2016-March/075206.html\r\n\r\nWe have found a couple of test errors in current statsmodels, on just one of the OpenBLAS kernels.\r\n\r\nhttps://github.com/matthew-brett/manylinux-builds/issues/4#issuecomment-203038517\r\n\r\n```\r\n======================================================================\r\nFAIL: statsmodels.tsa.statespace.tests.test_dynamic_factor.TestDynamicFactor_general_errors.test_bse_approx\r\n----------------------------------------------------------------------\r\nTraceback (most recent call last):\r\n  File \"/home/travis/build/matthew-brett/manylinux-testing/venv/lib/python2.7/site-packages/nose/case.py\", line 197, in runTest\r\n    self.test(*self.arg)\r\n  File \"/home/travis/build/matthew-brett/manylinux-testing/venv/lib/python2.7/site-packages/statsmodels/tsa/statespace/tests/test_dynamic_factor.py\", line 376, in test_bse_approx\r\n    assert_allclose(bse[-10:], self.true['var_oim'][-10:], atol=1e-4)\r\n  File \"/home/travis/build/matthew-brett/manylinux-testing/venv/lib/python2.7/site-packages/numpy/testing/utils.py\", line 1391, in assert_allclose\r\n    verbose=verbose, header=header)\r\n  File \"/home/travis/build/matthew-brett/manylinux-testing/venv/lib/python2.7/site-packages/numpy/testing/utils.py\", line 733, in assert_array_compare\r\n    raise AssertionError(msg)\r\nAssertionError: \r\nNot equal to tolerance rtol=1e-07, atol=0.0001\r\n(mismatch 10.0%)\r\n x: array([  2.142085e-04,   1.307592e-02,   2.915582e-01,   4.320433e-01,\r\n         7.689779e-04,   1.742122e-02,   2.201651e-02,   5.543138e-04,\r\n         1.456134e-02,   1.707097e-02])\r\n y: array([  2.142100e-04,   1.307587e-02,   2.916752e-01,   4.320406e-01,\r\n         7.689900e-04,   1.742173e-02,   2.201610e-02,   5.543500e-04,\r\n         1.456365e-02,   1.707167e-02])\r\n======================================================================\r\nFAIL: statsmodels.tsa.statespace.tests.test_sarimax.TestFriedmanStateRegression.test_mle\r\n----------------------------------------------------------------------\r\nTraceback (most recent call last):\r\n  File \"/home/travis/build/matthew-brett/manylinux-testing/venv/lib/python2.7/site-packages/nose/case.py\", line 197, in runTest\r\n    self.test(*self.arg)\r\n  File \"/home/travis/build/matthew-brett/manylinux-testing/venv/lib/python2.7/site-packages/statsmodels/tsa/statespace/tests/test_sarimax.py\", line 746, in test_mle\r\n    atol=1e-1, rtol=1e-1\r\n  File \"/home/travis/build/matthew-brett/manylinux-testing/venv/lib/python2.7/site-packages/numpy/testing/utils.py\", line 1391, in assert_allclose\r\n    verbose=verbose, header=header)\r\n  File \"/home/travis/build/matthew-brett/manylinux-testing/venv/lib/python2.7/site-packages/numpy/testing/utils.py\", line 733, in assert_array_compare\r\n    raise AssertionError(msg)\r\nAssertionError: \r\nNot equal to tolerance rtol=0.1, atol=0.1\r\n(mismatch 33.3333333333%)\r\n x: array([   0.970836,    0.299662,  103.355682])\r\n y: array([  0.934849,   0.309059,  93.224973])\r\n----------------------------------------------------------------------\r\nRan 4548 tests in 2554.303s\r\nFAILED (SKIP=164, failures=2)\r\n```\r\n\r\nI can replicate on a recent Linux machine with (in a virtualenv):\r\n\r\n```\r\npython -m pip install -U pip\r\npip install -f https://d9a97980b71d47cde94d-aae005c4999d7244ac63632f8b80e089.ssl.cf2.rackcdn.com numpy scipy pandas nose cython\r\npip install https://github.com/statsmodels/statsmodels/archive/eed4ae5a90a1d1ac312c24bbebf2bff88fe87da9.zip\r\nOPENBLAS_VERBOSE=2 OPENBLAS_CORETYPE=sandybridge nosetests statsmodels.tsa.statespace.tests.test_dynamic_factor statsmodels.tsa.statespace.tests.test_sarimax\r\n```\r\n\r\nI noticed the very liberal precision you are using here.  Is it possible this difference is within reasonable floating point limits?"},"comment":{"url":"https://api.github.com/repos/statsmodels/statsmodels/issues/comments/203510232","html_url":"https://github.com/statsmodels/statsmodels/issues/2876#issuecomment-203510232","issue_url":"https://api.github.com/repos/statsmodels/statsmodels/issues/2876","id":203510232,"user":{"login":"matthew-brett","id":67612,"avatar_url":"https://avatars.githubusercontent.com/u/67612?v=3","gravatar_id":"","url":"https://api.github.com/users/matthew-brett","html_url":"https://github.com/matthew-brett","followers_url":"https://api.github.com/users/matthew-brett/followers","following_url":"https://api.github.com/users/matthew-brett/following{/other_user}","gists_url":"https://api.github.com/users/matthew-brett/gists{/gist_id}","starred_url":"https://api.github.com/users/matthew-brett/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matthew-brett/subscriptions","organizations_url":"https://api.github.com/users/matthew-brett/orgs","repos_url":"https://api.github.com/users/matthew-brett/repos","events_url":"https://api.github.com/users/matthew-brett/events{/privacy}","received_events_url":"https://api.github.com/users/matthew-brett/received_events","type":"User","site_admin":false},"created_at":"2016-03-30T16:18:04Z","updated_at":"2016-03-30T16:18:04Z","body":"@xianyi - thank you.   I don't understand this code well enough to follow where the precision errors are coming from.  @ChadFulton, who wrote the code, thinks it's reasonable to attribute this to numerical error.  So, I don't think we have any good evidence yet that this is really a bug in OpenBLAS rather than some acceptable precision difference in the calculation."}}