{"action":"created","issue":{"url":"https://api.github.com/repos/JohnLangford/vowpal_wabbit/issues/1411","repository_url":"https://api.github.com/repos/JohnLangford/vowpal_wabbit","labels_url":"https://api.github.com/repos/JohnLangford/vowpal_wabbit/issues/1411/labels{/name}","comments_url":"https://api.github.com/repos/JohnLangford/vowpal_wabbit/issues/1411/comments","events_url":"https://api.github.com/repos/JohnLangford/vowpal_wabbit/issues/1411/events","html_url":"https://github.com/JohnLangford/vowpal_wabbit/issues/1411","id":296503532,"number":1411,"title":"heisenbug reproduced in test 143 (needs many attempts)","user":{"login":"arielf","id":159664,"avatar_url":"https://avatars2.githubusercontent.com/u/159664?v=4","gravatar_id":"","url":"https://api.github.com/users/arielf","html_url":"https://github.com/arielf","followers_url":"https://api.github.com/users/arielf/followers","following_url":"https://api.github.com/users/arielf/following{/other_user}","gists_url":"https://api.github.com/users/arielf/gists{/gist_id}","starred_url":"https://api.github.com/users/arielf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arielf/subscriptions","organizations_url":"https://api.github.com/users/arielf/orgs","repos_url":"https://api.github.com/users/arielf/repos","events_url":"https://api.github.com/users/arielf/events{/privacy}","received_events_url":"https://api.github.com/users/arielf/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-02-12T19:54:06Z","updated_at":"2018-03-19T18:13:10Z","closed_at":"2018-03-19T18:13:10Z","author_association":"COLLABORATOR","body":"The old heisen-bug seems to be triggered more often after recent changes.\r\n\r\nThe heisen-bug may be a race condition between the parsing and learning threads. It manifests by a wrong (one-off) number of passes when running the test-suite, e.g:\r\n\r\n    finished run\r\n    -passes used = 8\r\n    ...\r\n    +passes used = 7\r\n    ...\r\n\r\nTrying to reproduce, it looks like test 143 seems like a relatively-good case to try and reproduce.\r\n\r\nYou may use the attached script to reproduce (it attempts to hit the bug by running a single test multiple times.  Most tests never hit the issue, but in my env, test 143 now hits the issue with high probability,  within ~50 attempts when trying 500 times):\r\n\r\n```\r\n#!/bin/bash\r\n\r\nOutLog=/tmp/vw.heisen.bug\r\nMaxCount=500\r\nTimeout=10s\r\n\r\ntest_one() {\r\n    TestNo=$1\r\n\r\n    echo \"=== Trying Test $TestNo ...\" > /dev/tty\r\n\r\n    rm -f RunTests.last.times\r\n\r\n    counter=1\r\n    while [ $counter -lt $MaxCount ]; do\r\n\r\n        timeout $Timeout ./RunTests -d -fe -E 0.001 $TestNo ../vowpalwabbit/vw ../vowpalwabbit/vw >$OutLog 2>&1\r\n\r\n        case $? in\r\n            0)  : -- no heisenbug, keep trying\r\n                printf '\\r%d' $counter > /dev/tty\r\n                ;;\r\n            124) printf '\\r' > /dev/tty\r\n                echo \"=== timed-out after Timeout=$Timeout (no heisenbug)\"\r\n                ;;\r\n            *)  printf '\\r' > /dev/tty\r\n                echo \"=== +++ Test $TestNo: heisen-bug HIT ($counter attempts)\"\r\n                # Need to auto-find the relevant stderr\r\n                diff_files=`grep FAILED $OutLog | perl -ne '\r\n                    /\\(([^)]+)\\)[^(]+\\(([^)]+)/ && print \"$1 $2\\n\"'`\r\n                diff -u $diff_files\r\n                exit 1\r\n                ;;\r\n        esac\r\n        counter=$(($counter + 1))\r\n    done\r\n    printf '\\r' > /dev/tty\r\n    echo \"=== --- Test $TestNo: no heisen-bug ($counter attempts)\"\r\n}\r\n\r\n#\r\n# --- main\r\n#\r\ncase \"$@\" in\r\n    [1-9]*) TestNumbers=\"$@\" ;;\r\n    *) TestNumbers=143 ;;\r\nesac\r\n\r\ncase `pwd` in\r\n    */test) : ;;\r\n    *) cd test ;;\r\nesac\r\n\r\nfor tno in $TestNumbers; do\r\n    test_one $tno\r\ndone\r\n\r\nexit 0\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/JohnLangford/vowpal_wabbit/issues/comments/374312537","html_url":"https://github.com/JohnLangford/vowpal_wabbit/issues/1411#issuecomment-374312537","issue_url":"https://api.github.com/repos/JohnLangford/vowpal_wabbit/issues/1411","id":374312537,"user":{"login":"JohnLangford","id":79590,"avatar_url":"https://avatars3.githubusercontent.com/u/79590?v=4","gravatar_id":"","url":"https://api.github.com/users/JohnLangford","html_url":"https://github.com/JohnLangford","followers_url":"https://api.github.com/users/JohnLangford/followers","following_url":"https://api.github.com/users/JohnLangford/following{/other_user}","gists_url":"https://api.github.com/users/JohnLangford/gists{/gist_id}","starred_url":"https://api.github.com/users/JohnLangford/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JohnLangford/subscriptions","organizations_url":"https://api.github.com/users/JohnLangford/orgs","repos_url":"https://api.github.com/users/JohnLangford/repos","events_url":"https://api.github.com/users/JohnLangford/events{/privacy}","received_events_url":"https://api.github.com/users/JohnLangford/received_events","type":"User","site_admin":false},"created_at":"2018-03-19T18:13:10Z","updated_at":"2018-03-19T18:13:10Z","author_association":"OWNER","body":"Should be fixed now! :-)"}}
{"actor":{"display_login":"JohnLangford"}}