{"action":"opened","issue":{"url":"https://api.github.com/repos/TheTorProject/ooni-probe/issues/493","repository_url":"https://api.github.com/repos/TheTorProject/ooni-probe","labels_url":"https://api.github.com/repos/TheTorProject/ooni-probe/issues/493/labels{/name}","comments_url":"https://api.github.com/repos/TheTorProject/ooni-probe/issues/493/comments","events_url":"https://api.github.com/repos/TheTorProject/ooni-probe/issues/493/events","html_url":"https://github.com/TheTorProject/ooni-probe/issues/493","id":152814713,"number":493,"title":"The localOptions are overwritten when the same test is run with different options in a deck","user":{"login":"hellais","id":424620,"avatar_url":"https://avatars.githubusercontent.com/u/424620?v=3","gravatar_id":"","url":"https://api.github.com/users/hellais","html_url":"https://github.com/hellais","followers_url":"https://api.github.com/users/hellais/followers","following_url":"https://api.github.com/users/hellais/following{/other_user}","gists_url":"https://api.github.com/users/hellais/gists{/gist_id}","starred_url":"https://api.github.com/users/hellais/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hellais/subscriptions","organizations_url":"https://api.github.com/users/hellais/orgs","repos_url":"https://api.github.com/users/hellais/repos","events_url":"https://api.github.com/users/hellais/events{/privacy}","received_events_url":"https://api.github.com/users/hellais/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/TheTorProject/ooni-probe/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/TheTorProject/ooni-probe/labels/ooni-probe","name":"ooni-probe","color":"49A2DD"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-05-03T16:16:15Z","updated_at":"2016-05-03T16:16:15Z","closed_at":null,"body":"The current mechanism for setting the localOptions of tests was written when there was no notion of test decks.\r\n\r\nAs a result of this the object lifecycle was such that it was OK to have the `localOptions` attribute of `NetTestCase` be static. Since the introduction of test decks it's possible to have multiple tests of the same type run with different options, though since `localOptions` is a static attribute it will be overwritten with the last parsed option in the deck by subsequent calls to `NetTestLoader.checkOptions`.\r\n\r\nHere is a minimal way of reproducing this bug:\r\n\r\n```\r\n#!/bin/bash\r\ncat <<EOF > broken.deck\r\n- options:\r\n    collector: null\r\n    help: 0\r\n    logfile: null\r\n    no-default-reporter: 0\r\n    parallelism: null\r\n    pcapfile: null\r\n    reportfile: null\r\n    resume: 0\r\n    subargs: [-f, 'list1.txt']\r\n    test_file: blocking/tcp_connect\r\n- options:\r\n    collector: null\r\n    help: 0\r\n    logfile: null\r\n    no-default-reporter: 0\r\n    parallelism: null\r\n    pcapfile: null\r\n    reportfile: null\r\n    resume: 0\r\n    subargs: [-f, 'list2.txt']\r\n    test_file: blocking/tcp_connect\r\nEOF\r\necho \"8.8.8.8:53\" > list1.txt\r\necho \"8.8.4.4:53\" > list2.txt\r\n```\r\nThen run it with `ooniprobe -n -i broken.deck`.\r\n\r\nBy inspecting the tcp_connect reports you can then confirm that both of them include only the `8.8.4.4`  IP address and the first one doesn't contain the `8.8.8.8`.\r\n\r\nThe current recommended workaround is to not use conflicting options in decks when using the same test name.\r\n\r\nFixing this bug should be of very high priority as I believe it also impacts the ability to properly run the default decks generated with `oonideckgen`."}}