{"action":"created","comment":{"url":"https://api.github.com/repos/stb-tester/stb-tester/pulls/comments/58027607","id":58027607,"diff_hunk":"@@ -77,7 +80,6 @@ def main(argv):\n     subparsers.add_parser('validate', help='Run (and check) the auto-selftests')\n \n     cmdline_args = parser.parse_args(argv[1:])\n-    screenshots_root = cmdline_args.screenshots_root or \"selftest/screenshots\"","path":"stbt_auto_selftest.py","position":null,"original_position":16,"commit_id":"8d7d4efd4cbe6ce7794f06429895553e2cd80512","original_commit_id":"e48079ef0d06b14c5523f1ddf50d1222423b7d6c","user":{"login":"wmanley","id":494659,"avatar_url":"https://avatars.githubusercontent.com/u/494659?v=3","gravatar_id":"","url":"https://api.github.com/users/wmanley","html_url":"https://github.com/wmanley","followers_url":"https://api.github.com/users/wmanley/followers","following_url":"https://api.github.com/users/wmanley/following{/other_user}","gists_url":"https://api.github.com/users/wmanley/gists{/gist_id}","starred_url":"https://api.github.com/users/wmanley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wmanley/subscriptions","organizations_url":"https://api.github.com/users/wmanley/orgs","repos_url":"https://api.github.com/users/wmanley/repos","events_url":"https://api.github.com/users/wmanley/events{/privacy}","received_events_url":"https://api.github.com/users/wmanley/received_events","type":"User","site_admin":false},"body":"> So, to be clear, my commit is a refactoring that doesn't change the existing behaviour, but now you think the behaviour should be different?\r\n\r\nYes, that's right.  Originally I'd added `--screenshots-root` for testing but didn't end up using it.  I'll remove it instead.","created_at":"2016-03-31T09:51:57Z","updated_at":"2016-03-31T09:51:57Z","html_url":"https://github.com/stb-tester/stb-tester/pull/343#discussion_r58027607","pull_request_url":"https://api.github.com/repos/stb-tester/stb-tester/pulls/343","_links":{"self":{"href":"https://api.github.com/repos/stb-tester/stb-tester/pulls/comments/58027607"},"html":{"href":"https://github.com/stb-tester/stb-tester/pull/343#discussion_r58027607"},"pull_request":{"href":"https://api.github.com/repos/stb-tester/stb-tester/pulls/343"}}},"pull_request":{"url":"https://api.github.com/repos/stb-tester/stb-tester/pulls/343","id":62165597,"html_url":"https://github.com/stb-tester/stb-tester/pull/343","diff_url":"https://github.com/stb-tester/stb-tester/pull/343.diff","patch_url":"https://github.com/stb-tester/stb-tester/pull/343.patch","issue_url":"https://api.github.com/repos/stb-tester/stb-tester/issues/343","number":343,"state":"open","locked":false,"title":"Add tool stbt auto-selftests","user":{"login":"wmanley","id":494659,"avatar_url":"https://avatars.githubusercontent.com/u/494659?v=3","gravatar_id":"","url":"https://api.github.com/users/wmanley","html_url":"https://github.com/wmanley","followers_url":"https://api.github.com/users/wmanley/followers","following_url":"https://api.github.com/users/wmanley/following{/other_user}","gists_url":"https://api.github.com/users/wmanley/gists{/gist_id}","starred_url":"https://api.github.com/users/wmanley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wmanley/subscriptions","organizations_url":"https://api.github.com/users/wmanley/orgs","repos_url":"https://api.github.com/users/wmanley/repos","events_url":"https://api.github.com/users/wmanley/events{/privacy}","received_events_url":"https://api.github.com/users/wmanley/received_events","type":"User","site_admin":false},"body":"An implementation of the technique described in [Improve black-box testing agility: automatic self-regression tests](https://stb-tester.com/blog/2015/09/24/automatic-self-regression-tests).\r\n\r\nYou put screenshots of your UI into selftest/screenshots in your test pack and run\r\n\r\n    stbt auto-selftests generate\r\n\r\nand it will produce a bunch of python doctests in selftest/auto-selftests describing how each of your `FrameObject`s interpret each of these screenshots.\r\n\r\nThen make some changes to your FrameObjects and rerun\r\n\r\n    stbt auto-selftests generate\r\n\r\nand look at the diff to see how they've changed.\r\n\r\nBy default it will try every `FrameObject` against every screenshot printing a result to the doctests only if the resulting `FrameObject` is `True`.  Otherwise the result is omitted from the doctests.  This behaviour can be more finely controlled by setting `AUTO_SELFTEST_EXPRESSIONS`, `AUTO_SELFTEST_SCREENSHOTS` or `AUTO_SELFTEST_EXCLUDES` members of your `FrameObject`s.\r\n\r\n**TODO:**\r\n\r\n- [x] Merge #332 and rebase\r\n- [x] Update `Makefile` and `stbt.sh.in` to install it\r\n- [x] Add more tests\r\n- [x] Test operating on a whole test-pack\r\n- [x] Don't rely on test_frame_object.py for testing `stbt auto-selftests`, create some specific tests instead and leave it unchanged.\r\n- [x] Add `stbt auto-selftests validate` tool\r\n- [x] Reference documentation\r\n- [x] Remove `AUTO_SELFTEST_EXCLUDES` in favour of a way of specifying which images it should opportunistically test against. This would default to `'*.png'`\r\n- [x] Test `stbt auto-selftest validate` failing to validate\r\n- [x] Add release note\r\n- [x] Consider functions that throw exceptions \"Truthy\"\r\n- [x] Handle unicode & utf8-byte-strings\r\n- [x] Put it behind `stbt --with-experimental` flag?\r\n- [x] Bash completion\r\n- [x] Fix failing unicode in `AUTO_SELFTEST_EXPRESSIONS`\r\n- [x] Change exit status of `stbt selftest validate` when there are no `FrameObject`s or screenshots\r\n- [x] Port (private) auto-regressiontests to this system and test.\r\n- [x] Document `AUTO_*_*`\r\n- [x] Don't leave behind a bunch of `_bare.pyc` files\r\n- [x] Remove empty directories\r\n- [x] Don't generate tests for classes that have just been `import`ed from other files\r\n- [ ] `--screenshots-root`, if given, should be relative to current directory, not test-pack root.\r\n\r\n**For a future PR:**\r\n\r\n- Caching of expensive stbt operations to make it feasible to run with very many images\r\n- Better story for calling other properties from `is_visible`. Currently it goes into an infinite loop. Options:\r\n    - Automatically disable the \"return None if not visible\" behaviour for properties if we detect that `is_visible` (for the *same object*) is on the call stack.\r\n    - Detect and raise a more useful exception early.\r\n    - Don't detect it at runtime, write a `stbt lint` checker instead.\r\n- Add `stbt auto-selftests retro-regenerate` tool? - very useful, but maybe too advanced and too dangerous?\r\n- Some sort of pretty-printing\r\n- More general documentation covering best-practices and how it all fits together\r\n- A `stbt check` command to run all the checks that we recognise.","created_at":"2016-03-09T01:19:12Z","updated_at":"2016-03-31T09:51:57Z","closed_at":null,"merged_at":null,"merge_commit_sha":"d2556605fed5985b12f2ba8c9a74afefbbae0915","assignee":{"login":"drothlis","id":12424,"avatar_url":"https://avatars.githubusercontent.com/u/12424?v=3","gravatar_id":"","url":"https://api.github.com/users/drothlis","html_url":"https://github.com/drothlis","followers_url":"https://api.github.com/users/drothlis/followers","following_url":"https://api.github.com/users/drothlis/following{/other_user}","gists_url":"https://api.github.com/users/drothlis/gists{/gist_id}","starred_url":"https://api.github.com/users/drothlis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drothlis/subscriptions","organizations_url":"https://api.github.com/users/drothlis/orgs","repos_url":"https://api.github.com/users/drothlis/repos","events_url":"https://api.github.com/users/drothlis/events{/privacy}","received_events_url":"https://api.github.com/users/drothlis/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/stb-tester/stb-tester/pulls/343/commits","review_comments_url":"https://api.github.com/repos/stb-tester/stb-tester/pulls/343/comments","review_comment_url":"https://api.github.com/repos/stb-tester/stb-tester/pulls/comments{/number}","comments_url":"https://api.github.com/repos/stb-tester/stb-tester/issues/343/comments","statuses_url":"https://api.github.com/repos/stb-tester/stb-tester/statuses/8d7d4efd4cbe6ce7794f06429895553e2cd80512","head":{"label":"wmanley:auto-regression-tests","ref":"auto-regression-tests","sha":"8d7d4efd4cbe6ce7794f06429895553e2cd80512","user":{"login":"wmanley","id":494659,"avatar_url":"https://avatars.githubusercontent.com/u/494659?v=3","gravatar_id":"","url":"https://api.github.com/users/wmanley","html_url":"https://github.com/wmanley","followers_url":"https://api.github.com/users/wmanley/followers","following_url":"https://api.github.com/users/wmanley/following{/other_user}","gists_url":"https://api.github.com/users/wmanley/gists{/gist_id}","starred_url":"https://api.github.com/users/wmanley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wmanley/subscriptions","organizations_url":"https://api.github.com/users/wmanley/orgs","repos_url":"https://api.github.com/users/wmanley/repos","events_url":"https://api.github.com/users/wmanley/events{/privacy}","received_events_url":"https://api.github.com/users/wmanley/received_events","type":"User","site_admin":false},"repo":{"id":4783326,"name":"stb-tester","full_name":"wmanley/stb-tester","owner":{"login":"wmanley","id":494659,"avatar_url":"https://avatars.githubusercontent.com/u/494659?v=3","gravatar_id":"","url":"https://api.github.com/users/wmanley","html_url":"https://github.com/wmanley","followers_url":"https://api.github.com/users/wmanley/followers","following_url":"https://api.github.com/users/wmanley/following{/other_user}","gists_url":"https://api.github.com/users/wmanley/gists{/gist_id}","starred_url":"https://api.github.com/users/wmanley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wmanley/subscriptions","organizations_url":"https://api.github.com/users/wmanley/orgs","repos_url":"https://api.github.com/users/wmanley/repos","events_url":"https://api.github.com/users/wmanley/events{/privacy}","received_events_url":"https://api.github.com/users/wmanley/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/wmanley/stb-tester","description":"A record + playback testing system for set-top boxes","fork":true,"url":"https://api.github.com/repos/wmanley/stb-tester","forks_url":"https://api.github.com/repos/wmanley/stb-tester/forks","keys_url":"https://api.github.com/repos/wmanley/stb-tester/keys{/key_id}","collaborators_url":"https://api.github.com/repos/wmanley/stb-tester/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/wmanley/stb-tester/teams","hooks_url":"https://api.github.com/repos/wmanley/stb-tester/hooks","issue_events_url":"https://api.github.com/repos/wmanley/stb-tester/issues/events{/number}","events_url":"https://api.github.com/repos/wmanley/stb-tester/events","assignees_url":"https://api.github.com/repos/wmanley/stb-tester/assignees{/user}","branches_url":"https://api.github.com/repos/wmanley/stb-tester/branches{/branch}","tags_url":"https://api.github.com/repos/wmanley/stb-tester/tags","blobs_url":"https://api.github.com/repos/wmanley/stb-tester/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/wmanley/stb-tester/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/wmanley/stb-tester/git/refs{/sha}","trees_url":"https://api.github.com/repos/wmanley/stb-tester/git/trees{/sha}","statuses_url":"https://api.github.com/repos/wmanley/stb-tester/statuses/{sha}","languages_url":"https://api.github.com/repos/wmanley/stb-tester/languages","stargazers_url":"https://api.github.com/repos/wmanley/stb-tester/stargazers","contributors_url":"https://api.github.com/repos/wmanley/stb-tester/contributors","subscribers_url":"https://api.github.com/repos/wmanley/stb-tester/subscribers","subscription_url":"https://api.github.com/repos/wmanley/stb-tester/subscription","commits_url":"https://api.github.com/repos/wmanley/stb-tester/commits{/sha}","git_commits_url":"https://api.github.com/repos/wmanley/stb-tester/git/commits{/sha}","comments_url":"https://api.github.com/repos/wmanley/stb-tester/comments{/number}","issue_comment_url":"https://api.github.com/repos/wmanley/stb-tester/issues/comments{/number}","contents_url":"https://api.github.com/repos/wmanley/stb-tester/contents/{+path}","compare_url":"https://api.github.com/repos/wmanley/stb-tester/compare/{base}...{head}","merges_url":"https://api.github.com/repos/wmanley/stb-tester/merges","archive_url":"https://api.github.com/repos/wmanley/stb-tester/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/wmanley/stb-tester/downloads","issues_url":"https://api.github.com/repos/wmanley/stb-tester/issues{/number}","pulls_url":"https://api.github.com/repos/wmanley/stb-tester/pulls{/number}","milestones_url":"https://api.github.com/repos/wmanley/stb-tester/milestones{/number}","notifications_url":"https://api.github.com/repos/wmanley/stb-tester/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/wmanley/stb-tester/labels{/name}","releases_url":"https://api.github.com/repos/wmanley/stb-tester/releases{/id}","deployments_url":"https://api.github.com/repos/wmanley/stb-tester/deployments","created_at":"2012-06-25T15:17:46Z","updated_at":"2015-02-24T16:36:05Z","pushed_at":"2016-03-29T11:15:47Z","git_url":"git://github.com/wmanley/stb-tester.git","ssh_url":"git@github.com:wmanley/stb-tester.git","clone_url":"https://github.com/wmanley/stb-tester.git","svn_url":"https://github.com/wmanley/stb-tester","homepage":null,"size":23305,"stargazers_count":2,"watchers_count":2,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":2,"default_branch":"master"}},"base":{"label":"stb-tester:master","ref":"master","sha":"5da44ccc19bd66cec14045962d30086ae7085a70","user":{"login":"stb-tester","id":5528358,"avatar_url":"https://avatars.githubusercontent.com/u/5528358?v=3","gravatar_id":"","url":"https://api.github.com/users/stb-tester","html_url":"https://github.com/stb-tester","followers_url":"https://api.github.com/users/stb-tester/followers","following_url":"https://api.github.com/users/stb-tester/following{/other_user}","gists_url":"https://api.github.com/users/stb-tester/gists{/gist_id}","starred_url":"https://api.github.com/users/stb-tester/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stb-tester/subscriptions","organizations_url":"https://api.github.com/users/stb-tester/orgs","repos_url":"https://api.github.com/users/stb-tester/repos","events_url":"https://api.github.com/users/stb-tester/events{/privacy}","received_events_url":"https://api.github.com/users/stb-tester/received_events","type":"Organization","site_admin":false},"repo":{"id":4739298,"name":"stb-tester","full_name":"stb-tester/stb-tester","owner":{"login":"stb-tester","id":5528358,"avatar_url":"https://avatars.githubusercontent.com/u/5528358?v=3","gravatar_id":"","url":"https://api.github.com/users/stb-tester","html_url":"https://github.com/stb-tester","followers_url":"https://api.github.com/users/stb-tester/followers","following_url":"https://api.github.com/users/stb-tester/following{/other_user}","gists_url":"https://api.github.com/users/stb-tester/gists{/gist_id}","starred_url":"https://api.github.com/users/stb-tester/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stb-tester/subscriptions","organizations_url":"https://api.github.com/users/stb-tester/orgs","repos_url":"https://api.github.com/users/stb-tester/repos","events_url":"https://api.github.com/users/stb-tester/events{/privacy}","received_events_url":"https://api.github.com/users/stb-tester/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/stb-tester/stb-tester","description":"Automated Testing for Set-Top Boxes and Smart TVs","fork":false,"url":"https://api.github.com/repos/stb-tester/stb-tester","forks_url":"https://api.github.com/repos/stb-tester/stb-tester/forks","keys_url":"https://api.github.com/repos/stb-tester/stb-tester/keys{/key_id}","collaborators_url":"https://api.github.com/repos/stb-tester/stb-tester/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/stb-tester/stb-tester/teams","hooks_url":"https://api.github.com/repos/stb-tester/stb-tester/hooks","issue_events_url":"https://api.github.com/repos/stb-tester/stb-tester/issues/events{/number}","events_url":"https://api.github.com/repos/stb-tester/stb-tester/events","assignees_url":"https://api.github.com/repos/stb-tester/stb-tester/assignees{/user}","branches_url":"https://api.github.com/repos/stb-tester/stb-tester/branches{/branch}","tags_url":"https://api.github.com/repos/stb-tester/stb-tester/tags","blobs_url":"https://api.github.com/repos/stb-tester/stb-tester/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/stb-tester/stb-tester/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/stb-tester/stb-tester/git/refs{/sha}","trees_url":"https://api.github.com/repos/stb-tester/stb-tester/git/trees{/sha}","statuses_url":"https://api.github.com/repos/stb-tester/stb-tester/statuses/{sha}","languages_url":"https://api.github.com/repos/stb-tester/stb-tester/languages","stargazers_url":"https://api.github.com/repos/stb-tester/stb-tester/stargazers","contributors_url":"https://api.github.com/repos/stb-tester/stb-tester/contributors","subscribers_url":"https://api.github.com/repos/stb-tester/stb-tester/subscribers","subscription_url":"https://api.github.com/repos/stb-tester/stb-tester/subscription","commits_url":"https://api.github.com/repos/stb-tester/stb-tester/commits{/sha}","git_commits_url":"https://api.github.com/repos/stb-tester/stb-tester/git/commits{/sha}","comments_url":"https://api.github.com/repos/stb-tester/stb-tester/comments{/number}","issue_comment_url":"https://api.github.com/repos/stb-tester/stb-tester/issues/comments{/number}","contents_url":"https://api.github.com/repos/stb-tester/stb-tester/contents/{+path}","compare_url":"https://api.github.com/repos/stb-tester/stb-tester/compare/{base}...{head}","merges_url":"https://api.github.com/repos/stb-tester/stb-tester/merges","archive_url":"https://api.github.com/repos/stb-tester/stb-tester/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/stb-tester/stb-tester/downloads","issues_url":"https://api.github.com/repos/stb-tester/stb-tester/issues{/number}","pulls_url":"https://api.github.com/repos/stb-tester/stb-tester/pulls{/number}","milestones_url":"https://api.github.com/repos/stb-tester/stb-tester/milestones{/number}","notifications_url":"https://api.github.com/repos/stb-tester/stb-tester/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/stb-tester/stb-tester/labels{/name}","releases_url":"https://api.github.com/repos/stb-tester/stb-tester/releases{/id}","deployments_url":"https://api.github.com/repos/stb-tester/stb-tester/deployments","created_at":"2012-06-21T13:11:28Z","updated_at":"2016-03-24T18:38:31Z","pushed_at":"2016-03-29T12:16:12Z","git_url":"git://github.com/stb-tester/stb-tester.git","ssh_url":"git@github.com:stb-tester/stb-tester.git","clone_url":"https://github.com/stb-tester/stb-tester.git","svn_url":"https://github.com/stb-tester/stb-tester","homepage":"https://stb-tester.com","size":23828,"stargazers_count":53,"watchers_count":53,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":48,"mirror_url":null,"open_issues_count":22,"forks":48,"open_issues":22,"watchers":53,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/stb-tester/stb-tester/pulls/343"},"html":{"href":"https://github.com/stb-tester/stb-tester/pull/343"},"issue":{"href":"https://api.github.com/repos/stb-tester/stb-tester/issues/343"},"comments":{"href":"https://api.github.com/repos/stb-tester/stb-tester/issues/343/comments"},"review_comments":{"href":"https://api.github.com/repos/stb-tester/stb-tester/pulls/343/comments"},"review_comment":{"href":"https://api.github.com/repos/stb-tester/stb-tester/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/stb-tester/stb-tester/pulls/343/commits"},"statuses":{"href":"https://api.github.com/repos/stb-tester/stb-tester/statuses/8d7d4efd4cbe6ce7794f06429895553e2cd80512"}}}}