{"action":"created","issue":{"url":"https://api.github.com/repos/stb-tester/stb-tester/issues/354","repository_url":"https://api.github.com/repos/stb-tester/stb-tester","labels_url":"https://api.github.com/repos/stb-tester/stb-tester/issues/354/labels{/name}","comments_url":"https://api.github.com/repos/stb-tester/stb-tester/issues/354/comments","events_url":"https://api.github.com/repos/stb-tester/stb-tester/issues/354/events","html_url":"https://github.com/stb-tester/stb-tester/pull/354","id":153486195,"number":354,"title":"stbt auto-selftest: Add caching of image processing result for speedup","user":{"login":"wmanley","id":494659,"avatar_url":"https://avatars.githubusercontent.com/u/494659?v=3","gravatar_id":"","url":"https://api.github.com/users/wmanley","html_url":"https://github.com/wmanley","followers_url":"https://api.github.com/users/wmanley/followers","following_url":"https://api.github.com/users/wmanley/following{/other_user}","gists_url":"https://api.github.com/users/wmanley/gists{/gist_id}","starred_url":"https://api.github.com/users/wmanley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wmanley/subscriptions","organizations_url":"https://api.github.com/users/wmanley/orgs","repos_url":"https://api.github.com/users/wmanley/repos","events_url":"https://api.github.com/users/wmanley/events{/privacy}","received_events_url":"https://api.github.com/users/wmanley/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/stb-tester/stb-tester/milestones/8","html_url":"https://github.com/stb-tester/stb-tester/milestones/v25","labels_url":"https://api.github.com/repos/stb-tester/stb-tester/milestones/8/labels","id":1757751,"number":8,"title":"v25","description":null,"creator":{"login":"drothlis","id":12424,"avatar_url":"https://avatars.githubusercontent.com/u/12424?v=3","gravatar_id":"","url":"https://api.github.com/users/drothlis","html_url":"https://github.com/drothlis","followers_url":"https://api.github.com/users/drothlis/followers","following_url":"https://api.github.com/users/drothlis/following{/other_user}","gists_url":"https://api.github.com/users/drothlis/gists{/gist_id}","starred_url":"https://api.github.com/users/drothlis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drothlis/subscriptions","organizations_url":"https://api.github.com/users/drothlis/orgs","repos_url":"https://api.github.com/users/drothlis/repos","events_url":"https://api.github.com/users/drothlis/events{/privacy}","received_events_url":"https://api.github.com/users/drothlis/received_events","type":"User","site_admin":false},"open_issues":5,"closed_issues":1,"state":"open","created_at":"2016-05-10T13:08:37Z","updated_at":"2016-05-10T13:28:51Z","due_on":null,"closed_at":null},"comments":13,"created_at":"2016-05-06T16:14:38Z","updated_at":"2016-05-10T13:47:23Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/stb-tester/stb-tester/pulls/354","html_url":"https://github.com/stb-tester/stb-tester/pull/354","diff_url":"https://github.com/stb-tester/stb-tester/pull/354.diff","patch_url":"https://github.com/stb-tester/stb-tester/pull/354.patch"},"body":"This makes running `stbt auto-selftest generate` practical on large test-packs with many selftest screenshots.  On a private repo with 196 selftest-screenshots and 73 expressions (14308 doctest items to evaluate) the results are rather promising:\r\n\r\n* 1:01:13 without caching, median expression time 120ms\r\n* 37:50 with a cold cache, median expression time 38ms (1.6x speed up)\r\n* 2:20 total with a hot cache.  Median time per expression: 3.4ms (26x speed up)\r\n\r\nThis is still a little slower than I'd like, but certainly makes `stbt auto-selftest` usable on large test-packs where it just wasn't fast enough before.  Further improvements can come in future PRs.  I'll need to profile first.  Possible speedup: Avoid the need to load the selftest images from disk (up to 17*3=51s saving)\r\n\r\nIn this instance the cache file weighs in at 8.2MB with 22843 entries (361B/entry).\r\n\r\nWe use the xxhash library for fast (non-crypto) image hashing.  I evaluated sha1 and python's `hash()` builtin and they were too slow taking >3ms for a 720p image.  xxhash does the same in 240us.\r\n\r\nWe use lmdb as the on-disk cache database because it's small, fast and has good Python bindings.\r\n\r\nTODO:\r\n\r\n* [x] Measure performance with caching disabled and with a cold cache\r\n* [x] Some tests\r\n* [x] Check Fedora package building\r\n* [x] Bundle py-lmdb as it's not available in Ubuntu 14.04\r\n* [x] Decide how to distribute xxhash and pylmdb?\r\n\r\n    * ~~Include in this repo?~~\r\n    * **Submodules**\r\n    * ~~Download as part of build process?~~\r\n* [x] Think about hashing \"serialisation\".  It's currently rather ugly and not very robust.  Perhaps use json instead with a custom `JSONEncoder` that knows about `numpy.ndarray` and `Gst.Sample`?\r\n* [ ] Mention versions of submodules in commit messages"},"comment":{"url":"https://api.github.com/repos/stb-tester/stb-tester/issues/comments/218162131","html_url":"https://github.com/stb-tester/stb-tester/pull/354#issuecomment-218162131","issue_url":"https://api.github.com/repos/stb-tester/stb-tester/issues/354","id":218162131,"user":{"login":"wmanley","id":494659,"avatar_url":"https://avatars.githubusercontent.com/u/494659?v=3","gravatar_id":"","url":"https://api.github.com/users/wmanley","html_url":"https://github.com/wmanley","followers_url":"https://api.github.com/users/wmanley/followers","following_url":"https://api.github.com/users/wmanley/following{/other_user}","gists_url":"https://api.github.com/users/wmanley/gists{/gist_id}","starred_url":"https://api.github.com/users/wmanley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wmanley/subscriptions","organizations_url":"https://api.github.com/users/wmanley/orgs","repos_url":"https://api.github.com/users/wmanley/repos","events_url":"https://api.github.com/users/wmanley/events{/privacy}","received_events_url":"https://api.github.com/users/wmanley/received_events","type":"User","site_admin":false},"created_at":"2016-05-10T13:47:23Z","updated_at":"2016-05-10T13:47:23Z","body":"> Consider adding this to the last commit's message, or to the imgproc_cache.py docstring, or both:\r\n\r\nI've added a bunch of documentation to `memoize`."}}