{"action":"created","issue":{"url":"https://api.github.com/repos/svaarala/duktape/issues/1094","repository_url":"https://api.github.com/repos/svaarala/duktape","labels_url":"https://api.github.com/repos/svaarala/duktape/issues/1094/labels{/name}","comments_url":"https://api.github.com/repos/svaarala/duktape/issues/1094/comments","events_url":"https://api.github.com/repos/svaarala/duktape/issues/1094/events","html_url":"https://github.com/svaarala/duktape/pull/1094","id":190534986,"number":1094,"title":"Add util/runtest.py for running individual testcases","user":{"login":"svaarala","id":6377415,"avatar_url":"https://avatars.githubusercontent.com/u/6377415?v=3","gravatar_id":"","url":"https://api.github.com/users/svaarala","html_url":"https://github.com/svaarala","followers_url":"https://api.github.com/users/svaarala/followers","following_url":"https://api.github.com/users/svaarala/following{/other_user}","gists_url":"https://api.github.com/users/svaarala/gists{/gist_id}","starred_url":"https://api.github.com/users/svaarala/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/svaarala/subscriptions","organizations_url":"https://api.github.com/users/svaarala/orgs","repos_url":"https://api.github.com/users/svaarala/repos","events_url":"https://api.github.com/users/svaarala/events{/privacy}","received_events_url":"https://api.github.com/users/svaarala/received_events","type":"User","site_admin":false},"labels":[{"id":253655632,"url":"https://api.github.com/repos/svaarala/duktape/labels/testing","name":"testing","color":"fef2c0","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":30,"created_at":"2016-11-20T02:33:46Z","updated_at":"2016-12-06T18:41:23Z","closed_at":"2016-11-25T00:12:30Z","pull_request":{"url":"https://api.github.com/repos/svaarala/duktape/pulls/1094","html_url":"https://github.com/svaarala/duktape/pull/1094","diff_url":"https://github.com/svaarala/duktape/pull/1094.diff","patch_url":"https://github.com/svaarala/duktape/pull/1094.patch"},"body":"Add `util/runtest.py` which allows running individual testcases manually and from scripts:\r\n- Parse testcase metadata from input file.\r\n- Parse testcase expect string from input file.\r\n- Add a testcase framework prefix (with e.g. print/console.log harmonization) and process includes to create a stand-alone prepared testcase (which can be executed as is).\r\n- Execute the test, with optional valgrind wrapping.\r\n- Analyze test results.\r\n- Print a test summary with a diff snippet.\r\n- Output test results into a JSON file for other tools to process (e.g. when running a test suite the suite summary can be created using this output).\r\n\r\nThe tool can also just prepare the test without running it, so that it's then easy to run it manually using e.g. Node.js and edit the test.\r\n\r\nTasks:\r\n- [x] Add a minimally functioning version\r\n- [x] Resolve FIXMEs\r\n- [x] Use \"diff\" by default, and if missing on Windows default to FC\r\n- [x] Check overlap with `util/prep_test.py`, e.g. to complete minifier code\r\n- [x] Create knownissue files for existing known issues\r\n- [x] Test on Windows\r\n- [x] Documentation for revised testcase format\r\n\r\nThe runtest.py tool supports YAML for metadata test case to allow more compact syntax:\r\n\r\n```\r\n/*---\r\ncustom: true\r\n---*/\r\n```\r\n\r\nand allows an inline expect string syntax:\r\n\r\n```\r\nprint('aiee');  //>aiee\r\n```\r\n\r\nKnown issues are now described by individual files in `tests/knownissues/`. The file contains a YAML document which provides a summary and other metadata for the known issue, optionally followed by `---` and the expected output for that known issue (i.e. the exact buggy output):\r\n\r\n```\r\nsummary: prints 'aiee' incorrectly\r\ndescription: >\r\n    Prints 'aiee' incorrectly because foo is not bar.\r\n----\r\nfoo\r\nbar\r\naiee\r\n```\r\n\r\nThe expected (broken) output can also be specified using an MD5 key which is useful for huge outputs:\r\n\r\n\r\n```\r\nsummary: prints 'aiee' incorrectly\r\ndescription: >\r\n    Prints 'aiee' incorrectly because foo is not bar.\r\nmd5: bf8759a584cfe6d800c45de80fc59582\r\n```\r\n\r\nThis allows known issue to be tracked more precisely; previously `doc/testcase-known-issues.yaml` just listed failing tests with a description but the output itself was not checked. So it was possible for a test to fail differently than before without being noticed.\r\n\r\nThis pull doesn't change runtests or any existing test infrastructure yet. Later pulls will change runtests to use this tool for executing tests and eventually the Javascript-based runtests.js will be replaced with something a bit better."},"comment":{"url":"https://api.github.com/repos/svaarala/duktape/issues/comments/265234516","html_url":"https://github.com/svaarala/duktape/pull/1094#issuecomment-265234516","issue_url":"https://api.github.com/repos/svaarala/duktape/issues/1094","id":265234516,"user":{"login":"fatcerberus","id":4097841,"avatar_url":"https://avatars.githubusercontent.com/u/4097841?v=3","gravatar_id":"","url":"https://api.github.com/users/fatcerberus","html_url":"https://github.com/fatcerberus","followers_url":"https://api.github.com/users/fatcerberus/followers","following_url":"https://api.github.com/users/fatcerberus/following{/other_user}","gists_url":"https://api.github.com/users/fatcerberus/gists{/gist_id}","starred_url":"https://api.github.com/users/fatcerberus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fatcerberus/subscriptions","organizations_url":"https://api.github.com/users/fatcerberus/orgs","repos_url":"https://api.github.com/users/fatcerberus/repos","events_url":"https://api.github.com/users/fatcerberus/events{/privacy}","received_events_url":"https://api.github.com/users/fatcerberus/received_events","type":"User","site_admin":false},"created_at":"2016-12-06T18:41:23Z","updated_at":"2016-12-06T18:41:23Z","body":"Hm, there doesn't seem to be a way to get it to print the entire output of a test, so I can compare manually?  The diffs often don't have enough context for me to gauge where things went wrong."}}
{"actor":{"display_login":"fatcerberus"}}