{"action":"created","issue":{"url":"https://api.github.com/repos/carp-lang/Carp/issues/216","repository_url":"https://api.github.com/repos/carp-lang/Carp","labels_url":"https://api.github.com/repos/carp-lang/Carp/issues/216/labels{/name}","comments_url":"https://api.github.com/repos/carp-lang/Carp/issues/216/comments","events_url":"https://api.github.com/repos/carp-lang/Carp/issues/216/events","html_url":"https://github.com/carp-lang/Carp/issues/216","id":310367909,"number":216,"title":"Tests should also check output","user":{"login":"mkfifo","id":75518,"avatar_url":"https://avatars0.githubusercontent.com/u/75518?v=4","gravatar_id":"","url":"https://api.github.com/users/mkfifo","html_url":"https://github.com/mkfifo","followers_url":"https://api.github.com/users/mkfifo/followers","following_url":"https://api.github.com/users/mkfifo/following{/other_user}","gists_url":"https://api.github.com/users/mkfifo/gists{/gist_id}","starred_url":"https://api.github.com/users/mkfifo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mkfifo/subscriptions","organizations_url":"https://api.github.com/users/mkfifo/orgs","repos_url":"https://api.github.com/users/mkfifo/repos","events_url":"https://api.github.com/users/mkfifo/events{/privacy}","received_events_url":"https://api.github.com/users/mkfifo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-04-02T00:45:27Z","updated_at":"2018-04-07T03:07:21Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Ideally we should have a suite of tests where each is a pair of `foo.carp` (input program) and `foo.out.expected` (expected stdout output) to be compared against\r\n\r\nwe can then have these run within each Travis test, so we can detect unexpected ouput changes"},"comment":{"url":"https://api.github.com/repos/carp-lang/Carp/issues/comments/379427774","html_url":"https://github.com/carp-lang/Carp/issues/216#issuecomment-379427774","issue_url":"https://api.github.com/repos/carp-lang/Carp/issues/216","id":379427774,"user":{"login":"mkfifo","id":75518,"avatar_url":"https://avatars0.githubusercontent.com/u/75518?v=4","gravatar_id":"","url":"https://api.github.com/users/mkfifo","html_url":"https://github.com/mkfifo","followers_url":"https://api.github.com/users/mkfifo/followers","following_url":"https://api.github.com/users/mkfifo/following{/other_user}","gists_url":"https://api.github.com/users/mkfifo/gists{/gist_id}","starred_url":"https://api.github.com/users/mkfifo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mkfifo/subscriptions","organizations_url":"https://api.github.com/users/mkfifo/orgs","repos_url":"https://api.github.com/users/mkfifo/repos","events_url":"https://api.github.com/users/mkfifo/events{/privacy}","received_events_url":"https://api.github.com/users/mkfifo/received_events","type":"User","site_admin":false},"created_at":"2018-04-07T03:07:21Z","updated_at":"2018-04-07T03:07:21Z","author_association":"CONTRIBUTOR","body":"Generally unit tests test a unit in isolation, this is often a single function/feature, testing input against output over a set of cases. For this I was thinking more integration or system tests.\r\n\r\nI was thinking of including a set of decent sized programs and testing these against their expected output as a whole, a good start would be examples/*.carp and test/*.carp.\r\n\r\nThe idea of these is to form a set of larger tests which try to capture a large cunk of the Carp sytem and runtime in order to capture regressions.\r\n\r\nThese also begin to form some kind of documentation about the system.\r\n\r\n\r\n\r\nFor my toy language Icarus I had:\r\n\r\n* 'unit tests' [1] written in C (the host language) to test individual units of the language implementation\r\n* 'milestone' tests [2] each covering one new feature more than the previous run and trying to test this feature end-to-end in isolation\r\n* 'custom' tests [3] [4] [5] [6] which try to capture a wide array of small programs\r\n\r\nboth 'milestones' and my 'custom' tests are testing the output, some of my custom tests also test the various intermediate forms and compilation targets (AST, IR [7], Bytecode [8], C [9], ...).\r\n\r\nThese tests meant that whenever I had a regression I was able to quickly narrow it down to the exact phase it happened in, as well as giving me a large set of examples of where it fails/pases.\r\n\r\nThese kinds of tests only make sense if your language syntax and behaviour is reasonably fixed, adding new features is possible, but large-scale changes means you have lots of tests to also update.\r\n\r\n[1] https://github.com/mkfifo/icarus/tree/master/t/unit \r\n[2] https://github.com/mkfifo/icarus/tree/master/t/milestones\r\n[3] https://github.com/mkfifo/icarus/tree/master/t/custom\r\n[4] https://github.com/mkfifo/icarus/blob/master/t/custom/test_generics_progress.pl\r\n[5] https://github.com/mkfifo/icarus/blob/master/t/custom/test_backend_pancake_end_to_end_output_only.pl\r\n[6] https://github.com/mkfifo/icarus/blob/master/t/custom/test_backend_2c_end_to_end_output_only.pl\r\n[7] https://github.com/mkfifo/icarus/blob/master/t/custom/test_transform_simple.pl#L10\r\n[8] https://github.com/mkfifo/icarus/blob/master/t/custom/test_backend_pancake_end_to_end.pl#L51\r\n[9] https://github.com/mkfifo/icarus/blob/master/t/custom/test_backend_2c_end_to_end.pl#L24\r\n"}}
{"actor":{"display_login":"mkfifo"}}