{"action":"created","issue":{"url":"https://api.github.com/repos/project-fuzzball/test/issues/5","repository_url":"https://api.github.com/repos/project-fuzzball/test","labels_url":"https://api.github.com/repos/project-fuzzball/test/issues/5/labels{/name}","comments_url":"https://api.github.com/repos/project-fuzzball/test/issues/5/comments","events_url":"https://api.github.com/repos/project-fuzzball/test/issues/5/events","html_url":"https://github.com/project-fuzzball/test/issues/5","id":160780066,"number":5,"title":"Overall Design","user":{"login":"rtfeldman","id":1094080,"avatar_url":"https://avatars.githubusercontent.com/u/1094080?v=3","gravatar_id":"","url":"https://api.github.com/users/rtfeldman","html_url":"https://github.com/rtfeldman","followers_url":"https://api.github.com/users/rtfeldman/followers","following_url":"https://api.github.com/users/rtfeldman/following{/other_user}","gists_url":"https://api.github.com/users/rtfeldman/gists{/gist_id}","starred_url":"https://api.github.com/users/rtfeldman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rtfeldman/subscriptions","organizations_url":"https://api.github.com/users/rtfeldman/orgs","repos_url":"https://api.github.com/users/rtfeldman/repos","events_url":"https://api.github.com/users/rtfeldman/events{/privacy}","received_events_url":"https://api.github.com/users/rtfeldman/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":13,"created_at":"2016-06-16T23:03:09Z","updated_at":"2016-06-24T04:02:57Z","closed_at":null,"body":"**NOT READY FOR PRODUCTION USE YET!**\r\n\r\n# Project Fuzzball: Revamping elm-test\r\n\r\nThis is something I've been working on with @mgold, with design input from @evancz, @eeue56, @scottcorgan, and @avh4.\r\n\r\nThe goal is for this to replace `elm-test`, but we wanted to get feedback first so we're doing a beta release under the codename Project Fuzzball. Once we're satisfied that it's a worthy successor for `elm-test`, we'll merge these changes into `elm-community/elm-test` and deprecate this package.\r\n\r\nThis is called \"project-fuzzball\" and not something like \"elm-test-revamp\" to avoid confusion on the package repo. This is not battle-tested yet!\r\n\r\n## Trying the beta\r\n\r\nGet the beta release with `npm install -g project-fuzzball-test` - but BEWARE! The binary it produces is called `elm-test`, so this **will** replace your global `elm-test` installation if you have one!\r\n\r\nDocumentation:\r\n\r\n[Test](http://package.elm-lang.org/packages/project-fuzzball/test/1.0.0/Test) (basic API)\r\n[Node.js runner](http://package.elm-lang.org/packages/project-fuzzball/node/1.0.0/Test-Runner-Node) (use with the `elm-test` binary)\r\n[HTML runner](http://package.elm-lang.org/packages/project-fuzzball/test-runner/1.0.0/Test-Runner-Html) (use with `elm-reactor`)\r\n\r\n## Upgrading Gradually\r\n\r\nThere are a bunch of API changes in here, but upgrading is not an all-or-nothing proposition. [legacy-elm-test](https://github.com/rtfeldman/legacy-elm-test) is a drop-in replacement for today's `elm-test` API (including `elm-check` integration) that is written in terms of the new API. The idea is that you can drop it in and start using the new runners right away, without rewriting your tests.\r\n\r\n(You'll still want to upgrade eventually, to get the benefits of the new stuff, but this gives you a more gradual path forward.)\r\n\r\n## Screenshots\r\n\r\nThis is what you get when a test run fails after installing running something like `elm-test ExampleFailingTests.elm`.\r\n\r\nColors are set using [chalk](https://www.npmjs.com/package/chalk), which detects when being run on a CI server and doesn't bother outputting them. Windows users see intentionally-chosen ASCII alternatives to the unicode characters here.\r\n\r\n<img width=\"494\" alt=\"screen shot 2016-06-16 at 4 01 01 pm\" src=\"https://cloud.githubusercontent.com/assets/1094080/16135939/9711ada8-33db-11e6-8dbc-e8caec22fe9b.png\">\r\n\r\nHere's the HTML runner's output.\r\n\r\nYou get this by starting `elm-reactor` and opening your test file.\r\n\r\n<img alt=\"screen shot 2016-06-16 at 3 59 30 pm\" src=\"https://cloud.githubusercontent.com/assets/1094080/16135890/5a864b00-33db-11e6-8ab0-9ea60152f785.png\">\r\n\r\n## First-class fuzz tests\r\n\r\nThis is the big change. In the new API, here's how you'd write a test that runs once:\r\n\r\n```elm\r\ntestRemoveNumbers : Test\r\ntestRemoveNumbers =\r\n    describe \"the removeNumbers function\"\r\n        [ test \"returns an empty string when there are no numbers\" <|\r\n            \\_ ->\r\n                removeNumbers \"blah blah\"\r\n                    |> Assert.equal \"\"\r\n        ]\r\n```\r\n\r\nHere's how you write a [fuzz test](https://en.wikipedia.org/wiki/Fuzz_testing) that runs several times with randomly-generated inputs:\r\n\r\n```elm\r\ntestRemoveNumbers : Test\r\ntestRemoveNumbers =\r\n    describe \"the removeNumbers function\"\r\n        [ fuzz3 string float string \"removes a number in the middle of a string\" <|\r\n            \\prefix num suffix ->\r\n                removeNumbers (prefix ++ toString num ++ suffix)\r\n                    |> Assert.equal (prefix ++ suffix)\r\n        ]\r\n```\r\n\r\nThat's it.\r\n\r\nThis will generate a random string, a random float, and a random string, and pass them as arguments to that anonymous function. The function will get run 100 times with different randomly-generated arguments each time. (You can configure the run count if you don't want 100.)\r\n\r\nIf all of those runs pass, the test as a whole passes. If any fail, `elm-test` will do a \"shrinking\" process to try and find the minimal set of randomly-generated inputs that reproduce the failure (so you don't see redundant failures). It'll then report all the failures it encountered for this test, followed by the randomly-generated inputs that led to the failure:\r\n\r\n```\r\n↓ the removeNumbers function\r\n✗ removes a number in the middle of a string\r\n\r\n► Given (\"foo\", 1.1, \"bar\")\r\n\r\nExpected \"foo.bar\"\r\nto equal \"foobar\"\r\n\r\n► Given (\"foo\", -1, \"bar\")\r\n\r\nExpected \"foo-bar\"\r\nto equal \"foobar\"\r\n```\r\n\r\nThis one failed test surfaces two bugs. We wrote a function that removes numbers, but although we're removing digits, we aren't properly detecting and removing decimal points or minus signs.\r\n\r\nThese \"fuzz tests\" are now both easier to write (no need to separately integrate `elm-check`; it just works right out the box) and give you nicer output (failures stay grouped under the originating test, and include the exact inputs that caused the failure).\r\n\r\n## Non-blocking execution\r\n\r\nAll tests are now wrapped in functions, so they don't get evaluated right when the module loads. This means test runners can do things like reporting progress, running in parallel, etc.\r\n\r\nThe status quo is that the runner evaluates all tests as soon as the module loads, thereby blocking the thread until all the tests have completed. It has made features like those impossible, but the new API means we can actually do things like this.\r\n\r\n## API Change Overview\r\n\r\n* `test` now takes a `(() -> Assertion)` function as its final argument instead of an `Assertion`\r\n* `Assertion` now just wraps either a success or failure (with message), and has no further knowledge of what kind of assertion it is.\r\n* `suite` is now called `describe`\r\n* `Assert` exposes more assertion functions than before\r\n* There are different kinds of runners, which present test run output differently. Check out the [Node runner](http://package.elm-lang.org/packages/project-fuzzball/node/1.0.0/Test-Runner-Node) and the [HTML runner](http://package.elm-lang.org/packages/project-fuzzball/test-runner/1.0.0/Test-Runner-Html)\r\n\r\n## Future Plans\r\n\r\n* Running tests in parallel, so they use all cores, via [elm-web-workers](https://github.com/rtfeldman/elm-web-workers/) (which works on Node too. Yes, really! Multithreading on one process!) Designing the API for non-blocking execution makes this possible without test authors having to do extra work; the feature can just ship and things get faster.\r\n* Providing ways to get CI-friendly structured output, so CI servers can know more than just \"everything passed\" or \"there were some failures\"\r\n* Investigating similarly better support for [testing effects](http://package.elm-lang.org/packages/avh4/elm-testable) and html stuff."},"comment":{"url":"https://api.github.com/repos/project-fuzzball/test/issues/comments/228253366","html_url":"https://github.com/project-fuzzball/test/issues/5#issuecomment-228253366","issue_url":"https://api.github.com/repos/project-fuzzball/test/issues/5","id":228253366,"user":{"login":"rtfeldman","id":1094080,"avatar_url":"https://avatars.githubusercontent.com/u/1094080?v=3","gravatar_id":"","url":"https://api.github.com/users/rtfeldman","html_url":"https://github.com/rtfeldman","followers_url":"https://api.github.com/users/rtfeldman/followers","following_url":"https://api.github.com/users/rtfeldman/following{/other_user}","gists_url":"https://api.github.com/users/rtfeldman/gists{/gist_id}","starred_url":"https://api.github.com/users/rtfeldman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rtfeldman/subscriptions","organizations_url":"https://api.github.com/users/rtfeldman/orgs","repos_url":"https://api.github.com/users/rtfeldman/repos","events_url":"https://api.github.com/users/rtfeldman/events{/privacy}","received_events_url":"https://api.github.com/users/rtfeldman/received_events","type":"User","site_admin":false},"created_at":"2016-06-24T04:02:57Z","updated_at":"2016-06-24T04:02:57Z","body":"Made a PR to see how everything would shake out with `Expect` style: https://github.com/project-fuzzball/test/pull/10"}}
{"actor":{"display_login":"rtfeldman"}}