{"action":"created","issue":{"url":"https://api.github.com/repos/qunitjs/qunit/issues/1064","repository_url":"https://api.github.com/repos/qunitjs/qunit","labels_url":"https://api.github.com/repos/qunitjs/qunit/issues/1064/labels{/name}","comments_url":"https://api.github.com/repos/qunitjs/qunit/issues/1064/comments","events_url":"https://api.github.com/repos/qunitjs/qunit/issues/1064/events","html_url":"https://github.com/qunitjs/qunit/issues/1064","id":186110803,"number":1064,"title":"Reordering should not cause a module's tests to be split before/after other modules","user":{"login":"wyantb","id":726818,"avatar_url":"https://avatars.githubusercontent.com/u/726818?v=3","gravatar_id":"","url":"https://api.github.com/users/wyantb","html_url":"https://github.com/wyantb","followers_url":"https://api.github.com/users/wyantb/followers","following_url":"https://api.github.com/users/wyantb/following{/other_user}","gists_url":"https://api.github.com/users/wyantb/gists{/gist_id}","starred_url":"https://api.github.com/users/wyantb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wyantb/subscriptions","organizations_url":"https://api.github.com/users/wyantb/orgs","repos_url":"https://api.github.com/users/wyantb/repos","events_url":"https://api.github.com/users/wyantb/events{/privacy}","received_events_url":"https://api.github.com/users/wyantb/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-10-30T01:21:32Z","updated_at":"2016-11-12T20:31:42Z","closed_at":null,"body":"Please bear with me a moment for an explanation - there's a good chance you won't agree with me initially, and I have a sliding scale of thoughts on what could be done with my experience.\n\nTo start, as demonstration, suppose I have a suite that acts like the following:\n\n```\nvar globalFlag = false;\nQUnit.module('A: should never fail anything, but can cleanup global things', {\n    after : function () {\n        globalFlag = true;\n    }\n});\nQUnit.test('always passes', function (assert) {\n    assert.ok(true);\n});\n\nQUnit.module('B: should I have one fail, or two?');\nQUnit.test('fail if globalFlag is true', function (assert) {\n    assert.notOk(globalFlag);\n});\nQUnit.test('fail if globalFlag is true, or just because', function (assert) {\n    assert.notOk(globalFlag);\n    assert.ok(false);\n});\n```\n\nAfter running the tests in my browser, and refreshing once or twice, I see the following pattern; on one refresh, 1 failure, on my next refresh, 2 failures:\n\n\"Module interrupted\" scenario - two failures\n\n```\nB's \"before\"\n*** B: should I have one fail, or two? *** Test : fail if globalFlag is true, or just because\n*** A: should never fail anything, but can cleanup global things *** Test : always passes\n*** B: should I have one fail, or two? *** Test : fail if globalFlag is true\nB's \"after\"\n```\n\n\"Module uninterrupted\" scenario - one failure\n\n```\nB's \"before\"\n*** B: should I have one fail, or two? *** Test : fail if globalFlag is true\n*** B: should I have one fail, or two? *** Test : fail if globalFlag is true, or just because\nB's \"after\"\n*** A: should never fail anything, but can cleanup global things *** Test : always passes\n```\n## So, why do I consider this an issue?\n\nI accidentally got myself into a situation that acted like the above yesterday.  In practice, I was unit testing a jQuery widget, and giving the QUnit 2 - style API with `before`/`after` a try for the first time.  Historically, our tests using the jQ widget create a widget on `#qunit-fixture` in `beforeEach`, and tear it down in `afterEach`.\n\nThis is a little inefficient, taking some 20-50milliseconds, so I wanted to try out a new way.  I wanted to create the widget in `before`, reuse it for each test by performing much cheaper `clearAll`-type methods in `afterEach`, then finally tear it down in `after`.\n\nI was successful in this, but then as I was putting together a new test, before the test was good, it failed all the time.  I was _very_ confused when more than just this test failed, though.  It was because other modules were jutting in, trying to create a widget on `#qunit-fixture`, actually getting the existing instance from the `before` of my new module (since `after` hadn't run yet, since module B hadn't finished all its tests yet), and tearing down the grid that module B had created.  Since module B only created the widget in `before`, subsequent tests from module B always failed.\n## What do I propose?\n\nMy guiding instinct here is that `before`/`after` are not all that useful if there's no guarantee that a module will run all of its cases together; if they can get interrupted by unrelated tests, people can burn a few hours trying to figure out why.  In my case, this was a jQuery widget, but I could see similar circumstances for [sinon](http://sinonjs.org/) spy/stub injection on prototypes, dependency injection swaps, etc.\n\nSo, I propose one of the following, in order of personal preference:\n- If `QUnit.config.reorder` is set to true, randomize the order that modules run in, and the order that tests run in within a module if desired, but _do not_ reorder such that a module B's tests can have module A's tests run in between B's `before` and `after` hooks\n  - Correct me if I'm wrong, but I feel this is what JUnit does for Java.  Perhaps I haven't looked carefully enough\n- If `QUnit.config.reorder` is set to true, ensure that any module that defines `before`/`after` hooks in some fashion cannot get interrupted by tests from another module (but leave legacy behavior intact for modules w/o `before`/`after`)\n- Add a new configuration option, defaulted to whatever, that if toggled on would enable one of the two above behaviors\n- Update the documentation to indicate that modules can be interrupted in this fashion, linking to that explanation from the docs for `before`, `after`, and `QUnit.config.reorder`\n## Overall\n\nI do like QUnit, I just wonder if folks other than me might run into this same situation.  But anyway, thank you for reading this far!  \n"},"comment":{"url":"https://api.github.com/repos/qunitjs/qunit/issues/comments/260146530","html_url":"https://github.com/qunitjs/qunit/issues/1064#issuecomment-260146530","issue_url":"https://api.github.com/repos/qunitjs/qunit/issues/1064","id":260146530,"user":{"login":"platinumazure","id":284282,"avatar_url":"https://avatars.githubusercontent.com/u/284282?v=3","gravatar_id":"","url":"https://api.github.com/users/platinumazure","html_url":"https://github.com/platinumazure","followers_url":"https://api.github.com/users/platinumazure/followers","following_url":"https://api.github.com/users/platinumazure/following{/other_user}","gists_url":"https://api.github.com/users/platinumazure/gists{/gist_id}","starred_url":"https://api.github.com/users/platinumazure/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/platinumazure/subscriptions","organizations_url":"https://api.github.com/users/platinumazure/orgs","repos_url":"https://api.github.com/users/platinumazure/repos","events_url":"https://api.github.com/users/platinumazure/events{/privacy}","received_events_url":"https://api.github.com/users/platinumazure/received_events","type":"User","site_admin":false},"created_at":"2016-11-12T20:31:41Z","updated_at":"2016-11-12T20:31:41Z","body":"@trentmwillis What about call counts? If `before` for a given module is run multiple times as the test runner jumps around modules, that's not ideal either. Personally, I think we need to rethink ordering given `before`/`after`-- when reordering was conceived originally, there was only `beforeEach`/`afterEach` (known as setup/teardown then)."}}
{"actor":{"display_login":"platinumazure"}}