{"action":"created","issue":{"url":"https://api.github.com/repos/sebastianbergmann/phpunit/issues/3300","repository_url":"https://api.github.com/repos/sebastianbergmann/phpunit","labels_url":"https://api.github.com/repos/sebastianbergmann/phpunit/issues/3300/labels{/name}","comments_url":"https://api.github.com/repos/sebastianbergmann/phpunit/issues/3300/comments","events_url":"https://api.github.com/repos/sebastianbergmann/phpunit/issues/3300/events","html_url":"https://github.com/sebastianbergmann/phpunit/issues/3300","id":358562803,"node_id":"MDU6SXNzdWUzNTg1NjI4MDM=","number":3300,"title":"Exit code 0 on failures when having more than one testsuite","user":{"login":"taisph","id":814148,"node_id":"MDQ6VXNlcjgxNDE0OA==","avatar_url":"https://avatars1.githubusercontent.com/u/814148?v=4","gravatar_id":"","url":"https://api.github.com/users/taisph","html_url":"https://github.com/taisph","followers_url":"https://api.github.com/users/taisph/followers","following_url":"https://api.github.com/users/taisph/following{/other_user}","gists_url":"https://api.github.com/users/taisph/gists{/gist_id}","starred_url":"https://api.github.com/users/taisph/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/taisph/subscriptions","organizations_url":"https://api.github.com/users/taisph/orgs","repos_url":"https://api.github.com/users/taisph/repos","events_url":"https://api.github.com/users/taisph/events{/privacy}","received_events_url":"https://api.github.com/users/taisph/received_events","type":"User","site_admin":false},"labels":[{"id":904686731,"node_id":"MDU6TGFiZWw5MDQ2ODY3MzE=","url":"https://api.github.com/repos/sebastianbergmann/phpunit/labels/stale","name":"stale","color":"ffffff","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-09-10T10:30:35Z","updated_at":"2018-11-16T13:47:47Z","closed_at":"2018-11-16T11:04:43Z","author_association":"NONE","body":"| Q                   | A\r\n| --------------------| ---------------\r\n| PHPUnit version     | 7.3.5\r\n| PHP version         | 7.2.9\r\n| Installation Method | Composer\r\n\r\nI have two testsuites defined and I introduced an assertion failure in both suites. If I run phpunit normally, both suites are run and the assertion failures are hit in both - but the exit code is 0. If I run each testsuite individually, the exit code is 1. I expect any failure to always return non-zero when test ends. Oddly a test error always seem to return non-zero regardless of the number of suites.\r\n\r\nConfig:\r\n```xml\r\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\n<phpunit xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\r\n         xsi:noNamespaceSchemaLocation=\"http://schema.phpunit.de/7.3/phpunit.xsd\"\r\n         verbose=\"false\"\r\n         backupGlobals=\"false\"\r\n         backupStaticAttributes=\"false\"\r\n         bootstrap=\"bootstrap/app.php\"\r\n         colors=\"true\"\r\n         convertErrorsToExceptions=\"true\"\r\n         convertNoticesToExceptions=\"true\"\r\n         convertWarningsToExceptions=\"true\"\r\n         processIsolation=\"false\"\r\n         stopOnFailure=\"false\">\r\n  <testsuites>\r\n    <testsuite name=\"Unit\">\r\n      <directory suffix=\"Test.php\">tests/Unit</directory>\r\n    </testsuite>\r\n    <testsuite name=\"Integration\">\r\n      <directory suffix=\"Test.php\">tests/Integration</directory>\r\n    </testsuite>\r\n  </testsuites>\r\n  <filter>\r\n    <whitelist processUncoveredFilesFromWhitelist=\"true\">\r\n      <directory suffix=\".php\">./app</directory>\r\n    </whitelist>\r\n  </filter>\r\n  <php>\r\n    <env name=\"APP_TIMEZONE\" value=\"UTC\"/>\r\n    <ini name=\"memory_limit\" value=\"2048M\"/>\r\n    <ini name=\"max_execution_time\" value=\"0\"/>\r\n  </php>\r\n</phpunit>\r\n```\r\n\r\nRunning both:\r\n```\r\nvendor/bin/phpunit ; echo $?\r\nPHPUnit 7.3.5 by Sebastian Bergmann and contributors.\r\n\r\n...............................................................  63 / 240 ( 26%)\r\nS......S.................................................F..... 126 / 240 ( 52%)\r\n............................................................... 189 / 240 ( 78%)\r\n...........................................F.......             240 / 240 (100%)\r\n\r\nTime: 10.78 seconds, Memory: 44.00MB\r\n\r\nThere were 2 failures:\r\n\r\n1) Test\\Unit\\FailTest::testFailNow\r\nFailed asserting that false is true.\r\n\r\n/app/tests/Unit/FailTest.php:9\r\n\r\n2) Test\\Integration\\FailTest::testFailNow\r\nFailed asserting that false is true.\r\n\r\n/app/tests/Integration/FailTest.php:9\r\n\r\nFAILURES!\r\nTests: 240, Assertions: 619, Failures: 2, Skipped: 2.\r\n0\r\n```\r\n\r\nRunning one suite:\r\n```\r\nvendor/bin/phpunit --testsuite Unit ; echo $?\r\nPHPUnit 7.3.5 by Sebastian Bergmann and contributors.\r\n\r\n...............................................................  63 / 219 ( 28%)\r\nS......S.................................................F..... 126 / 219 ( 57%)\r\n............................................................... 189 / 219 ( 86%)\r\n..............................                                  219 / 219 (100%)\r\n\r\nTime: 7.13 seconds, Memory: 42.00MB\r\n\r\nThere was 1 failure:\r\n\r\n1) Test\\Unit\\FailTest::testFailNow\r\nFailed asserting that false is true.\r\n\r\n/app/tests/Unit/FailTest.php:9\r\n\r\nFAILURES!\r\nTests: 219, Assertions: 522, Failures: 1, Skipped: 2.\r\n1\r\n```\r\n\r\ncomposer info:\r\n```\r\ndnoegel/php-xdg-base-dir              0.1                implementation of ...\r\ndoctrine/inflector                    v1.3.0             Common String Mani...\r\ndoctrine/instantiator                 1.1.0              A small, lightweig...\r\ndragonmantank/cron-expression         v2.2.0             CRON for PHP: Calc...\r\nfzaninotto/faker                      v1.8.0             Faker is a PHP lib...\r\nhamcrest/hamcrest-php                 v2.0.0             This is the PHP po...\r\nilluminate/auth                       v5.6.34            The Illuminate Aut...\r\nilluminate/broadcasting               v5.6.34            The Illuminate Bro...\r\nilluminate/bus                        v5.6.34            The Illuminate Bus...\r\nilluminate/cache                      v5.6.34            The Illuminate Cac...\r\nilluminate/config                     v5.6.34            The Illuminate Con...\r\nilluminate/console                    v5.6.34            The Illuminate Con...\r\nilluminate/container                  v5.6.34            The Illuminate Con...\r\nilluminate/contracts                  v5.6.34            The Illuminate Con...\r\nilluminate/database                   v5.6.34            The Illuminate Dat...\r\nilluminate/encryption                 v5.6.34            The Illuminate Enc...\r\nilluminate/events                     v5.6.34            The Illuminate Eve...\r\nilluminate/filesystem                 v5.6.34            The Illuminate Fil...\r\nilluminate/hashing                    v5.6.34            The Illuminate Has...\r\nilluminate/http                       v5.6.34            The Illuminate Htt...\r\nilluminate/log                        v5.6.34            The Illuminate Log...\r\nilluminate/pagination                 v5.6.34            The Illuminate Pag...\r\nilluminate/pipeline                   v5.6.34            The Illuminate Pip...\r\nilluminate/queue                      v5.6.34            The Illuminate Que...\r\nilluminate/session                    v5.6.34            The Illuminate Ses...\r\nilluminate/support                    v5.6.34            The Illuminate Sup...\r\nilluminate/translation                v5.6.34            The Illuminate Tra...\r\nilluminate/validation                 v5.6.34            The Illuminate Val...\r\nilluminate/view                       v5.6.33            The Illuminate Vie...\r\njakub-onderka/php-console-color       0.1\r\njakub-onderka/php-console-highlighter v0.3.2\r\nlaravel/lumen-framework               v5.6.4             The Laravel Lumen ...\r\nleague/flysystem                      1.0.46             Filesystem abstrac...\r\nmetroplex-systems/edifact             1.1.0              Parser and Seriali...\r\nmockery/mockery                       1.1.0              Mockery is a simpl...\r\nmonolog/monolog                       1.23.0             Sends your logs to...\r\nmyclabs/deep-copy                     1.8.1              Create deep copies...\r\nnesbot/carbon                         1.33.0             A simple API exten...\r\nnikic/fast-route                      v1.3.0             Fast request route...\r\nnikic/php-parser                      v4.0.3             A PHP parser writt...\r\nparagonie/random_compat               v2.0.17            PHP 5.x polyfill f...\r\npda/pheanstalk                        v3.1.0             PHP client for bea...\r\nphar-io/manifest                      1.0.3              Component for read...\r\nphar-io/version                       2.0.1              Library for handli...\r\nphpdocumentor/reflection-common       1.0.1              Common reflection ...\r\nphpdocumentor/reflection-docblock     4.3.0              With this componen...\r\nphpdocumentor/type-resolver           0.4.0\r\nphpspec/prophecy                      1.8.0              Highly opinionated...\r\nphpunit/php-code-coverage             6.0.7              Library that provi...\r\nphpunit/php-file-iterator             2.0.1              FilterIterator imp...\r\nphpunit/php-text-template             1.2.1              Simple template en...\r\nphpunit/php-timer                     2.0.0              Utility class for ...\r\nphpunit/php-token-stream              3.0.0              Wrapper around PHP...\r\nphpunit/phpunit                       7.3.5              The PHP Unit Testi...\r\npsr/container                         1.0.0              Common Container I...\r\npsr/log                               1.0.2              Common interface f...\r\npsr/simple-cache                      1.0.1              Common interfaces ...\r\npsy/psysh                             v0.9.7             An interactive she...\r\nramsey/uuid                           3.8.0              Formerly rhumsaa/u...\r\nsebastian/code-unit-reverse-lookup    1.0.1              Looks up which fun...\r\nsebastian/comparator                  3.0.2              Provides the funct...\r\nsebastian/diff                        3.0.1              Diff implementation\r\nsebastian/environment                 3.1.0              Provides functiona...\r\nsebastian/exporter                    3.1.0              Provides the funct...\r\nsebastian/global-state                2.0.0              Snapshotting of gl...\r\nsebastian/object-enumerator           3.0.3              Traverses array st...\r\nsebastian/object-reflector            1.1.1              Allows reflection ...\r\nsebastian/recursion-context           3.0.0              Provides functiona...\r\nsebastian/resource-operations         1.0.0              Provides a list of...\r\nsebastian/version                     2.0.1              Library that helps...\r\nstefk/jval                            dev-master 1c26dd2 JSON Schema validator\r\nswiftmailer/swiftmailer               v5.4.12            Swiftmailer, free ...\r\nsymfony/console                       v4.1.3             Symfony Console Co...\r\nsymfony/debug                         v4.1.3             Symfony Debug Comp...\r\nsymfony/event-dispatcher              v4.1.3             Symfony EventDispa...\r\nsymfony/finder                        v4.1.3             Symfony Finder Com...\r\nsymfony/http-foundation               v4.1.3             Symfony HttpFounda...\r\nsymfony/http-kernel                   v4.1.3             Symfony HttpKernel...\r\nsymfony/inflector                     v4.1.3             Symfony Inflector ...\r\nsymfony/options-resolver              v4.1.3             Symfony OptionsRes...\r\nsymfony/polyfill-ctype                v1.9.0             Symfony polyfill f...\r\nsymfony/polyfill-mbstring             v1.9.0             Symfony polyfill f...\r\nsymfony/polyfill-php72                v1.9.0             Symfony polyfill b...\r\nsymfony/process                       v4.1.3             Symfony Process Co...\r\nsymfony/property-access               v4.1.3             Symfony PropertyAc...\r\nsymfony/translation                   v4.1.3             Symfony Translatio...\r\nsymfony/var-dumper                    v4.1.3             Symfony mechanism ...\r\ntedivm/fetch                          v0.7.1             A PHP IMAP Library\r\ntheseer/tokenizer                     1.1.0              A small library fo...\r\nvlucas/phpdotenv                      v2.5.1             Loads environment ...\r\nvluzrmos/tinker                       v1.4.0             An Interactive She...\r\nwebmozart/assert                      1.3.0              Assertions to vali...\r\nyohang/finite                         1.1.2              A simple PHP5.3+ F...\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/sebastianbergmann/phpunit/issues/comments/439397763","html_url":"https://github.com/sebastianbergmann/phpunit/issues/3300#issuecomment-439397763","issue_url":"https://api.github.com/repos/sebastianbergmann/phpunit/issues/3300","id":439397763,"node_id":"MDEyOklzc3VlQ29tbWVudDQzOTM5Nzc2Mw==","user":{"login":"epdenouden","id":26651359,"node_id":"MDQ6VXNlcjI2NjUxMzU5","avatar_url":"https://avatars1.githubusercontent.com/u/26651359?v=4","gravatar_id":"","url":"https://api.github.com/users/epdenouden","html_url":"https://github.com/epdenouden","followers_url":"https://api.github.com/users/epdenouden/followers","following_url":"https://api.github.com/users/epdenouden/following{/other_user}","gists_url":"https://api.github.com/users/epdenouden/gists{/gist_id}","starred_url":"https://api.github.com/users/epdenouden/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/epdenouden/subscriptions","organizations_url":"https://api.github.com/users/epdenouden/orgs","repos_url":"https://api.github.com/users/epdenouden/repos","events_url":"https://api.github.com/users/epdenouden/events{/privacy}","received_events_url":"https://api.github.com/users/epdenouden/received_events","type":"User","site_admin":false},"created_at":"2018-11-16T13:47:47Z","updated_at":"2018-11-16T13:47:47Z","author_association":"CONTRIBUTOR","body":"@taisph I just did a quick checking using PHPUnit's own test suite and @kubawerlos is right, I also cannot replicate what you found. If you want this issue looked at please provide a self-contained and small example.\r\n\r\nTo test I broke one test in each of the `unit` and `end-to-end` suites and this gave me an exit code of `1` as expected:\r\n\r\n```\r\n./phpunit ; echo $?\r\n```\r\n\r\ngave me\r\n\r\n```\r\nPHPUnit 7.5-g342c076c7 by Sebastian Bergmann and contributors.\r\n\r\nRuntime:       PHP 7.2.11 with Xdebug 2.6.0\r\n✂️\r\n\r\nThere were 2 failures:\r\n\r\n1) PHPUnit\\Framework\\AssertTest::testBlerp\r\nFailed asserting that false is true.\r\n✂️ \r\n\r\n2) /Users/ewout/proj/phpunit-new-order/tests/end-to-end/testdox.phpt\r\nFailed asserting that string matches format description.\r\n--- Expected\r\n+++ Actual\r\n@@ @@\r\n-xPHPUnit %s by Sebastian Bergmann and contributors.\r\n+PHPUnit 7.5-g342c076c7 by Sebastian Bergmann and contributors.\r\n✂️\r\n\r\nFAILURES!\r\nTests: 2033, Assertions: 3525, Failures: 2, Skipped: 4.\r\n1\r\n```"}}
{"actor":{"display_login":"epdenouden"}}