{"action":"created","issue":{"url":"https://api.github.com/repos/ARMmbed/greentea/issues/66","labels_url":"https://api.github.com/repos/ARMmbed/greentea/issues/66/labels{/name}","comments_url":"https://api.github.com/repos/ARMmbed/greentea/issues/66/comments","events_url":"https://api.github.com/repos/ARMmbed/greentea/issues/66/events","html_url":"https://github.com/ARMmbed/greentea/issues/66","id":126879379,"number":66,"title":"[Feature] Add support for test case id tagging","user":{"login":"PrzemekWirkus","id":6360720,"avatar_url":"https://avatars.githubusercontent.com/u/6360720?v=3","gravatar_id":"","url":"https://api.github.com/users/PrzemekWirkus","html_url":"https://github.com/PrzemekWirkus","followers_url":"https://api.github.com/users/PrzemekWirkus/followers","following_url":"https://api.github.com/users/PrzemekWirkus/following{/other_user}","gists_url":"https://api.github.com/users/PrzemekWirkus/gists{/gist_id}","starred_url":"https://api.github.com/users/PrzemekWirkus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PrzemekWirkus/subscriptions","organizations_url":"https://api.github.com/users/PrzemekWirkus/orgs","repos_url":"https://api.github.com/users/PrzemekWirkus/repos","events_url":"https://api.github.com/users/PrzemekWirkus/events{/privacy}","received_events_url":"https://api.github.com/users/PrzemekWirkus/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ARMmbed/greentea/labels/enhancement","name":"enhancement","color":"84b6eb"},{"url":"https://api.github.com/repos/ARMmbed/greentea/labels/mirrored","name":"mirrored","color":"ededed"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2016-01-15T14:09:27Z","updated_at":"2016-01-19T10:29:13Z","closed_at":null,"body":"# Description\r\nOne binary can contain may test cases (test fixtures) with a unique test case ids.\r\nWe could add support to yield many test case id results per one test case execution.\r\n\r\n## Related PRs and changes:\r\n* https://github.com/ARMmbed/greentea/pull/67 (Added reporting, still JUnit reporting is not added)\r\n* https://github.com/ARMmbed/htrun/pull/58 (Added APIs and macros detection - test case result)\r\n* https://github.com/ARMmbed/mbed-drivers/pull/141 (Added new test_env APIs to handle new macros)\r\n\r\n## New success code for test case\r\nNew success code for test case is an integer with following properties:\r\nRange of ```MBED_HOSTTEST_TESTCASE_FINISH()``` success:\r\n```\r\n+---------------+--------------+---------------+\r\n|  success < 0  | success == 0 |  success > 0  |\r\n+---------------+--------------+---------------+\r\n| <------ ERROR |      OK      | FAIL -------> |\r\n+---------------+--------------+---------------+\r\n```\r\n\r\n### Example simple test case inside current test source code\r\n```c++\r\n// Somewhere in C++ test case file in your yotta module...\r\n// My test case TESTCASE_NAME implementation\r\n{\r\n    int success = 0; // 0 for success! ;)\r\n    MBED_HOSTTEST_TESTCASE_START(TESTCASE_NAME);\r\n\r\n    // Do your testing bit here...\r\n\r\n    // Make sure to set \"success\" to proper value:\r\n    // 0 - 'success' result of the test case\r\n    // >0 - 'failure' result of the test case\r\n    // <0 - 'inconclusive' result of the test case\r\n\r\n    MBED_HOSTTEST_TESTCASE_FINISH(TESTCASE_NAME,success);\r\n}\r\n```\r\nYou can think about this also like this:\r\n```c++\r\nif (result_of_my_test_case == 0) {\r\n    // OK case !\r\n    success = 0;\r\n} else if (result_of_my_test_case > 0) {\r\n    // FAILed case ! \r\n    // You can add any arbitrary and meaningful for you value < 0.\r\n    // E.g. you can do success = 404; to mark in your test case \r\n    // that \"server\" returned 404 error.\r\n    success = 404;    \r\n} else if (result_of_my_test_case < 0) {\r\n    // ERROR, test case execution was inconclusive\r\n    // You can add any arbitrary and meaningful for you value > 0.\r\n    // E.g. success = -1; when your test framework cough exception.\r\n    success = -1;\r\n}\r\n```\r\n\r\n# Rationale\r\nWe need to move to new mode of work where one binary (test suite) can report multiple test cases results from automated run.\r\n\r\n# Changes\r\n* Added macros to ```mbed-drivers/test_env.h```:\r\n```c++\r\n#define MBED_HOSTTEST_TESTCASE_START(TESTCASE)  notify_testcase_start(#TESTCASE)\r\n#define MBED_HOSTTEST_TESTCASE_FINISH(TESTCASE,SUCCESS)   notify_testcase_completion(#TESTCASE,SUCCESS)\r\n```\r\n* Added macros implementation:\r\n```c++\r\n/** \\brief Notifies test case start\r\n  * \\param Test Case ID name\r\n  *\r\n  * This function notifies test environment abort test case execution start.\r\n  *\r\n  */\r\nvoid notify_testcase_start(const char *testcase_id)\r\n{\r\n    printf(\"{{testcase_start;%s}}\" NL, testcase_id);\r\n}\r\n\r\n/** \\brief Return partial (test case) result from test suite\r\n  * \\param Test Case ID name\r\n  * \\param Success code, 0 - success, >0 failure reason, <0 inconclusive\r\n  *\r\n  * This function passes partial test suite's test case result to test\r\n  * environment.\r\n  * Each test suite (in many cases one binary with tests) can return\r\n  * multiple partial results used to track test case results.\r\n  *\r\n  * Test designers can use success code to return test case:\r\n  * success == 0 - PASS, test case execution was successful.\r\n  * success > 0  - FAILure, e.g. success == 404 can be used to\r\n  *                pass \"Server not found\".\r\n  * success < 0  - Inconclusive test case execution, e.g.\r\n  *\r\n  */\r\nvoid notify_testcase_completion(const char *testcase_id, const int success)\r\n{\r\n    printf(\"{{testcase_finish;%s;%d}}\" NL, testcase_id, success);\r\n}\r\n```\r\n* Added corresponding macros' equivalents on host test side:\r\n```python\r\nselftest.testcase_start(testcase_name)\r\nselftest.testcase_finish(testcase_name, testcase_result)\r\n```\r\n\r\nExample:\r\n```python\r\nclass RTCTest(BaseHostTest):\r\n    def test(self, selftest):\r\n        test_result = True\r\n        selftest.testcase_start(\"RTC0001\")\r\n        # Test case execution\r\n        selftest.testcase_finish(\"RTC0001\", test_result)\r\n        # ...\r\n```\r\n# Example \"STL\" mbed-drivers test case source code with new macros in use\r\nNote: We will modify in this example [mbed-driver's STL](https://github.com/ARMmbed/mbed-drivers/blob/master/test/stl/main.cpp#L76). \r\n\r\nWe are adding two macros at the beginning of the test case and the end of it:\r\n```c++\r\nMBED_HOSTTEST_TESTCASE_START(STL001);\r\n// test case execution\r\nMBED_HOSTTEST_TESTCASE_FINISH(STL001, success_code);\r\n```\r\n\r\nOriginally this test case was designed to run multiple checks and return one result, but now we can get multiple results from each test case and summary result from test suite passed via already used macro: \r\n```c++\r\nMBED_HOSTTEST_RESULT();\r\n```\r\n\r\n```c++\r\nvoid runTest()\r\n{\r\n    int p_integers[] = {POSITIVE_INTEGERS};\r\n    int n_integers[] = {NEGATIVE_INTEGERS};\r\n    float floats[] = {FLOATS};\r\n    bool result = true;\r\n\r\n    MBED_HOSTTEST_TIMEOUT(10);\r\n    MBED_HOSTTEST_SELECT(default_auto);\r\n    MBED_HOSTTEST_DESCRIPTION(C++ STL);\r\n    MBED_HOSTTEST_START(\"MBED_A3\"); \r\n\r\n    // Test case STL001\r\n    {\r\n        MBED_HOSTTEST_TESTCASE_START(STL001);   // New macro\r\n        std::vector<int> v_pints(p_integers, p_integers + TABLE_SIZE(p_integers));\r\n        bool equal_result = std::equal(v_pints.begin(), v_pints.end(), p_integers);\r\n        result = result && equal_result;\r\n        MBED_HOSTTEST_TESTCASE_FINISH(STL001, !result); // New macro\r\n    }\r\n\r\n    // Test case STL002\r\n    {\r\n        MBED_HOSTTEST_TESTCASE_START(STL002);   // New macro\r\n        const char* floats_str[] = {FLOATS_STR};\r\n        float floats_transform[TABLE_SIZE(floats_str)] = {0.0};\r\n        std::transform(floats_str, floats_str + TABLE_SIZE(floats_str), floats_transform, atof);\r\n        bool equal_result = std::equal(floats_transform, floats_transform + TABLE_SIZE(floats_transform), floats);\r\n        result = result && equal_result;\r\n        MBED_HOSTTEST_TESTCASE_FINISH(STL002, !result); // New macro\r\n\r\n        std::for_each(floats_str, floats_str + TABLE_SIZE(floats_str), printString());\r\n        printf(\"\\r\\n\");\r\n        std::for_each(floats_transform, floats_transform + TABLE_SIZE(floats_transform), printFloat());\r\n        printf(\"\\r\\n\");\r\n    }\r\n```\r\n# Example host test for RTC test cases with new functions\r\nIn this examples we refactored ```test(self, selftest)``` function and added two new functions with two very similar test cases (we are only changing look count).\r\n\r\nChanges:\r\n* We will still use return value from ```test(self, selftest)``` to give feedback from test suite.\r\n* We will add new two functions:\r\n```python\r\nselftest.testcase_start(\"RTC0001\")\r\n# Your test case here...\r\nselftest.testcase_finish(\"RTC0001\", test_result)\r\n```\r\nNote that in this case test case is captured inside host test and not inside binary with mbed code.\r\n\r\n```python\r\nclass RTCTest(BaseHostTest):\r\n    PATTERN_RTC_VALUE = \"\\[(\\d+)\\] \\[(\\d+-\\d+-\\d+ \\d+:\\d+:\\d+ [AaPpMm]{2})\\]\"\r\n    re_detect_rtc_value = re.compile(PATTERN_RTC_VALUE)\r\n\r\n    def test(self, selftest):\r\n        test_result = []\r\n        test_result.append(self.test_rtc001(selftest))\r\n        test_result.append(self.test_rtc002(selftest))\r\n        return selftest.RESULT_SUCCESS if all(test_result) else selftest.RESULT_FAILURE\r\n\r\n    def test_rtc001(self, selftest):\r\n        test_result = True\r\n        start = time()\r\n        sec_prev = 0\r\n        selftest.testcase_start(\"RTC0001\")\r\n        for i in range(0, 3):\r\n            # Timeout changed from default: we need to wait longer for some boards to start-up\r\n            c = selftest.mbed.serial_readline(timeout=10)\r\n            if c is None:\r\n                return selftest.RESULT_IO_SERIAL\r\n            selftest.notify(c.strip())\r\n            delta = time() - start\r\n            m = self.re_detect_rtc_value.search(c)\r\n            if m and len(m.groups()):\r\n                sec = int(m.groups()[0])\r\n                time_str = m.groups()[1]\r\n                correct_time_str = strftime(\"%Y-%m-%d %H:%M:%S %p\", gmtime(float(sec)))\r\n                single_result = time_str == correct_time_str and sec > 0 and sec > sec_prev\r\n                test_result = test_result and single_result\r\n                result_msg = \"OK\" if single_result else \"FAIL\"\r\n                selftest.notify(\"HOST: [%s] [%s] received time %+d sec after %.2f sec... %s\"% (sec, time_str, sec - sec_prev, delta, result_msg))\r\n                sec_prev = sec\r\n            else:\r\n                test_result = False\r\n                break\r\n            start = time()\r\n        selftest.testcase_finish(\"RTC0001\", test_result)\r\n        return test_result\r\n\r\n    def test_rtc002(self, selftest):\r\n        test_result = True\r\n        start = time()\r\n        sec_prev = 0\r\n        selftest.testcase_start(\"RTC0002\")\r\n        for i in range(0, 5):\r\n            # Timeout changed from default: we need to wait longer for some boards to start-up\r\n            c = selftest.mbed.serial_readline(timeout=10)\r\n            if c is None:\r\n                return selftest.RESULT_IO_SERIAL\r\n            selftest.notify(c.strip())\r\n            delta = time() - start\r\n            m = self.re_detect_rtc_value.search(c)\r\n            if m and len(m.groups()):\r\n                sec = int(m.groups()[0])\r\n                time_str = m.groups()[1]\r\n                correct_time_str = strftime(\"%Y-%m-%d %H:%M:%S %p\", gmtime(float(sec)))\r\n                single_result = time_str == correct_time_str and sec > 0 and sec > sec_prev\r\n                test_result = test_result and single_result\r\n                result_msg = \"OK\" if single_result else \"FAIL\"\r\n                selftest.notify(\"HOST: [%s] [%s] received time %+d sec after %.2f sec... %s\"% (sec, time_str, sec - sec_prev, delta, result_msg))\r\n                sec_prev = sec\r\n            else:\r\n                test_result = False\r\n                break\r\n            start = time()\r\n        selftest.testcase_finish(\"RTC0002\", test_result)\r\n        return test_result\r\n```\r\n\r\n# Example reporting\r\n```\r\n$ mbedgt -V -n mbed-drivers-test-rtc,mbed-drivers-test-stl,mbed-drivers-test-cstring\r\n...\r\nmbedgt: mbed-host-test-runner: stopped\r\nmbedgt: mbed-host-test-runner: returned 'OK'\r\nmbedgt: test on hardware with target id: 02400226489a1e6c000000000000000000000000b543e3d4\r\nmbedgt: test suite 'mbed-drivers-test-stl' ........................................................... OK in 2.91 sec\r\n        test case 'STL001' ........................................................................... OK in 0.00 sec\r\n        test case 'STL002' ........................................................................... OK in 0.01 sec\r\n        test case 'STL003' ........................................................................... OK in 0.01 sec\r\n        test case 'STL004' ........................................................................... OK in 0.00 sec\r\nmbedgt: test suite report:\r\n+---------------+---------------+---------------------------+--------+--------------------+-------------+\r\n| target        | platform_name | test suite                | result | elapsed_time (sec) | copy_method |\r\n+---------------+---------------+---------------------------+--------+--------------------+-------------+\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-rtc     | OK     | 8.71               | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-cstring | FAIL   | 2.95               | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-stl     | OK     | 2.91               | shell       |\r\n+---------------+---------------+---------------------------+--------+--------------------+-------------+\r\nmbedgt: test suite results: 1 FAIL / 2 OK\r\nmbedgt: test case report:\r\n+---------------+---------------+---------------------------+------------+--------+--------------------+-------------+\r\n| target        | platform_name | test suite                | test case  | result | elapsed_time (sec) | copy_method |\r\n+---------------+---------------+---------------------------+------------+--------+--------------------+-------------+\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-rtc     | RTC0001    | OK     | 3.12               | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-rtc     | RTC0002    | OK     | 4.19               | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-cstring | STRINGS001 | OK     | 0.01               | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-cstring | STRINGS002 | OK     | 0.01               | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-cstring | STRINGS003 | OK     | 0.01               | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-cstring | STRINGS004 | FAIL   | 0.0                | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-cstring | STRINGS005 | FAIL   | 0.0                | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-cstring | STRINGS006 | FAIL   | 0.0                | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-cstring | STRINGS007 | FAIL   | 0.01               | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-cstring | STRINGS008 | FAIL   | 0.0                | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-stl     | STL001     | OK     | 0.0                | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-stl     | STL002     | OK     | 0.01               | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-stl     | STL003     | OK     | 0.01               | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-stl     | STL004     | OK     | 0.0                | shell       |\r\n+---------------+---------------+---------------------------+------------+--------+--------------------+-------------+\r\nmbedgt: test case results: 5 FAIL / 9 OK\r\nmbedgt: completed in 105.04 sec\r\nmbedgt: exited with code 1\r\n```"},"comment":{"url":"https://api.github.com/repos/ARMmbed/greentea/issues/comments/172811074","html_url":"https://github.com/ARMmbed/greentea/issues/66#issuecomment-172811074","issue_url":"https://api.github.com/repos/ARMmbed/greentea/issues/66","id":172811074,"user":{"login":"mazimkhan","id":13520658,"avatar_url":"https://avatars.githubusercontent.com/u/13520658?v=3","gravatar_id":"","url":"https://api.github.com/users/mazimkhan","html_url":"https://github.com/mazimkhan","followers_url":"https://api.github.com/users/mazimkhan/followers","following_url":"https://api.github.com/users/mazimkhan/following{/other_user}","gists_url":"https://api.github.com/users/mazimkhan/gists{/gist_id}","starred_url":"https://api.github.com/users/mazimkhan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mazimkhan/subscriptions","organizations_url":"https://api.github.com/users/mazimkhan/orgs","repos_url":"https://api.github.com/users/mazimkhan/repos","events_url":"https://api.github.com/users/mazimkhan/events{/privacy}","received_events_url":"https://api.github.com/users/mazimkhan/received_events","type":"User","site_admin":false},"created_at":"2016-01-19T10:29:12Z","updated_at":"2016-01-19T10:29:12Z","body":"Comment on grouping of tests. The example mbedgt test report gives idea:\r\n1. That ```rtc``` and ```cstring``` binaries will have subtests. Is it true.\r\n2. Does it also mean that <repo>/test/<test name>/main.cpp will have sub tests?\r\n3. Is the one binary multiple tests solution only for subtests?\r\n\r\nMy understanding of one binary having multiple tests was that one binary can have all or a group of tests based on the target memory size? and it will use clitest interface for test indexing, start and verdict."}}