{"action":"opened","number":178,"pull_request":{"url":"https://api.github.com/repos/Juniper/jsnapy/pulls/178","id":87597982,"html_url":"https://github.com/Juniper/jsnapy/pull/178","diff_url":"https://github.com/Juniper/jsnapy/pull/178.diff","patch_url":"https://github.com/Juniper/jsnapy/pull/178.patch","issue_url":"https://api.github.com/repos/Juniper/jsnapy/issues/178","number":178,"state":"open","locked":false,"title":"Operators for chained tests -- Fix #46","user":{"login":"eeishaan","id":4725770,"avatar_url":"https://avatars.githubusercontent.com/u/4725770?v=3","gravatar_id":"","url":"https://api.github.com/users/eeishaan","html_url":"https://github.com/eeishaan","followers_url":"https://api.github.com/users/eeishaan/followers","following_url":"https://api.github.com/users/eeishaan/following{/other_user}","gists_url":"https://api.github.com/users/eeishaan/gists{/gist_id}","starred_url":"https://api.github.com/users/eeishaan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eeishaan/subscriptions","organizations_url":"https://api.github.com/users/eeishaan/orgs","repos_url":"https://api.github.com/users/eeishaan/repos","events_url":"https://api.github.com/users/eeishaan/events{/privacy}","received_events_url":"https://api.github.com/users/eeishaan/received_events","type":"User","site_admin":false},"body":"Fix #46 \r\n\r\nGlossary:\r\n`ETC: Elementary test condition`\r\n\r\nAs proposed [here](https://github.com/Juniper/jsnapy/issues/46#issuecomment-241703311), the grammar is as follows \r\n```\r\ntests:\r\n          - exp\r\n```\r\nwhere \r\n_exp_  ->  _catA-op_  _exp_ _exp_  | _catB-op_ _ETC_\r\n\r\n_catA-op_  -> _AND_ | _OR_   \r\n_catB-op_  -> _NOT_ | _epsilon_\r\n\r\nThis grammar translates to the following example in yaml syntax:\r\n\r\n```yaml\r\ntests:\r\n- OR:\r\n    - AND:\r\n        - not-equal: last-state,Idle       # element in which test is performed\r\n                err: \"Test Failed!! last state is <{{post['last-state']}}>\"\r\n                info: \"Test succeeded!! last state is not equal to idle, it is: <{{post['last-state']}}>\"\r\n\r\n        - NOT:\r\n            - all-same: flap-count\r\n                err: \"Test Failed!!! flap count are not all same!!, it is <{{post['flap-count']}}> \"\r\n                info: \"Test Succeeded!! flap count are all same, it is now <{{post['flap-count']}}>!!!\"\r\n\r\n   - is-equal: flap-count, 0\r\n            err: \"Test Failed!!! flap count is not equal to 0, it is: <{{post['flap-count']}}> \"\r\n            info: \"Test Succeeded!! flap count is equal to <{{post['flap-count']}}> !!\"\r\n```\r\n\r\n\r\n`AND, OR` are binary operators and `NOT` is a unary operator.\r\nIf any deviation is observed, for ex, passing more than one ETC to a unary operator or passing one ETC to a binary operator, the parser will simply ignore that instance and continue evaluating remaining sub-expressions.\r\nIf all sub-expressions are skipped then no ETCs are executed. For module version, this means that Operator.result will now be `None` instead of earlier `True`.\r\n\r\nChanges to reporting:\r\nReporting works exactly as previously, nothing is removed only new things are added.\r\nNow, each test's status is also shown in the final report.\r\nFor module version, one can access the new Operator.result_dict dictionary (`key => test_name, value=> result[True/False/None])` to access the result of each test. User should ensure that test names in a test suite remain unique. If they repeat the result of older instance gets overwritten.\r\nAlso please note that one should not confuse the no. of test cases passed or fail reported at the end with the no. of sub-expressions failed or passed. \r\n\r\nExample:\r\n\r\ntest.yml:\r\n```yaml\r\ntests_include:\r\n  - test_command_version\r\n\r\ntest_command_version:\r\n  - command: show bgp neighbor\r\n  - iterate:\r\n      xpath: '/bgp-information/bgp-peer'\r\n      id: peer-address\r\n      tests:\r\n          - OR:\r\n            - NOT:\r\n                - contains: //bgp-options, LogUpDown\r\n            - all-same: flap-count\r\n              err: \"Test Failed!!! bgp-peer {{id_0}}!!! has mismatched flap count {{post['flap-count']}}\"\r\n              info: \"Test Succeeded!! flap count is {{post['flap-count']}} for bgp-peer {{id_0}}!!!\"\r\n          - AND:\r\n              - NOT:\r\n                  - contains: //bgp-options, LogUpDown\r\n                  - contains: //bgp-options, LogUpDown\r\n              - is-equal: local-address, unspecified\r\n```\r\n\r\nOutput:\r\n```bash\r\njnpr.jsnapy.check: INFO: *************************** Device: 10.216.193.114 ***************************\r\njnpr.jsnapy.check: INFO: Tests Included: test_command_version \r\njnpr.jsnapy.check: INFO: ************************* Command: show bgp neighbor *************************\r\njnpr.jsnapy.operator: INFO: PASS | All \"//bgp-options\" contains LogUpDown [ 9 matched ]\r\njnpr.jsnapy.operator: INFO: PASS | Value of all \"flap-count\" at xpath \"/bgp-information/bgp-peer\" is same [ 3 matched ]\r\njnpr.jsnapy.check: INFO: ERROR!!! Malformed sub-expression\r\njnpr.jsnapy.operator: INFO: PASS | All \"local-address\" is equal to \"unspecified\" [ 3 matched ]\r\njnpr.jsnapy.operator: INFO: ------------------------------- Final Result!! -------------------------------\r\njnpr.jsnapy.operator: INFO: test_command_version : Passed\r\njnpr.jsnapy.operator: INFO: Total No of tests passed: 3\r\njnpr.jsnapy.operator: INFO: Total No of tests failed: 0 \r\njnpr.jsnapy.operator: INFO: Overall Tests passed!!! \r\n```\r\n\r\nNotice the error produced due to the malformed `NOT` sub-expression. The parser skipped the incorrect part and executed the rest ETCs and sub-expressions. As 3 of the ETCs were passed, the no of tests passed is reported to be so.\r\n\r\nOne thing to note: In cases where during the course of sub-expression evaluation, the number of valid sub-expressions drops below two for any of the binary operator, the operator will be rendered ineffective as \r\n```yaml\r\nAND:\r\n      - is-equal: local-address, unspecified\r\n``` \r\nis meaningless. The ETC in question ( `- is-equal: local-address, unspecified` here ) will then become part of list of the parent operator.\r\n","created_at":"2016-10-01T10:07:12Z","updated_at":"2016-10-01T10:07:12Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/Juniper/jsnapy/pulls/178/commits","review_comments_url":"https://api.github.com/repos/Juniper/jsnapy/pulls/178/comments","review_comment_url":"https://api.github.com/repos/Juniper/jsnapy/pulls/comments{/number}","comments_url":"https://api.github.com/repos/Juniper/jsnapy/issues/178/comments","statuses_url":"https://api.github.com/repos/Juniper/jsnapy/statuses/0a550f17488e459108441bcc8bf0153852642e8a","head":{"label":"eeishaan:chained-tests-no-conflict","ref":"chained-tests-no-conflict","sha":"0a550f17488e459108441bcc8bf0153852642e8a","user":{"login":"eeishaan","id":4725770,"avatar_url":"https://avatars.githubusercontent.com/u/4725770?v=3","gravatar_id":"","url":"https://api.github.com/users/eeishaan","html_url":"https://github.com/eeishaan","followers_url":"https://api.github.com/users/eeishaan/followers","following_url":"https://api.github.com/users/eeishaan/following{/other_user}","gists_url":"https://api.github.com/users/eeishaan/gists{/gist_id}","starred_url":"https://api.github.com/users/eeishaan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eeishaan/subscriptions","organizations_url":"https://api.github.com/users/eeishaan/orgs","repos_url":"https://api.github.com/users/eeishaan/repos","events_url":"https://api.github.com/users/eeishaan/events{/privacy}","received_events_url":"https://api.github.com/users/eeishaan/received_events","type":"User","site_admin":false},"repo":{"id":65788428,"name":"jsnapy","full_name":"eeishaan/jsnapy","owner":{"login":"eeishaan","id":4725770,"avatar_url":"https://avatars.githubusercontent.com/u/4725770?v=3","gravatar_id":"","url":"https://api.github.com/users/eeishaan","html_url":"https://github.com/eeishaan","followers_url":"https://api.github.com/users/eeishaan/followers","following_url":"https://api.github.com/users/eeishaan/following{/other_user}","gists_url":"https://api.github.com/users/eeishaan/gists{/gist_id}","starred_url":"https://api.github.com/users/eeishaan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eeishaan/subscriptions","organizations_url":"https://api.github.com/users/eeishaan/orgs","repos_url":"https://api.github.com/users/eeishaan/repos","events_url":"https://api.github.com/users/eeishaan/events{/privacy}","received_events_url":"https://api.github.com/users/eeishaan/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/eeishaan/jsnapy","description":"Python version of Junos Snapshot Administrator","fork":true,"url":"https://api.github.com/repos/eeishaan/jsnapy","forks_url":"https://api.github.com/repos/eeishaan/jsnapy/forks","keys_url":"https://api.github.com/repos/eeishaan/jsnapy/keys{/key_id}","collaborators_url":"https://api.github.com/repos/eeishaan/jsnapy/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/eeishaan/jsnapy/teams","hooks_url":"https://api.github.com/repos/eeishaan/jsnapy/hooks","issue_events_url":"https://api.github.com/repos/eeishaan/jsnapy/issues/events{/number}","events_url":"https://api.github.com/repos/eeishaan/jsnapy/events","assignees_url":"https://api.github.com/repos/eeishaan/jsnapy/assignees{/user}","branches_url":"https://api.github.com/repos/eeishaan/jsnapy/branches{/branch}","tags_url":"https://api.github.com/repos/eeishaan/jsnapy/tags","blobs_url":"https://api.github.com/repos/eeishaan/jsnapy/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/eeishaan/jsnapy/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/eeishaan/jsnapy/git/refs{/sha}","trees_url":"https://api.github.com/repos/eeishaan/jsnapy/git/trees{/sha}","statuses_url":"https://api.github.com/repos/eeishaan/jsnapy/statuses/{sha}","languages_url":"https://api.github.com/repos/eeishaan/jsnapy/languages","stargazers_url":"https://api.github.com/repos/eeishaan/jsnapy/stargazers","contributors_url":"https://api.github.com/repos/eeishaan/jsnapy/contributors","subscribers_url":"https://api.github.com/repos/eeishaan/jsnapy/subscribers","subscription_url":"https://api.github.com/repos/eeishaan/jsnapy/subscription","commits_url":"https://api.github.com/repos/eeishaan/jsnapy/commits{/sha}","git_commits_url":"https://api.github.com/repos/eeishaan/jsnapy/git/commits{/sha}","comments_url":"https://api.github.com/repos/eeishaan/jsnapy/comments{/number}","issue_comment_url":"https://api.github.com/repos/eeishaan/jsnapy/issues/comments{/number}","contents_url":"https://api.github.com/repos/eeishaan/jsnapy/contents/{+path}","compare_url":"https://api.github.com/repos/eeishaan/jsnapy/compare/{base}...{head}","merges_url":"https://api.github.com/repos/eeishaan/jsnapy/merges","archive_url":"https://api.github.com/repos/eeishaan/jsnapy/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/eeishaan/jsnapy/downloads","issues_url":"https://api.github.com/repos/eeishaan/jsnapy/issues{/number}","pulls_url":"https://api.github.com/repos/eeishaan/jsnapy/pulls{/number}","milestones_url":"https://api.github.com/repos/eeishaan/jsnapy/milestones{/number}","notifications_url":"https://api.github.com/repos/eeishaan/jsnapy/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/eeishaan/jsnapy/labels{/name}","releases_url":"https://api.github.com/repos/eeishaan/jsnapy/releases{/id}","deployments_url":"https://api.github.com/repos/eeishaan/jsnapy/deployments","created_at":"2016-08-16T04:49:46Z","updated_at":"2016-08-16T04:49:48Z","pushed_at":"2016-10-01T10:04:17Z","git_url":"git://github.com/eeishaan/jsnapy.git","ssh_url":"git@github.com:eeishaan/jsnapy.git","clone_url":"https://github.com/eeishaan/jsnapy.git","svn_url":"https://github.com/eeishaan/jsnapy","homepage":null,"size":1103,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"Juniper:master","ref":"master","sha":"6a9e6cb9e096978ae9a8e684c91aa0f6f1d75885","user":{"login":"Juniper","id":3935164,"avatar_url":"https://avatars.githubusercontent.com/u/3935164?v=3","gravatar_id":"","url":"https://api.github.com/users/Juniper","html_url":"https://github.com/Juniper","followers_url":"https://api.github.com/users/Juniper/followers","following_url":"https://api.github.com/users/Juniper/following{/other_user}","gists_url":"https://api.github.com/users/Juniper/gists{/gist_id}","starred_url":"https://api.github.com/users/Juniper/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Juniper/subscriptions","organizations_url":"https://api.github.com/users/Juniper/orgs","repos_url":"https://api.github.com/users/Juniper/repos","events_url":"https://api.github.com/users/Juniper/events{/privacy}","received_events_url":"https://api.github.com/users/Juniper/received_events","type":"Organization","site_admin":false},"repo":{"id":31305772,"name":"jsnapy","full_name":"Juniper/jsnapy","owner":{"login":"Juniper","id":3935164,"avatar_url":"https://avatars.githubusercontent.com/u/3935164?v=3","gravatar_id":"","url":"https://api.github.com/users/Juniper","html_url":"https://github.com/Juniper","followers_url":"https://api.github.com/users/Juniper/followers","following_url":"https://api.github.com/users/Juniper/following{/other_user}","gists_url":"https://api.github.com/users/Juniper/gists{/gist_id}","starred_url":"https://api.github.com/users/Juniper/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Juniper/subscriptions","organizations_url":"https://api.github.com/users/Juniper/orgs","repos_url":"https://api.github.com/users/Juniper/repos","events_url":"https://api.github.com/users/Juniper/events{/privacy}","received_events_url":"https://api.github.com/users/Juniper/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Juniper/jsnapy","description":"Python version of Junos Snapshot Administrator","fork":false,"url":"https://api.github.com/repos/Juniper/jsnapy","forks_url":"https://api.github.com/repos/Juniper/jsnapy/forks","keys_url":"https://api.github.com/repos/Juniper/jsnapy/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Juniper/jsnapy/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Juniper/jsnapy/teams","hooks_url":"https://api.github.com/repos/Juniper/jsnapy/hooks","issue_events_url":"https://api.github.com/repos/Juniper/jsnapy/issues/events{/number}","events_url":"https://api.github.com/repos/Juniper/jsnapy/events","assignees_url":"https://api.github.com/repos/Juniper/jsnapy/assignees{/user}","branches_url":"https://api.github.com/repos/Juniper/jsnapy/branches{/branch}","tags_url":"https://api.github.com/repos/Juniper/jsnapy/tags","blobs_url":"https://api.github.com/repos/Juniper/jsnapy/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Juniper/jsnapy/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Juniper/jsnapy/git/refs{/sha}","trees_url":"https://api.github.com/repos/Juniper/jsnapy/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Juniper/jsnapy/statuses/{sha}","languages_url":"https://api.github.com/repos/Juniper/jsnapy/languages","stargazers_url":"https://api.github.com/repos/Juniper/jsnapy/stargazers","contributors_url":"https://api.github.com/repos/Juniper/jsnapy/contributors","subscribers_url":"https://api.github.com/repos/Juniper/jsnapy/subscribers","subscription_url":"https://api.github.com/repos/Juniper/jsnapy/subscription","commits_url":"https://api.github.com/repos/Juniper/jsnapy/commits{/sha}","git_commits_url":"https://api.github.com/repos/Juniper/jsnapy/git/commits{/sha}","comments_url":"https://api.github.com/repos/Juniper/jsnapy/comments{/number}","issue_comment_url":"https://api.github.com/repos/Juniper/jsnapy/issues/comments{/number}","contents_url":"https://api.github.com/repos/Juniper/jsnapy/contents/{+path}","compare_url":"https://api.github.com/repos/Juniper/jsnapy/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Juniper/jsnapy/merges","archive_url":"https://api.github.com/repos/Juniper/jsnapy/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Juniper/jsnapy/downloads","issues_url":"https://api.github.com/repos/Juniper/jsnapy/issues{/number}","pulls_url":"https://api.github.com/repos/Juniper/jsnapy/pulls{/number}","milestones_url":"https://api.github.com/repos/Juniper/jsnapy/milestones{/number}","notifications_url":"https://api.github.com/repos/Juniper/jsnapy/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Juniper/jsnapy/labels{/name}","releases_url":"https://api.github.com/repos/Juniper/jsnapy/releases{/id}","deployments_url":"https://api.github.com/repos/Juniper/jsnapy/deployments","created_at":"2015-02-25T09:15:53Z","updated_at":"2016-09-20T15:13:37Z","pushed_at":"2016-10-01T10:07:13Z","git_url":"git://github.com/Juniper/jsnapy.git","ssh_url":"git@github.com:Juniper/jsnapy.git","clone_url":"https://github.com/Juniper/jsnapy.git","svn_url":"https://github.com/Juniper/jsnapy","homepage":null,"size":1133,"stargazers_count":16,"watchers_count":16,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":6,"mirror_url":null,"open_issues_count":27,"forks":6,"open_issues":27,"watchers":16,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/Juniper/jsnapy/pulls/178"},"html":{"href":"https://github.com/Juniper/jsnapy/pull/178"},"issue":{"href":"https://api.github.com/repos/Juniper/jsnapy/issues/178"},"comments":{"href":"https://api.github.com/repos/Juniper/jsnapy/issues/178/comments"},"review_comments":{"href":"https://api.github.com/repos/Juniper/jsnapy/pulls/178/comments"},"review_comment":{"href":"https://api.github.com/repos/Juniper/jsnapy/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/Juniper/jsnapy/pulls/178/commits"},"statuses":{"href":"https://api.github.com/repos/Juniper/jsnapy/statuses/0a550f17488e459108441bcc8bf0153852642e8a"}},"merged":false,"mergeable":null,"mergeable_state":"unknown","merged_by":null,"comments":0,"review_comments":0,"commits":8,"additions":628,"deletions":106,"changed_files":16}}
{"actor":{"display_login":"eeishaan"}}