{"action":"created","comment":{"url":"https://api.github.com/repos/elastic/logstash/pulls/comments/65673197","id":65673197,"diff_hunk":"@@ -18,10 +18,128 @@ details on how to install it.\n \n _Inside the `qa` directory_\n \n-* Execute the command `bundle` this will pull the necessary dependencies in your environment.\n-* start your machines with `bundle exec test:setup`\n-* Run `rake test:ssh_config` to dump the ssh configuration to access the different vagrant machines, this will generate a file named `.vm_ssh_config` that is going to be used for the tests.\n-* Run `bundle exec rake test:acceptance:all` to run all acceptance test\n-  at once, there is also detailed tasks for platforms:\n- * `rake test:acceptance:debian` for debian platforms.\n- * `rake test:acceptance:centos` for centos platforms.\n+First of all execute the command `bundle` this will pull the necessary dependencies in your environment, after it this is the collection of task available for you:\n+\n+```\n+skywalker% rake -T\n+rake qa:acceptance:all              # Run all acceptance\n+rake qa:acceptance:debian           # Run acceptance test in debian machines\n+rake qa:acceptance:redhat           # Run acceptance test in redhat machines\n+rake qa:acceptance:single[machine]  # Run one single machine acceptance test\n+rake qa:acceptance:suse             # Run acceptance test in suse machines\n+rake qa:vm:halt[platform]           # Halt all VM's involved in the acceptance test round\n+rake qa:vm:setup[platform]          # Bootstrap all the VM's used for this tests\n+rake qa:vm:ssh_config               # Generate a valid ssh-config\n+```\n+\n+### How to run tests\n+\n+In this framework we're using ssh to connecto t acollection of Vagrant\n+machines, so first and most important is to generate a valid ssh config\n+file, this could be done running `rake qa:vm:ssh_config`. When this task\n+is finished a file named `.vm_ssh_config` will be generated with all the\n+necessary information to connect with the different machines.\n+\n+Now is time to run your test and to do that we have different options:\n+\n+* rake qa:acceptance:all              # Run all acceptance\n+* rake qa:acceptance:debian           # Run acceptance test in debian machines\n+* rake qa:acceptance:redhat           # Run acceptance test in redhat machines\n+* rake qa:acceptance:suse             # Run acceptance test in suse machines\n+* rake qa:acceptance:single[machine]  # Run one single machine acceptance test\n+\n+\n+## Architecture of the Framework\n+\n+If you wanna know more about how this framework works, here is your\n+section of information.\n+\n+### Directory structure\n+\n+* ```acceptance/``` here it goes all the specs definitions.\n+* ```config```  inside you can find all config files, for now only the\n+  platform definition.\n+* ```rspec``` here stay all framework parts necessary to get the test\n+  running, you will find the commands, the rspec matchers and a\n+collection of useful helpers for your test.\n+* ```sys``` a collection of bash scripts used to bootstrap the machines.\n+* ```vagrant``` classes and modules used to help us running vagrant.\n+\n+### The platform configuration file\n+\n+As noted in the config directory it life there the platforms.json, this\n+file is used to encapsulate the different platforms we test with.\n+Important bits here are:\n+\n+* under the ```latest``` keyword we've the latest released version of\n+  LS, this is necessary because we aim to test the upgrade scenario.\n+* inside the `platforms` key you will find the list of current available\n+  OS we tests with, this include the box name, their type and if they\n+have to go under specific bootstrap scripts (see ```specific: true ```\n+in the platform definition).\n+\n+This file is the one that you will use to know about differnt OS's\n+testes, add new ones, etc..\n+\n+### I want to add a test, whad should I do?\n+\n+To add a test you basically should start by the acceptance directory,\n+here you will find an already collection tests, most important locations\n+here are:\n+\n+* ```lib``` here is where the tests are living. If a test is not going\n+  to be reused it should be created here.\n+* ```shared_examples``` inside that directory should be living all tests\n+  that could be reused in different scenarios, like you can see the CLI\n+ones.\n+\n+and how is my tests going to look alike, might be your next question,\n+right? something like:\n+\n+```\n+  config = ServiceTester.configuration\n+  config.servers.each do |address|\n+    ##\n+    # ServiceTester::Artifact is the component used to interact with the\n+    # destination machineri and the one that keep the necessary logic\n+    # for it.\n+    ##\n+\n+    logstash = ServiceTester::Artifact.new(address, config.lookup[address])\n+\n+    ## your test code goes here.\n+  end\n+```\n+\n+this is important because as you know we test with different machines,\n+so the build out artifact will be the component necessary to run the\n+actions with the destination machine.\n+\n+but this is the main parts, to run your test you need the framework\n+located inside the ```rspec``` directory. Here you will find a\n+collection of commands, properly organized per operating system, that\n+will let you operate and get your tests done. But don't freek out, we\n+got all logic necessary to select the right one for your test.\n+\n+You most probably find here all you need already, but if not feel free to add it.\n+\n+FYI, this is how a command looks like:\n+\n+```\n+    def installed?(hosts, package)\n+      stdout = \"\"\n+      at(hosts, {in: :serial}) do |host|\n+        cmd = sudo_exec!(\"dpkg -s  #{package}\")\n+        stdout = cmd.stdout\n+      end\n+      stdout.match(/^Package: #{package}$/)\n+      stdout.match(/^Status: install ok installed$/)\n+  end\n+  ```\n+this is how we run operations and wrap them as ruby code.\n+\n+### Running a test (detailed level)\n+\n+There is also the possibitly to run your tests with more granularity by\n+using the `rspec` command, this will let you for example run a single\n+tests, a collection of them using filtering, etc. Check https://relishapp.com/rspec/rspec-core/v/3-4/docs/command-line for more details.","path":"qa/README.md","position":134,"original_position":135,"commit_id":"87627d937e0ed754ab9710c209fe8491d50ea941","original_commit_id":"cd3e75287fdfab7cfc232a445b3c87c732df6f20","user":{"login":"purbon","id":68540,"avatar_url":"https://avatars.githubusercontent.com/u/68540?v=3","gravatar_id":"","url":"https://api.github.com/users/purbon","html_url":"https://github.com/purbon","followers_url":"https://api.github.com/users/purbon/followers","following_url":"https://api.github.com/users/purbon/following{/other_user}","gists_url":"https://api.github.com/users/purbon/gists{/gist_id}","starred_url":"https://api.github.com/users/purbon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/purbon/subscriptions","organizations_url":"https://api.github.com/users/purbon/orgs","repos_url":"https://api.github.com/users/purbon/repos","events_url":"https://api.github.com/users/purbon/events{/privacy}","received_events_url":"https://api.github.com/users/purbon/received_events","type":"User","site_admin":false},"body":"++","created_at":"2016-06-03T08:32:48Z","updated_at":"2016-06-03T08:32:48Z","html_url":"https://github.com/elastic/logstash/pull/5436#discussion_r65673197","pull_request_url":"https://api.github.com/repos/elastic/logstash/pulls/5436","_links":{"self":{"href":"https://api.github.com/repos/elastic/logstash/pulls/comments/65673197"},"html":{"href":"https://github.com/elastic/logstash/pull/5436#discussion_r65673197"},"pull_request":{"href":"https://api.github.com/repos/elastic/logstash/pulls/5436"}}},"pull_request":{"url":"https://api.github.com/repos/elastic/logstash/pulls/5436","id":72374336,"html_url":"https://github.com/elastic/logstash/pull/5436","diff_url":"https://github.com/elastic/logstash/pull/5436.diff","patch_url":"https://github.com/elastic/logstash/pull/5436.patch","issue_url":"https://api.github.com/repos/elastic/logstash/issues/5436","number":5436,"state":"open","locked":false,"title":"Enhance QA acceptance testing docs and readmes","user":{"login":"purbon","id":68540,"avatar_url":"https://avatars.githubusercontent.com/u/68540?v=3","gravatar_id":"","url":"https://api.github.com/users/purbon","html_url":"https://github.com/purbon","followers_url":"https://api.github.com/users/purbon/followers","following_url":"https://api.github.com/users/purbon/following{/other_user}","gists_url":"https://api.github.com/users/purbon/gists{/gist_id}","starred_url":"https://api.github.com/users/purbon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/purbon/subscriptions","organizations_url":"https://api.github.com/users/purbon/orgs","repos_url":"https://api.github.com/users/purbon/repos","events_url":"https://api.github.com/users/purbon/events{/privacy}","received_events_url":"https://api.github.com/users/purbon/received_events","type":"User","site_admin":false},"body":"Basically this, more documentation and comments for the hole new QA acceptance testing framework.","created_at":"2016-06-02T16:10:59Z","updated_at":"2016-06-03T08:32:48Z","closed_at":null,"merged_at":null,"merge_commit_sha":"bea97592d61dd0518460c168c04616a0b4b05ff9","assignee":{"login":"suyograo","id":1595958,"avatar_url":"https://avatars.githubusercontent.com/u/1595958?v=3","gravatar_id":"","url":"https://api.github.com/users/suyograo","html_url":"https://github.com/suyograo","followers_url":"https://api.github.com/users/suyograo/followers","following_url":"https://api.github.com/users/suyograo/following{/other_user}","gists_url":"https://api.github.com/users/suyograo/gists{/gist_id}","starred_url":"https://api.github.com/users/suyograo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/suyograo/subscriptions","organizations_url":"https://api.github.com/users/suyograo/orgs","repos_url":"https://api.github.com/users/suyograo/repos","events_url":"https://api.github.com/users/suyograo/events{/privacy}","received_events_url":"https://api.github.com/users/suyograo/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/elastic/logstash/pulls/5436/commits","review_comments_url":"https://api.github.com/repos/elastic/logstash/pulls/5436/comments","review_comment_url":"https://api.github.com/repos/elastic/logstash/pulls/comments{/number}","comments_url":"https://api.github.com/repos/elastic/logstash/issues/5436/comments","statuses_url":"https://api.github.com/repos/elastic/logstash/statuses/87627d937e0ed754ab9710c209fe8491d50ea941","head":{"label":"purbon:fix/enhance_qa_readme","ref":"fix/enhance_qa_readme","sha":"87627d937e0ed754ab9710c209fe8491d50ea941","user":{"login":"purbon","id":68540,"avatar_url":"https://avatars.githubusercontent.com/u/68540?v=3","gravatar_id":"","url":"https://api.github.com/users/purbon","html_url":"https://github.com/purbon","followers_url":"https://api.github.com/users/purbon/followers","following_url":"https://api.github.com/users/purbon/following{/other_user}","gists_url":"https://api.github.com/users/purbon/gists{/gist_id}","starred_url":"https://api.github.com/users/purbon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/purbon/subscriptions","organizations_url":"https://api.github.com/users/purbon/orgs","repos_url":"https://api.github.com/users/purbon/repos","events_url":"https://api.github.com/users/purbon/events{/privacy}","received_events_url":"https://api.github.com/users/purbon/received_events","type":"User","site_admin":false},"repo":{"id":26748658,"name":"logstash","full_name":"purbon/logstash","owner":{"login":"purbon","id":68540,"avatar_url":"https://avatars.githubusercontent.com/u/68540?v=3","gravatar_id":"","url":"https://api.github.com/users/purbon","html_url":"https://github.com/purbon","followers_url":"https://api.github.com/users/purbon/followers","following_url":"https://api.github.com/users/purbon/following{/other_user}","gists_url":"https://api.github.com/users/purbon/gists{/gist_id}","starred_url":"https://api.github.com/users/purbon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/purbon/subscriptions","organizations_url":"https://api.github.com/users/purbon/orgs","repos_url":"https://api.github.com/users/purbon/repos","events_url":"https://api.github.com/users/purbon/events{/privacy}","received_events_url":"https://api.github.com/users/purbon/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/purbon/logstash","description":"logstash - logs/event transport, processing, management, search.","fork":true,"url":"https://api.github.com/repos/purbon/logstash","forks_url":"https://api.github.com/repos/purbon/logstash/forks","keys_url":"https://api.github.com/repos/purbon/logstash/keys{/key_id}","collaborators_url":"https://api.github.com/repos/purbon/logstash/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/purbon/logstash/teams","hooks_url":"https://api.github.com/repos/purbon/logstash/hooks","issue_events_url":"https://api.github.com/repos/purbon/logstash/issues/events{/number}","events_url":"https://api.github.com/repos/purbon/logstash/events","assignees_url":"https://api.github.com/repos/purbon/logstash/assignees{/user}","branches_url":"https://api.github.com/repos/purbon/logstash/branches{/branch}","tags_url":"https://api.github.com/repos/purbon/logstash/tags","blobs_url":"https://api.github.com/repos/purbon/logstash/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/purbon/logstash/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/purbon/logstash/git/refs{/sha}","trees_url":"https://api.github.com/repos/purbon/logstash/git/trees{/sha}","statuses_url":"https://api.github.com/repos/purbon/logstash/statuses/{sha}","languages_url":"https://api.github.com/repos/purbon/logstash/languages","stargazers_url":"https://api.github.com/repos/purbon/logstash/stargazers","contributors_url":"https://api.github.com/repos/purbon/logstash/contributors","subscribers_url":"https://api.github.com/repos/purbon/logstash/subscribers","subscription_url":"https://api.github.com/repos/purbon/logstash/subscription","commits_url":"https://api.github.com/repos/purbon/logstash/commits{/sha}","git_commits_url":"https://api.github.com/repos/purbon/logstash/git/commits{/sha}","comments_url":"https://api.github.com/repos/purbon/logstash/comments{/number}","issue_comment_url":"https://api.github.com/repos/purbon/logstash/issues/comments{/number}","contents_url":"https://api.github.com/repos/purbon/logstash/contents/{+path}","compare_url":"https://api.github.com/repos/purbon/logstash/compare/{base}...{head}","merges_url":"https://api.github.com/repos/purbon/logstash/merges","archive_url":"https://api.github.com/repos/purbon/logstash/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/purbon/logstash/downloads","issues_url":"https://api.github.com/repos/purbon/logstash/issues{/number}","pulls_url":"https://api.github.com/repos/purbon/logstash/pulls{/number}","milestones_url":"https://api.github.com/repos/purbon/logstash/milestones{/number}","notifications_url":"https://api.github.com/repos/purbon/logstash/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/purbon/logstash/labels{/name}","releases_url":"https://api.github.com/repos/purbon/logstash/releases{/id}","deployments_url":"https://api.github.com/repos/purbon/logstash/deployments","created_at":"2014-11-17T09:01:27Z","updated_at":"2016-01-11T14:38:17Z","pushed_at":"2016-06-03T08:26:47Z","git_url":"git://github.com/purbon/logstash.git","ssh_url":"git@github.com:purbon/logstash.git","clone_url":"https://github.com/purbon/logstash.git","svn_url":"https://github.com/purbon/logstash","homepage":"http://logstash.net/","size":74831,"stargazers_count":0,"watchers_count":0,"language":"Ruby","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"elastic:master","ref":"master","sha":"afce9e74ff738da3ecb79e79f8ce2068e94d33c9","user":{"login":"elastic","id":6764390,"avatar_url":"https://avatars.githubusercontent.com/u/6764390?v=3","gravatar_id":"","url":"https://api.github.com/users/elastic","html_url":"https://github.com/elastic","followers_url":"https://api.github.com/users/elastic/followers","following_url":"https://api.github.com/users/elastic/following{/other_user}","gists_url":"https://api.github.com/users/elastic/gists{/gist_id}","starred_url":"https://api.github.com/users/elastic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elastic/subscriptions","organizations_url":"https://api.github.com/users/elastic/orgs","repos_url":"https://api.github.com/users/elastic/repos","events_url":"https://api.github.com/users/elastic/events{/privacy}","received_events_url":"https://api.github.com/users/elastic/received_events","type":"Organization","site_admin":false},"repo":{"id":1090311,"name":"logstash","full_name":"elastic/logstash","owner":{"login":"elastic","id":6764390,"avatar_url":"https://avatars.githubusercontent.com/u/6764390?v=3","gravatar_id":"","url":"https://api.github.com/users/elastic","html_url":"https://github.com/elastic","followers_url":"https://api.github.com/users/elastic/followers","following_url":"https://api.github.com/users/elastic/following{/other_user}","gists_url":"https://api.github.com/users/elastic/gists{/gist_id}","starred_url":"https://api.github.com/users/elastic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elastic/subscriptions","organizations_url":"https://api.github.com/users/elastic/orgs","repos_url":"https://api.github.com/users/elastic/repos","events_url":"https://api.github.com/users/elastic/events{/privacy}","received_events_url":"https://api.github.com/users/elastic/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/elastic/logstash","description":"logstash - transport and process your logs, events, or other data","fork":false,"url":"https://api.github.com/repos/elastic/logstash","forks_url":"https://api.github.com/repos/elastic/logstash/forks","keys_url":"https://api.github.com/repos/elastic/logstash/keys{/key_id}","collaborators_url":"https://api.github.com/repos/elastic/logstash/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/elastic/logstash/teams","hooks_url":"https://api.github.com/repos/elastic/logstash/hooks","issue_events_url":"https://api.github.com/repos/elastic/logstash/issues/events{/number}","events_url":"https://api.github.com/repos/elastic/logstash/events","assignees_url":"https://api.github.com/repos/elastic/logstash/assignees{/user}","branches_url":"https://api.github.com/repos/elastic/logstash/branches{/branch}","tags_url":"https://api.github.com/repos/elastic/logstash/tags","blobs_url":"https://api.github.com/repos/elastic/logstash/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/elastic/logstash/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/elastic/logstash/git/refs{/sha}","trees_url":"https://api.github.com/repos/elastic/logstash/git/trees{/sha}","statuses_url":"https://api.github.com/repos/elastic/logstash/statuses/{sha}","languages_url":"https://api.github.com/repos/elastic/logstash/languages","stargazers_url":"https://api.github.com/repos/elastic/logstash/stargazers","contributors_url":"https://api.github.com/repos/elastic/logstash/contributors","subscribers_url":"https://api.github.com/repos/elastic/logstash/subscribers","subscription_url":"https://api.github.com/repos/elastic/logstash/subscription","commits_url":"https://api.github.com/repos/elastic/logstash/commits{/sha}","git_commits_url":"https://api.github.com/repos/elastic/logstash/git/commits{/sha}","comments_url":"https://api.github.com/repos/elastic/logstash/comments{/number}","issue_comment_url":"https://api.github.com/repos/elastic/logstash/issues/comments{/number}","contents_url":"https://api.github.com/repos/elastic/logstash/contents/{+path}","compare_url":"https://api.github.com/repos/elastic/logstash/compare/{base}...{head}","merges_url":"https://api.github.com/repos/elastic/logstash/merges","archive_url":"https://api.github.com/repos/elastic/logstash/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/elastic/logstash/downloads","issues_url":"https://api.github.com/repos/elastic/logstash/issues{/number}","pulls_url":"https://api.github.com/repos/elastic/logstash/pulls{/number}","milestones_url":"https://api.github.com/repos/elastic/logstash/milestones{/number}","notifications_url":"https://api.github.com/repos/elastic/logstash/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/elastic/logstash/labels{/name}","releases_url":"https://api.github.com/repos/elastic/logstash/releases{/id}","deployments_url":"https://api.github.com/repos/elastic/logstash/deployments","created_at":"2010-11-18T01:31:50Z","updated_at":"2016-06-03T07:41:09Z","pushed_at":"2016-06-03T08:26:48Z","git_url":"git://github.com/elastic/logstash.git","ssh_url":"git@github.com:elastic/logstash.git","clone_url":"https://github.com/elastic/logstash.git","svn_url":"https://github.com/elastic/logstash","homepage":"https://www.elastic.co/products/logstash","size":88629,"stargazers_count":5951,"watchers_count":5951,"language":"Ruby","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1791,"mirror_url":null,"open_issues_count":735,"forks":1791,"open_issues":735,"watchers":5951,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/elastic/logstash/pulls/5436"},"html":{"href":"https://github.com/elastic/logstash/pull/5436"},"issue":{"href":"https://api.github.com/repos/elastic/logstash/issues/5436"},"comments":{"href":"https://api.github.com/repos/elastic/logstash/issues/5436/comments"},"review_comments":{"href":"https://api.github.com/repos/elastic/logstash/pulls/5436/comments"},"review_comment":{"href":"https://api.github.com/repos/elastic/logstash/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/elastic/logstash/pulls/5436/commits"},"statuses":{"href":"https://api.github.com/repos/elastic/logstash/statuses/87627d937e0ed754ab9710c209fe8491d50ea941"}}}}
{"actor":{"display_login":"purbon"}}