{"action":"created","comment":{"url":"https://api.github.com/repos/elastic/beats/pulls/comments/240390197","pull_request_review_id":183397452,"id":240390197,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDI0MDM5MDE5Nw==","diff_hunk":"@@ -150,3 +145,45 @@ func (c *publishClient) Test(d testing.Driver) {\n func (c *publishClient) String() string {\n \treturn \"publish(\" + c.es.String() + \")\"\n }\n+\n+func (c *publishClient) bulkToProduction(params map[string]string, event publisher.Event, docType interface{}) error {\n+\taction := common.MapStr{\n+\t\t\"index\": common.MapStr{\n+\t\t\t\"_type\":    docType,\n+\t\t\t\"_index\":   \"\",\n+\t\t\t\"_routing\": nil,\n+\t\t},\n+\t}\n+\tdocument := report.Event{\n+\t\tTimestamp: event.Content.Timestamp,\n+\t\tFields:    event.Content.Fields,\n+\t}\n+\tbulk := [2]interface{}{action, document}\n+\n+\t// Currently one request per event is sent. Reason is that each event can contain different\n+\t// interval params and X-Pack requires to send the interval param.\n+\t_, err := c.es.BulkWith(\"_xpack\", \"monitoring\", params, nil, bulk[:])\n+\treturn err\n+}\n+\n+// TODO: figure out why this isn't actually indexing anything!\n+func (c *publishClient) bulkToMonitoring(event publisher.Event) error {\n+\taction := common.MapStr{\n+\t\t\"index\": common.MapStr{\n+\t\t\t\"_type\":    \"doc\",","path":"libbeat/monitoring/report/elasticsearch/client.go","position":68,"original_position":68,"commit_id":"4eb0c4e106ad4b420c7ac3ea026d00aec2378870","original_commit_id":"4eb0c4e106ad4b420c7ac3ea026d00aec2378870","user":{"login":"urso","id":75886,"node_id":"MDQ6VXNlcjc1ODg2","avatar_url":"https://avatars1.githubusercontent.com/u/75886?v=4","gravatar_id":"","url":"https://api.github.com/users/urso","html_url":"https://github.com/urso","followers_url":"https://api.github.com/users/urso/followers","following_url":"https://api.github.com/users/urso/following{/other_user}","gists_url":"https://api.github.com/users/urso/gists{/gist_id}","starred_url":"https://api.github.com/users/urso/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/urso/subscriptions","organizations_url":"https://api.github.com/users/urso/orgs","repos_url":"https://api.github.com/users/urso/repos","events_url":"https://api.github.com/users/urso/events{/privacy}","received_events_url":"https://api.github.com/users/urso/received_events","type":"User","site_admin":false},"body":"this one is annoying. If the target ES version is 7.0 we will have to remove the `_type` field.","created_at":"2018-12-10T21:38:12Z","updated_at":"2018-12-10T21:47:07Z","html_url":"https://github.com/elastic/beats/pull/9260#discussion_r240390197","pull_request_url":"https://api.github.com/repos/elastic/beats/pulls/9260","author_association":"COLLABORATOR","_links":{"self":{"href":"https://api.github.com/repos/elastic/beats/pulls/comments/240390197"},"html":{"href":"https://github.com/elastic/beats/pull/9260#discussion_r240390197"},"pull_request":{"href":"https://api.github.com/repos/elastic/beats/pulls/9260"}}},"pull_request":{"url":"https://api.github.com/repos/elastic/beats/pulls/9260","id":234129875,"node_id":"MDExOlB1bGxSZXF1ZXN0MjM0MTI5ODc1","html_url":"https://github.com/elastic/beats/pull/9260","diff_url":"https://github.com/elastic/beats/pull/9260.diff","patch_url":"https://github.com/elastic/beats/pull/9260.patch","issue_url":"https://api.github.com/repos/elastic/beats/issues/9260","number":9260,"state":"open","locked":false,"title":"[WIP] Support shipping directly to monitoring cluster","user":{"login":"ycombinator","id":51061,"node_id":"MDQ6VXNlcjUxMDYx","avatar_url":"https://avatars2.githubusercontent.com/u/51061?v=4","gravatar_id":"","url":"https://api.github.com/users/ycombinator","html_url":"https://github.com/ycombinator","followers_url":"https://api.github.com/users/ycombinator/followers","following_url":"https://api.github.com/users/ycombinator/following{/other_user}","gists_url":"https://api.github.com/users/ycombinator/gists{/gist_id}","starred_url":"https://api.github.com/users/ycombinator/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ycombinator/subscriptions","organizations_url":"https://api.github.com/users/ycombinator/orgs","repos_url":"https://api.github.com/users/ycombinator/repos","events_url":"https://api.github.com/users/ycombinator/events{/privacy}","received_events_url":"https://api.github.com/users/ycombinator/received_events","type":"User","site_admin":false},"body":"Currently all beats ship their x-pack monitoring data to the production Elasticsearch cluster by `POST`ing it to the custom `_xpack/monitoring/_bulk` Elasticsearch API endpoint.\r\n\r\nIn the future we want to support beats shipping their x-pack monitoring data directly to a monitoring Elasticsearch cluster (which _could_ be the same as their production cluster but doesn't necessarily have to be) by `POST`ing it to the regular `_bulk` Elasticsearch API endpoint.\r\n\r\nThis PR introduces this \"direct shipping\" feature. Concretely:\r\n\r\n* [x] It introduces a new class of settings named `monitoring.*` that will parallel the current `xpack.monitoring.*` settings. If the user has set `xpack.monitoring.*` the beat will continue to ship to the production cluster as it does today. However, if the user has set the `monitoring.*`, the beat will instead ship directly to the monitoring cluster.\r\n\r\n* [x] It modifies the Elastisearch reporter to decide where to ship the data (production cluster, `_xpack/monitoring/_bulk` API endpoint -or- monitoring cluster, `_bulk` API endpoint).\r\n* [ ] If data is being shipped directly to the monitoring cluster AND the beat is configured to use an `elasticsearch` output, somehow (TBD) the Elasticsearch reporter will query the cluster corresponding to the `elasticsearch` output and fetch it's `cluster_uuid`. It will then inject this `cluster_uuid` into the monitoring data event before publishing it.\r\n\r\n","created_at":"2018-11-28T02:48:03Z","updated_at":"2018-12-10T21:47:07Z","closed_at":null,"merged_at":null,"merge_commit_sha":"0563833e983e21a8155dbdc4a4a35a610d2538c6","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[{"id":1135236287,"node_id":"MDU6TGFiZWwxMTM1MjM2Mjg3","url":"https://api.github.com/repos/elastic/beats/labels/Stack%20monitoring","name":"Stack monitoring","color":"e1ea69","default":false},{"id":216798909,"node_id":"MDU6TGFiZWwyMTY3OTg5MDk=","url":"https://api.github.com/repos/elastic/beats/labels/in%20progress","name":"in progress","color":"207de5","default":false},{"id":294278853,"node_id":"MDU6TGFiZWwyOTQyNzg4NTM=","url":"https://api.github.com/repos/elastic/beats/labels/libbeat","name":"libbeat","color":"008000","default":false}],"milestone":null,"commits_url":"https://api.github.com/repos/elastic/beats/pulls/9260/commits","review_comments_url":"https://api.github.com/repos/elastic/beats/pulls/9260/comments","review_comment_url":"https://api.github.com/repos/elastic/beats/pulls/comments{/number}","comments_url":"https://api.github.com/repos/elastic/beats/issues/9260/comments","statuses_url":"https://api.github.com/repos/elastic/beats/statuses/4eb0c4e106ad4b420c7ac3ea026d00aec2378870","head":{"label":"ycombinator:direct-monitoring","ref":"direct-monitoring","sha":"4eb0c4e106ad4b420c7ac3ea026d00aec2378870","user":{"login":"ycombinator","id":51061,"node_id":"MDQ6VXNlcjUxMDYx","avatar_url":"https://avatars2.githubusercontent.com/u/51061?v=4","gravatar_id":"","url":"https://api.github.com/users/ycombinator","html_url":"https://github.com/ycombinator","followers_url":"https://api.github.com/users/ycombinator/followers","following_url":"https://api.github.com/users/ycombinator/following{/other_user}","gists_url":"https://api.github.com/users/ycombinator/gists{/gist_id}","starred_url":"https://api.github.com/users/ycombinator/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ycombinator/subscriptions","organizations_url":"https://api.github.com/users/ycombinator/orgs","repos_url":"https://api.github.com/users/ycombinator/repos","events_url":"https://api.github.com/users/ycombinator/events{/privacy}","received_events_url":"https://api.github.com/users/ycombinator/received_events","type":"User","site_admin":false},"repo":{"id":135076276,"node_id":"MDEwOlJlcG9zaXRvcnkxMzUwNzYyNzY=","name":"beats","full_name":"ycombinator/beats","private":false,"owner":{"login":"ycombinator","id":51061,"node_id":"MDQ6VXNlcjUxMDYx","avatar_url":"https://avatars2.githubusercontent.com/u/51061?v=4","gravatar_id":"","url":"https://api.github.com/users/ycombinator","html_url":"https://github.com/ycombinator","followers_url":"https://api.github.com/users/ycombinator/followers","following_url":"https://api.github.com/users/ycombinator/following{/other_user}","gists_url":"https://api.github.com/users/ycombinator/gists{/gist_id}","starred_url":"https://api.github.com/users/ycombinator/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ycombinator/subscriptions","organizations_url":"https://api.github.com/users/ycombinator/orgs","repos_url":"https://api.github.com/users/ycombinator/repos","events_url":"https://api.github.com/users/ycombinator/events{/privacy}","received_events_url":"https://api.github.com/users/ycombinator/received_events","type":"User","site_admin":false},"html_url":"https://github.com/ycombinator/beats","description":":tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash ","fork":true,"url":"https://api.github.com/repos/ycombinator/beats","forks_url":"https://api.github.com/repos/ycombinator/beats/forks","keys_url":"https://api.github.com/repos/ycombinator/beats/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ycombinator/beats/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ycombinator/beats/teams","hooks_url":"https://api.github.com/repos/ycombinator/beats/hooks","issue_events_url":"https://api.github.com/repos/ycombinator/beats/issues/events{/number}","events_url":"https://api.github.com/repos/ycombinator/beats/events","assignees_url":"https://api.github.com/repos/ycombinator/beats/assignees{/user}","branches_url":"https://api.github.com/repos/ycombinator/beats/branches{/branch}","tags_url":"https://api.github.com/repos/ycombinator/beats/tags","blobs_url":"https://api.github.com/repos/ycombinator/beats/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ycombinator/beats/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ycombinator/beats/git/refs{/sha}","trees_url":"https://api.github.com/repos/ycombinator/beats/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ycombinator/beats/statuses/{sha}","languages_url":"https://api.github.com/repos/ycombinator/beats/languages","stargazers_url":"https://api.github.com/repos/ycombinator/beats/stargazers","contributors_url":"https://api.github.com/repos/ycombinator/beats/contributors","subscribers_url":"https://api.github.com/repos/ycombinator/beats/subscribers","subscription_url":"https://api.github.com/repos/ycombinator/beats/subscription","commits_url":"https://api.github.com/repos/ycombinator/beats/commits{/sha}","git_commits_url":"https://api.github.com/repos/ycombinator/beats/git/commits{/sha}","comments_url":"https://api.github.com/repos/ycombinator/beats/comments{/number}","issue_comment_url":"https://api.github.com/repos/ycombinator/beats/issues/comments{/number}","contents_url":"https://api.github.com/repos/ycombinator/beats/contents/{+path}","compare_url":"https://api.github.com/repos/ycombinator/beats/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ycombinator/beats/merges","archive_url":"https://api.github.com/repos/ycombinator/beats/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ycombinator/beats/downloads","issues_url":"https://api.github.com/repos/ycombinator/beats/issues{/number}","pulls_url":"https://api.github.com/repos/ycombinator/beats/pulls{/number}","milestones_url":"https://api.github.com/repos/ycombinator/beats/milestones{/number}","notifications_url":"https://api.github.com/repos/ycombinator/beats/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ycombinator/beats/labels{/name}","releases_url":"https://api.github.com/repos/ycombinator/beats/releases{/id}","deployments_url":"https://api.github.com/repos/ycombinator/beats/deployments","created_at":"2018-05-27T19:41:46Z","updated_at":"2018-09-26T14:15:09Z","pushed_at":"2018-11-29T09:39:07Z","git_url":"git://github.com/ycombinator/beats.git","ssh_url":"git@github.com:ycombinator/beats.git","clone_url":"https://github.com/ycombinator/beats.git","svn_url":"https://github.com/ycombinator/beats","homepage":"https://www.elastic.co/products/beats","size":97097,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"other","name":"Other","spdx_id":"NOASSERTION","url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"elastic:master","ref":"master","sha":"1c1b7d7427a422ec9da0707b1830e9cb03837266","user":{"login":"elastic","id":6764390,"node_id":"MDEyOk9yZ2FuaXphdGlvbjY3NjQzOTA=","avatar_url":"https://avatars0.githubusercontent.com/u/6764390?v=4","gravatar_id":"","url":"https://api.github.com/users/elastic","html_url":"https://github.com/elastic","followers_url":"https://api.github.com/users/elastic/followers","following_url":"https://api.github.com/users/elastic/following{/other_user}","gists_url":"https://api.github.com/users/elastic/gists{/gist_id}","starred_url":"https://api.github.com/users/elastic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elastic/subscriptions","organizations_url":"https://api.github.com/users/elastic/orgs","repos_url":"https://api.github.com/users/elastic/repos","events_url":"https://api.github.com/users/elastic/events{/privacy}","received_events_url":"https://api.github.com/users/elastic/received_events","type":"Organization","site_admin":false},"repo":{"id":16554739,"node_id":"MDEwOlJlcG9zaXRvcnkxNjU1NDczOQ==","name":"beats","full_name":"elastic/beats","private":false,"owner":{"login":"elastic","id":6764390,"node_id":"MDEyOk9yZ2FuaXphdGlvbjY3NjQzOTA=","avatar_url":"https://avatars0.githubusercontent.com/u/6764390?v=4","gravatar_id":"","url":"https://api.github.com/users/elastic","html_url":"https://github.com/elastic","followers_url":"https://api.github.com/users/elastic/followers","following_url":"https://api.github.com/users/elastic/following{/other_user}","gists_url":"https://api.github.com/users/elastic/gists{/gist_id}","starred_url":"https://api.github.com/users/elastic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elastic/subscriptions","organizations_url":"https://api.github.com/users/elastic/orgs","repos_url":"https://api.github.com/users/elastic/repos","events_url":"https://api.github.com/users/elastic/events{/privacy}","received_events_url":"https://api.github.com/users/elastic/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/elastic/beats","description":":tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash ","fork":false,"url":"https://api.github.com/repos/elastic/beats","forks_url":"https://api.github.com/repos/elastic/beats/forks","keys_url":"https://api.github.com/repos/elastic/beats/keys{/key_id}","collaborators_url":"https://api.github.com/repos/elastic/beats/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/elastic/beats/teams","hooks_url":"https://api.github.com/repos/elastic/beats/hooks","issue_events_url":"https://api.github.com/repos/elastic/beats/issues/events{/number}","events_url":"https://api.github.com/repos/elastic/beats/events","assignees_url":"https://api.github.com/repos/elastic/beats/assignees{/user}","branches_url":"https://api.github.com/repos/elastic/beats/branches{/branch}","tags_url":"https://api.github.com/repos/elastic/beats/tags","blobs_url":"https://api.github.com/repos/elastic/beats/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/elastic/beats/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/elastic/beats/git/refs{/sha}","trees_url":"https://api.github.com/repos/elastic/beats/git/trees{/sha}","statuses_url":"https://api.github.com/repos/elastic/beats/statuses/{sha}","languages_url":"https://api.github.com/repos/elastic/beats/languages","stargazers_url":"https://api.github.com/repos/elastic/beats/stargazers","contributors_url":"https://api.github.com/repos/elastic/beats/contributors","subscribers_url":"https://api.github.com/repos/elastic/beats/subscribers","subscription_url":"https://api.github.com/repos/elastic/beats/subscription","commits_url":"https://api.github.com/repos/elastic/beats/commits{/sha}","git_commits_url":"https://api.github.com/repos/elastic/beats/git/commits{/sha}","comments_url":"https://api.github.com/repos/elastic/beats/comments{/number}","issue_comment_url":"https://api.github.com/repos/elastic/beats/issues/comments{/number}","contents_url":"https://api.github.com/repos/elastic/beats/contents/{+path}","compare_url":"https://api.github.com/repos/elastic/beats/compare/{base}...{head}","merges_url":"https://api.github.com/repos/elastic/beats/merges","archive_url":"https://api.github.com/repos/elastic/beats/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/elastic/beats/downloads","issues_url":"https://api.github.com/repos/elastic/beats/issues{/number}","pulls_url":"https://api.github.com/repos/elastic/beats/pulls{/number}","milestones_url":"https://api.github.com/repos/elastic/beats/milestones{/number}","notifications_url":"https://api.github.com/repos/elastic/beats/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/elastic/beats/labels{/name}","releases_url":"https://api.github.com/repos/elastic/beats/releases{/id}","deployments_url":"https://api.github.com/repos/elastic/beats/deployments","created_at":"2014-02-05T19:01:02Z","updated_at":"2018-12-10T21:29:29Z","pushed_at":"2018-12-10T21:45:47Z","git_url":"git://github.com/elastic/beats.git","ssh_url":"git@github.com:elastic/beats.git","clone_url":"https://github.com/elastic/beats.git","svn_url":"https://github.com/elastic/beats","homepage":"https://www.elastic.co/products/beats","size":106285,"stargazers_count":6527,"watchers_count":6527,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2083,"mirror_url":null,"archived":false,"open_issues_count":1122,"license":{"key":"other","name":"Other","spdx_id":"NOASSERTION","url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":2083,"open_issues":1122,"watchers":6527,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/elastic/beats/pulls/9260"},"html":{"href":"https://github.com/elastic/beats/pull/9260"},"issue":{"href":"https://api.github.com/repos/elastic/beats/issues/9260"},"comments":{"href":"https://api.github.com/repos/elastic/beats/issues/9260/comments"},"review_comments":{"href":"https://api.github.com/repos/elastic/beats/pulls/9260/comments"},"review_comment":{"href":"https://api.github.com/repos/elastic/beats/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/elastic/beats/pulls/9260/commits"},"statuses":{"href":"https://api.github.com/repos/elastic/beats/statuses/4eb0c4e106ad4b420c7ac3ea026d00aec2378870"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"urso"}}