{"action":"opened","number":168,"pull_request":{"url":"https://api.github.com/repos/jaegertracing/jaeger-operator/pulls/168","id":238795815,"node_id":"MDExOlB1bGxSZXF1ZXN0MjM4Nzk1ODE1","html_url":"https://github.com/jaegertracing/jaeger-operator/pull/168","diff_url":"https://github.com/jaegertracing/jaeger-operator/pull/168.diff","patch_url":"https://github.com/jaegertracing/jaeger-operator/pull/168.patch","issue_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/issues/168","number":168,"state":"open","locked":false,"title":"WIP: Add ingester (and kafka) support","user":{"login":"objectiser","id":164562,"node_id":"MDQ6VXNlcjE2NDU2Mg==","avatar_url":"https://avatars3.githubusercontent.com/u/164562?v=4","gravatar_id":"","url":"https://api.github.com/users/objectiser","html_url":"https://github.com/objectiser","followers_url":"https://api.github.com/users/objectiser/followers","following_url":"https://api.github.com/users/objectiser/following{/other_user}","gists_url":"https://api.github.com/users/objectiser/gists{/gist_id}","starred_url":"https://api.github.com/users/objectiser/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/objectiser/subscriptions","organizations_url":"https://api.github.com/users/objectiser/orgs","repos_url":"https://api.github.com/users/objectiser/repos","events_url":"https://api.github.com/users/objectiser/events{/privacy}","received_events_url":"https://api.github.com/users/objectiser/received_events","type":"User","site_admin":false},"body":"This is an initial suggestion for how we can easily switch from a simple production deployment (e.g. collector and query using a common backend persistent storage) to one where Kafka is used as a buffer between collectors (publishing to Kafka) and ingesters (consuming from Kafka and storing in the specified storage).\r\n\r\nThe difference between the two configurations would be:\r\n```\r\napiVersion: io.jaegertracing/v1alpha1\r\nkind: Jaeger\r\nmetadata:\r\n  name: simple-prod\r\nspec:\r\n  strategy: production\r\n  storage:\r\n    type: elasticsearch\r\n    options:\r\n      es:\r\n        server-urls: http://elasticsearch:9200\r\n```\r\nand\r\n```\r\napiVersion: io.jaegertracing/v1alpha1\r\nkind: Jaeger\r\nmetadata:\r\n  name: simple-prod\r\nspec:\r\n  strategy: production\r\n  ingester:\r\n    options:\r\n      kafka:\r\n        brokers: 127.0.0.1:9092\r\n        topic: jaeger-spans\r\n  storage:\r\n    type: elasticsearch\r\n    options:\r\n      es:\r\n        server-urls: http://elasticsearch:9200\r\n```\r\n\r\nThis is a simple change to the configuration, i.e. defining the ingester options, but not necessarily clear this results in a more complex architecture.\r\n\r\nIssues to consider:\r\n* Both the collector and ingester would have `storage` definitions - whereas the CR only has one area to define these. We could list storage types, e.g. `type: elasticsearch, kafka` with options for both, but it is still not clear which should be used in collector and ingester.\r\n* We need an `ingester` node in the CR, equivalent to the other top level nodes, to be able to override the image - but these are just used to define information about the individual components. Not necessarily how those components collaborate with each other.\r\n\r\nSo the question is whether defining the ingester should imply using the `collector->kafka->ingester` approach and that the storage option therefore moves to the final component in the chain - i.e. ingester rather than collector.\r\n\r\nIf the existence of ingester options is not considered appropriate, then I would recommend we define more explicit strategies:\r\n* `allInOne` as now, single deployment with agent, collector and query\r\n* collector and query with shared storage\r\n* collector->kafka->ingester and query with shared storage\r\n\r\n(what we name the 2nd and 3rd strategies is another matter)\r\n\r\nI think this is actually the best approach, as it is clear what architecture the user wants, and the CR can then be verified to ensure it has all of the required information defined for each of the components. It also means that the `ingester` node is optional - as the kafka options could just be listed in the `storage.options`, but would still result in the ingester being configured.\r\n\r\nFeedback on the current PR and proposed change would be appreciated - however I might not work on it much until new year.\r\n\r\n\r\n\r\nSigned-off-by: Gary Brown <gary@brownuk.com>","created_at":"2018-12-14T17:48:13Z","updated_at":"2018-12-14T17:48:13Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/pulls/168/commits","review_comments_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/pulls/168/comments","review_comment_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/pulls/comments{/number}","comments_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/issues/168/comments","statuses_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/statuses/cd44d5e5ae8579d06bfab42d23122ffc7f5d15dd","head":{"label":"objectiser:ingester","ref":"ingester","sha":"cd44d5e5ae8579d06bfab42d23122ffc7f5d15dd","user":{"login":"objectiser","id":164562,"node_id":"MDQ6VXNlcjE2NDU2Mg==","avatar_url":"https://avatars3.githubusercontent.com/u/164562?v=4","gravatar_id":"","url":"https://api.github.com/users/objectiser","html_url":"https://github.com/objectiser","followers_url":"https://api.github.com/users/objectiser/followers","following_url":"https://api.github.com/users/objectiser/following{/other_user}","gists_url":"https://api.github.com/users/objectiser/gists{/gist_id}","starred_url":"https://api.github.com/users/objectiser/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/objectiser/subscriptions","organizations_url":"https://api.github.com/users/objectiser/orgs","repos_url":"https://api.github.com/users/objectiser/repos","events_url":"https://api.github.com/users/objectiser/events{/privacy}","received_events_url":"https://api.github.com/users/objectiser/received_events","type":"User","site_admin":false},"repo":{"id":148118534,"node_id":"MDEwOlJlcG9zaXRvcnkxNDgxMTg1MzQ=","name":"jaeger-operator","full_name":"objectiser/jaeger-operator","private":false,"owner":{"login":"objectiser","id":164562,"node_id":"MDQ6VXNlcjE2NDU2Mg==","avatar_url":"https://avatars3.githubusercontent.com/u/164562?v=4","gravatar_id":"","url":"https://api.github.com/users/objectiser","html_url":"https://github.com/objectiser","followers_url":"https://api.github.com/users/objectiser/followers","following_url":"https://api.github.com/users/objectiser/following{/other_user}","gists_url":"https://api.github.com/users/objectiser/gists{/gist_id}","starred_url":"https://api.github.com/users/objectiser/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/objectiser/subscriptions","organizations_url":"https://api.github.com/users/objectiser/orgs","repos_url":"https://api.github.com/users/objectiser/repos","events_url":"https://api.github.com/users/objectiser/events{/privacy}","received_events_url":"https://api.github.com/users/objectiser/received_events","type":"User","site_admin":false},"html_url":"https://github.com/objectiser/jaeger-operator","description":"Jaeger Operator for Kubernetes","fork":true,"url":"https://api.github.com/repos/objectiser/jaeger-operator","forks_url":"https://api.github.com/repos/objectiser/jaeger-operator/forks","keys_url":"https://api.github.com/repos/objectiser/jaeger-operator/keys{/key_id}","collaborators_url":"https://api.github.com/repos/objectiser/jaeger-operator/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/objectiser/jaeger-operator/teams","hooks_url":"https://api.github.com/repos/objectiser/jaeger-operator/hooks","issue_events_url":"https://api.github.com/repos/objectiser/jaeger-operator/issues/events{/number}","events_url":"https://api.github.com/repos/objectiser/jaeger-operator/events","assignees_url":"https://api.github.com/repos/objectiser/jaeger-operator/assignees{/user}","branches_url":"https://api.github.com/repos/objectiser/jaeger-operator/branches{/branch}","tags_url":"https://api.github.com/repos/objectiser/jaeger-operator/tags","blobs_url":"https://api.github.com/repos/objectiser/jaeger-operator/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/objectiser/jaeger-operator/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/objectiser/jaeger-operator/git/refs{/sha}","trees_url":"https://api.github.com/repos/objectiser/jaeger-operator/git/trees{/sha}","statuses_url":"https://api.github.com/repos/objectiser/jaeger-operator/statuses/{sha}","languages_url":"https://api.github.com/repos/objectiser/jaeger-operator/languages","stargazers_url":"https://api.github.com/repos/objectiser/jaeger-operator/stargazers","contributors_url":"https://api.github.com/repos/objectiser/jaeger-operator/contributors","subscribers_url":"https://api.github.com/repos/objectiser/jaeger-operator/subscribers","subscription_url":"https://api.github.com/repos/objectiser/jaeger-operator/subscription","commits_url":"https://api.github.com/repos/objectiser/jaeger-operator/commits{/sha}","git_commits_url":"https://api.github.com/repos/objectiser/jaeger-operator/git/commits{/sha}","comments_url":"https://api.github.com/repos/objectiser/jaeger-operator/comments{/number}","issue_comment_url":"https://api.github.com/repos/objectiser/jaeger-operator/issues/comments{/number}","contents_url":"https://api.github.com/repos/objectiser/jaeger-operator/contents/{+path}","compare_url":"https://api.github.com/repos/objectiser/jaeger-operator/compare/{base}...{head}","merges_url":"https://api.github.com/repos/objectiser/jaeger-operator/merges","archive_url":"https://api.github.com/repos/objectiser/jaeger-operator/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/objectiser/jaeger-operator/downloads","issues_url":"https://api.github.com/repos/objectiser/jaeger-operator/issues{/number}","pulls_url":"https://api.github.com/repos/objectiser/jaeger-operator/pulls{/number}","milestones_url":"https://api.github.com/repos/objectiser/jaeger-operator/milestones{/number}","notifications_url":"https://api.github.com/repos/objectiser/jaeger-operator/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/objectiser/jaeger-operator/labels{/name}","releases_url":"https://api.github.com/repos/objectiser/jaeger-operator/releases{/id}","deployments_url":"https://api.github.com/repos/objectiser/jaeger-operator/deployments","created_at":"2018-09-10T07:47:45Z","updated_at":"2018-12-11T11:12:58Z","pushed_at":"2018-12-14T17:19:14Z","git_url":"git://github.com/objectiser/jaeger-operator.git","ssh_url":"git@github.com:objectiser/jaeger-operator.git","clone_url":"https://github.com/objectiser/jaeger-operator.git","svn_url":"https://github.com/objectiser/jaeger-operator","homepage":"","size":6259,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"jaegertracing:master","ref":"master","sha":"1e134beeb2409beed78d632b858a6691ed4f9cbf","user":{"login":"jaegertracing","id":28545596,"node_id":"MDEyOk9yZ2FuaXphdGlvbjI4NTQ1NTk2","avatar_url":"https://avatars1.githubusercontent.com/u/28545596?v=4","gravatar_id":"","url":"https://api.github.com/users/jaegertracing","html_url":"https://github.com/jaegertracing","followers_url":"https://api.github.com/users/jaegertracing/followers","following_url":"https://api.github.com/users/jaegertracing/following{/other_user}","gists_url":"https://api.github.com/users/jaegertracing/gists{/gist_id}","starred_url":"https://api.github.com/users/jaegertracing/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jaegertracing/subscriptions","organizations_url":"https://api.github.com/users/jaegertracing/orgs","repos_url":"https://api.github.com/users/jaegertracing/repos","events_url":"https://api.github.com/users/jaegertracing/events{/privacy}","received_events_url":"https://api.github.com/users/jaegertracing/received_events","type":"Organization","site_admin":false},"repo":{"id":146413101,"node_id":"MDEwOlJlcG9zaXRvcnkxNDY0MTMxMDE=","name":"jaeger-operator","full_name":"jaegertracing/jaeger-operator","private":false,"owner":{"login":"jaegertracing","id":28545596,"node_id":"MDEyOk9yZ2FuaXphdGlvbjI4NTQ1NTk2","avatar_url":"https://avatars1.githubusercontent.com/u/28545596?v=4","gravatar_id":"","url":"https://api.github.com/users/jaegertracing","html_url":"https://github.com/jaegertracing","followers_url":"https://api.github.com/users/jaegertracing/followers","following_url":"https://api.github.com/users/jaegertracing/following{/other_user}","gists_url":"https://api.github.com/users/jaegertracing/gists{/gist_id}","starred_url":"https://api.github.com/users/jaegertracing/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jaegertracing/subscriptions","organizations_url":"https://api.github.com/users/jaegertracing/orgs","repos_url":"https://api.github.com/users/jaegertracing/repos","events_url":"https://api.github.com/users/jaegertracing/events{/privacy}","received_events_url":"https://api.github.com/users/jaegertracing/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/jaegertracing/jaeger-operator","description":"Jaeger Operator for Kubernetes","fork":false,"url":"https://api.github.com/repos/jaegertracing/jaeger-operator","forks_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/forks","keys_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/keys{/key_id}","collaborators_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/teams","hooks_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/hooks","issue_events_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/issues/events{/number}","events_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/events","assignees_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/assignees{/user}","branches_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/branches{/branch}","tags_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/tags","blobs_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/git/refs{/sha}","trees_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/git/trees{/sha}","statuses_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/statuses/{sha}","languages_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/languages","stargazers_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/stargazers","contributors_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/contributors","subscribers_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/subscribers","subscription_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/subscription","commits_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/commits{/sha}","git_commits_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/git/commits{/sha}","comments_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/comments{/number}","issue_comment_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/issues/comments{/number}","contents_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/contents/{+path}","compare_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/compare/{base}...{head}","merges_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/merges","archive_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/downloads","issues_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/issues{/number}","pulls_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/pulls{/number}","milestones_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/milestones{/number}","notifications_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/labels{/name}","releases_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/releases{/id}","deployments_url":"https://api.github.com/repos/jaegertracing/jaeger-operator/deployments","created_at":"2018-08-28T08:03:12Z","updated_at":"2018-12-12T17:40:04Z","pushed_at":"2018-12-12T17:55:31Z","git_url":"git://github.com/jaegertracing/jaeger-operator.git","ssh_url":"git@github.com:jaegertracing/jaeger-operator.git","clone_url":"https://github.com/jaegertracing/jaeger-operator.git","svn_url":"https://github.com/jaegertracing/jaeger-operator","homepage":"","size":6447,"stargazers_count":91,"watchers_count":91,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":22,"mirror_url":null,"archived":false,"open_issues_count":50,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":22,"open_issues":50,"watchers":91,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/jaegertracing/jaeger-operator/pulls/168"},"html":{"href":"https://github.com/jaegertracing/jaeger-operator/pull/168"},"issue":{"href":"https://api.github.com/repos/jaegertracing/jaeger-operator/issues/168"},"comments":{"href":"https://api.github.com/repos/jaegertracing/jaeger-operator/issues/168/comments"},"review_comments":{"href":"https://api.github.com/repos/jaegertracing/jaeger-operator/pulls/168/comments"},"review_comment":{"href":"https://api.github.com/repos/jaegertracing/jaeger-operator/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/jaegertracing/jaeger-operator/pulls/168/commits"},"statuses":{"href":"https://api.github.com/repos/jaegertracing/jaeger-operator/statuses/cd44d5e5ae8579d06bfab42d23122ffc7f5d15dd"}},"author_association":"CONTRIBUTOR","merged":false,"mergeable":null,"rebaseable":null,"mergeable_state":"unknown","merged_by":null,"comments":0,"review_comments":0,"maintainer_can_modify":true,"commits":1,"additions":707,"deletions":2,"changed_files":10}}
{"actor":{"display_login":"objectiser"}}