{"action":"created","issue":{"url":"https://api.github.com/repos/pipelinedb/pipeline_kafka/issues/57","repository_url":"https://api.github.com/repos/pipelinedb/pipeline_kafka","labels_url":"https://api.github.com/repos/pipelinedb/pipeline_kafka/issues/57/labels{/name}","comments_url":"https://api.github.com/repos/pipelinedb/pipeline_kafka/issues/57/comments","events_url":"https://api.github.com/repos/pipelinedb/pipeline_kafka/issues/57/events","html_url":"https://github.com/pipelinedb/pipeline_kafka/issues/57","id":196591435,"number":57,"title":"Improvements","user":{"login":"simplesteph","id":20851561,"avatar_url":"https://avatars.githubusercontent.com/u/20851561?v=3","gravatar_id":"","url":"https://api.github.com/users/simplesteph","html_url":"https://github.com/simplesteph","followers_url":"https://api.github.com/users/simplesteph/followers","following_url":"https://api.github.com/users/simplesteph/following{/other_user}","gists_url":"https://api.github.com/users/simplesteph/gists{/gist_id}","starred_url":"https://api.github.com/users/simplesteph/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/simplesteph/subscriptions","organizations_url":"https://api.github.com/users/simplesteph/orgs","repos_url":"https://api.github.com/users/simplesteph/repos","events_url":"https://api.github.com/users/simplesteph/events{/privacy}","received_events_url":"https://api.github.com/users/simplesteph/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-12-20T05:46:16Z","updated_at":"2016-12-21T01:42:50Z","closed_at":"2016-12-20T16:46:53Z","body":"Hi,\r\n\r\nWe're giving this module a go but I have the following feedback:\r\n\r\n- You're not using kafka boostrap brokers. This is now the standard way to discover all the brokers (only specify 2 or 3 and then the entire cluster is discovered). Adding all the brokers may not be a long term solution because brokers go up or down and IP / DNS are changing. See `pipeline_kafka.add_broker ( hostname text )`\r\n\r\n- This API is inherently flawed: `pipeline_kafka.consume_begin ( topic text, stream text, format := ‘text’, delimiter := E’\\t’, quote := NULL, escape := NULL, batchsize := 1000, maxbytes := 32000000, parallelism := 1, start_offset := NULL )`. Start offset per topic does not make sense. Offsets are relevant per partition, as admins can add partitions down the road to increase throughtput, and offset 100 in the new partition doesn't really mean anything compared to offset 100 in the other partition\r\n\r\n- Kafka 0.10.1 has now the ability to map timestamps to offsets, (see offsetForTimes at https://kafka.apache.org/0101/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html). It maps nicely to a where query with a \"kafka_timestamp\" and would be cool if pipelinedb incorporated that (you'll notice that the API returns a map of offsets per partition). \r\n\r\n- I like the fact you're storing offsets in Pipeline DB instead of Kafka / ZK because it will allow you to achieve exactly once semantics. If you plan on using ZK or Kafka for storing offsets then you're going to be either at least once or at most once, which may be an issue for a database\r\n\r\n- Avro support (integration with the Kafka schema registry). \r\n\r\nLet me know your thoughts,\r\n\r\nRegards,\r\nStephane\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/pipelinedb/pipeline_kafka/issues/comments/268410997","html_url":"https://github.com/pipelinedb/pipeline_kafka/issues/57#issuecomment-268410997","issue_url":"https://api.github.com/repos/pipelinedb/pipeline_kafka/issues/57","id":268410997,"user":{"login":"derekjn","id":384666,"avatar_url":"https://avatars.githubusercontent.com/u/384666?v=3","gravatar_id":"","url":"https://api.github.com/users/derekjn","html_url":"https://github.com/derekjn","followers_url":"https://api.github.com/users/derekjn/followers","following_url":"https://api.github.com/users/derekjn/following{/other_user}","gists_url":"https://api.github.com/users/derekjn/gists{/gist_id}","starred_url":"https://api.github.com/users/derekjn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/derekjn/subscriptions","organizations_url":"https://api.github.com/users/derekjn/orgs","repos_url":"https://api.github.com/users/derekjn/repos","events_url":"https://api.github.com/users/derekjn/events{/privacy}","received_events_url":"https://api.github.com/users/derekjn/received_events","type":"User","site_admin":false},"created_at":"2016-12-21T01:42:50Z","updated_at":"2016-12-21T01:42:50Z","body":"@simplesteph awesome, thank you! Let's continue the discussion on #42."}}
{"actor":{"display_login":"derekjn"}}