{"action":"created","issue":{"url":"https://api.github.com/repos/akka/reactive-kafka/issues/139","repository_url":"https://api.github.com/repos/akka/reactive-kafka","labels_url":"https://api.github.com/repos/akka/reactive-kafka/issues/139/labels{/name}","comments_url":"https://api.github.com/repos/akka/reactive-kafka/issues/139/comments","events_url":"https://api.github.com/repos/akka/reactive-kafka/issues/139/events","html_url":"https://github.com/akka/reactive-kafka/pull/139","id":151790658,"number":139,"title":"TopicPartition API","user":{"login":"13h3r","id":435085,"avatar_url":"https://avatars.githubusercontent.com/u/435085?v=3","gravatar_id":"","url":"https://api.github.com/users/13h3r","html_url":"https://github.com/13h3r","followers_url":"https://api.github.com/users/13h3r/followers","following_url":"https://api.github.com/users/13h3r/following{/other_user}","gists_url":"https://api.github.com/users/13h3r/gists{/gist_id}","starred_url":"https://api.github.com/users/13h3r/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/13h3r/subscriptions","organizations_url":"https://api.github.com/users/13h3r/orgs","repos_url":"https://api.github.com/users/13h3r/repos","events_url":"https://api.github.com/users/13h3r/events{/privacy}","received_events_url":"https://api.github.com/users/13h3r/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":24,"created_at":"2016-04-29T04:25:06Z","updated_at":"2016-06-17T12:52:06Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/akka/reactive-kafka/pulls/139","html_url":"https://github.com/akka/reactive-kafka/pull/139","diff_url":"https://github.com/akka/reactive-kafka/pull/139.diff","patch_url":"https://github.com/akka/reactive-kafka/pull/139.patch"},"body":"This PR is implementations of `TopicPartitionSource`  - `Source[(TopicPartition, Source[Message])]`.\r\n\r\nThere is some small stuff to do, but I want to your feedback on the implementation.\r\n\r\nOccasionally I reimplement all current consumer sources. I did it because the 0.11.M1 implementation contains several design problems:\r\n- Strong coupling between stage and kafka consumer. With this coupling, it is very hard to reuse one kafka consumer from multiple (sub)sources.\r\n- A complete mess in the `CommittableConsumer`. Patrik tried to extract some reusable stuff into a `GraphStageLogic`, but it is still the mess of lifecycle control, poll scheduling and other stuff.\r\n\r\nMy initial thought was `TopicPartitionSource` implementation on the top of the `GraphStage` as we did with `PlainSource` and `CommittableSource`. After quick prototyping, I figured out that hiding kafka consumer behind the `GraphStageLogic` and accessing it via callbacks producing very much boilerplate. I extract kafka consumer with all scheduling stuff and provide an asynchronous interface via `Future`s, but such interface produces a lot of boilerplate in `GraphStageLogic` too. So I decide to move away from `GraphStage` to `ActorPublisher` to be able to communicate with kafka consumer in an asynchronous way.\r\n\r\nAnother challenge is how to provide commit support on top of `PlainSource` without exposing much from it. Having kafka consumer decoupled from stage allows us to pass it to the `CommittableMessage` and build `CommittableMessage` on to of any `ConsumerMessage`.\r\n\r\nWhat we have now:\r\n- `KafkaConsumerActor` as a low-level abstraction on the top of KafkaConsumer\r\n- `PlainSource`, `CommittableSource` and `TopicPatitionSource` implemented with `KafkaConsumerActor`\r\n- `MessageBuilder` trait with `PlainMessageBuilder` and `CommittableMessageBuilder` implementations.\r\n\r\nFeel free to comment the idea and the implementation.\r\n"},"comment":{"url":"https://api.github.com/repos/akka/reactive-kafka/issues/comments/226760825","html_url":"https://github.com/akka/reactive-kafka/pull/139#issuecomment-226760825","issue_url":"https://api.github.com/repos/akka/reactive-kafka/issues/139","id":226760825,"user":{"login":"13h3r","id":435085,"avatar_url":"https://avatars.githubusercontent.com/u/435085?v=3","gravatar_id":"","url":"https://api.github.com/users/13h3r","html_url":"https://github.com/13h3r","followers_url":"https://api.github.com/users/13h3r/followers","following_url":"https://api.github.com/users/13h3r/following{/other_user}","gists_url":"https://api.github.com/users/13h3r/gists{/gist_id}","starred_url":"https://api.github.com/users/13h3r/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/13h3r/subscriptions","organizations_url":"https://api.github.com/users/13h3r/orgs","repos_url":"https://api.github.com/users/13h3r/repos","events_url":"https://api.github.com/users/13h3r/events{/privacy}","received_events_url":"https://api.github.com/users/13h3r/received_events","type":"User","site_admin":false},"created_at":"2016-06-17T12:52:06Z","updated_at":"2016-06-17T12:52:06Z","body":"Could you please review? // @patriknw @drewhk @kciesielski \r\n\r\nThings that not done:\r\n- docs and examples (will be done after we agreed with implementation)\r\n- tests (will be done in separate PR)"}}
{"actor":{"display_login":"13h3r"}}