{"action":"created","issue":{"url":"https://api.github.com/repos/confluentinc/kafka-connect-hdfs/issues/314","repository_url":"https://api.github.com/repos/confluentinc/kafka-connect-hdfs","labels_url":"https://api.github.com/repos/confluentinc/kafka-connect-hdfs/issues/314/labels{/name}","comments_url":"https://api.github.com/repos/confluentinc/kafka-connect-hdfs/issues/314/comments","events_url":"https://api.github.com/repos/confluentinc/kafka-connect-hdfs/issues/314/events","html_url":"https://github.com/confluentinc/kafka-connect-hdfs/issues/314","id":313047651,"number":314,"title":"Kafka connect set offset issues.","user":{"login":"rupeshmore85","id":2333080,"avatar_url":"https://avatars2.githubusercontent.com/u/2333080?v=4","gravatar_id":"","url":"https://api.github.com/users/rupeshmore85","html_url":"https://github.com/rupeshmore85","followers_url":"https://api.github.com/users/rupeshmore85/followers","following_url":"https://api.github.com/users/rupeshmore85/following{/other_user}","gists_url":"https://api.github.com/users/rupeshmore85/gists{/gist_id}","starred_url":"https://api.github.com/users/rupeshmore85/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rupeshmore85/subscriptions","organizations_url":"https://api.github.com/users/rupeshmore85/orgs","repos_url":"https://api.github.com/users/rupeshmore85/repos","events_url":"https://api.github.com/users/rupeshmore85/events{/privacy}","received_events_url":"https://api.github.com/users/rupeshmore85/received_events","type":"User","site_admin":false},"labels":[{"id":214023529,"url":"https://api.github.com/repos/confluentinc/kafka-connect-hdfs/labels/question","name":"question","color":"cc317c","default":true}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-04-10T18:54:08Z","updated_at":"2018-04-13T21:14:39Z","closed_at":"2018-04-10T20:23:56Z","author_association":"NONE","body":"We are doing a cutover from Camus to Kafka Connect(HDFS sink connectors). We are concerned about maintaining the kafka topic offset consistent.\r\n1. Using kafka-consumer-groups, I am able to set the kafka topic partition within the kafka connect consumer group to current offset between earliest and latest, to the last offset which Camus wrote + 1. \r\n2. After I rebalance the connector, the connectors do not start from the set offset instead it writes from the earliest offset.\r\n\r\nIs there a setting i need to do in kafka-connect so the connectors understand to writes in the hdfs file from the set offset rather than the earliest offset ?"},"comment":{"url":"https://api.github.com/repos/confluentinc/kafka-connect-hdfs/issues/comments/381262296","html_url":"https://github.com/confluentinc/kafka-connect-hdfs/issues/314#issuecomment-381262296","issue_url":"https://api.github.com/repos/confluentinc/kafka-connect-hdfs/issues/314","id":381262296,"user":{"login":"rupeshmore85","id":2333080,"avatar_url":"https://avatars2.githubusercontent.com/u/2333080?v=4","gravatar_id":"","url":"https://api.github.com/users/rupeshmore85","html_url":"https://github.com/rupeshmore85","followers_url":"https://api.github.com/users/rupeshmore85/followers","following_url":"https://api.github.com/users/rupeshmore85/following{/other_user}","gists_url":"https://api.github.com/users/rupeshmore85/gists{/gist_id}","starred_url":"https://api.github.com/users/rupeshmore85/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rupeshmore85/subscriptions","organizations_url":"https://api.github.com/users/rupeshmore85/orgs","repos_url":"https://api.github.com/users/rupeshmore85/repos","events_url":"https://api.github.com/users/rupeshmore85/events{/privacy}","received_events_url":"https://api.github.com/users/rupeshmore85/received_events","type":"User","site_admin":false},"created_at":"2018-04-13T21:14:38Z","updated_at":"2018-04-13T21:14:38Z","author_association":"NONE","body":"Thanks Randall for your quick response!\r\nI downloaded https://hub.docker.com/r/confluentinc/cp-kafka-connect/tags/ with tag name 4.0.1\r\nI see it was uploaded 2 days back. Does it have the changes fixed for #299 ?\r\n\r\nI downloaded the image and did reset of the offset of one of kafka topic partition within the connect consumer group to some value less than last commited hdfs offset. I restarted the same connector  but it did not start from the set value rather it continued from the last committed offset.\r\nI then deleted the kafka connect hdfs logs which connectors write as WAL logs and restarted the connectors again, this time it didnot continue from the last commited offset rather began to write from earliest offset.\r\nWe are using kafka 0.10 where our topics lie and rely on Connector 4.0.0 for connectors. Does the above test mean connectors use hdfs to store the last commited offset? If yes do i have to write code to set the offset in connector log? \r\nCan you please guide as i am stuck ?\r\nThank you for all your help!"}}
{"actor":{"display_login":"rupeshmore85"}}