{"action":"created","issue":{"url":"https://api.github.com/repos/Shopify/sarama/issues/720","repository_url":"https://api.github.com/repos/Shopify/sarama","labels_url":"https://api.github.com/repos/Shopify/sarama/issues/720/labels{/name}","comments_url":"https://api.github.com/repos/Shopify/sarama/issues/720/comments","events_url":"https://api.github.com/repos/Shopify/sarama/issues/720/events","html_url":"https://github.com/Shopify/sarama/issues/720","id":169990295,"number":720,"title":"Consumer with API version set to V0_10_0_0 fails to consume compressed topics","user":{"login":"dynamix","id":5939,"avatar_url":"https://avatars.githubusercontent.com/u/5939?v=3","gravatar_id":"","url":"https://api.github.com/users/dynamix","html_url":"https://github.com/dynamix","followers_url":"https://api.github.com/users/dynamix/followers","following_url":"https://api.github.com/users/dynamix/following{/other_user}","gists_url":"https://api.github.com/users/dynamix/gists{/gist_id}","starred_url":"https://api.github.com/users/dynamix/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dynamix/subscriptions","organizations_url":"https://api.github.com/users/dynamix/orgs","repos_url":"https://api.github.com/users/dynamix/repos","events_url":"https://api.github.com/users/dynamix/events{/privacy}","received_events_url":"https://api.github.com/users/dynamix/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-08-08T18:21:25Z","updated_at":"2016-08-09T02:30:18Z","closed_at":null,"body":"##### Versions\r\n\r\n- Sarama Version: master/e8020bffa1cae3ddf8068cb03416fc53d1627f3d\r\n- Kafka Version: 0.10\r\n- Go Version: 1.6\r\n\r\n##### Configuration\r\n\r\n- out of the box Kafka config values\r\n- topic \"test\" created with ```--create --topic test --partitions 1 --replication-factor 1```\r\n- sarama config see code : https://gist.github.com/dynamix/e7682ff7a76d28bd500ebee52c036b28\r\n\r\n\r\n##### Logs\r\n\r\n```\r\n2016/08/08 15:21:57 Initializing new client\r\n2016/08/08 15:21:57 client/metadata fetching metadata for all topics from broker 192.168.0.132:9092\r\n2016/08/08 15:21:57 Connected to broker at 192.168.0.132:9092 (unregistered)\r\n2016/08/08 15:21:57 client/brokers registered new broker #0 at 192.168.0.132:9092\r\n2016/08/08 15:21:57 Successfully initialized new client\r\n2016/08/08 15:21:57 producer/broker/0 starting up\r\n2016/08/08 15:21:57 producer/broker/0 state change to [open] on test/0\r\n2016/08/08 15:21:57 Connected to broker at 192.168.0.132:9092 (registered as #0)\r\n2016/08/08 15:21:57 Initializing new client\r\n2016/08/08 15:21:57 client/metadata fetching metadata for all topics from broker 192.168.0.132:9092\r\n2016/08/08 15:21:57 Connected to broker at 192.168.0.132:9092 (unregistered)\r\n2016/08/08 15:21:57 client/brokers registered new broker #0 at 192.168.0.132:9092\r\n2016/08/08 15:21:57 Successfully initialized new client\r\n2016/08/08 15:21:57 Connected to broker at 192.168.0.132:9092 (registered as #0)\r\n2016/08/08 15:21:57 consumer/broker/0 added subscription to test/0\r\n2016/08/08 15:21:57 kafka: error while consuming test/0: kafka: response did not contain all the expected topic/partition blocks\r\n2016/08/08 15:21:57 consumer/broker/0 abandoned subscription to test/0 because kafka: response did not contain all the expected topic/partition blocks\r\n2016/08/08 15:21:59 consumer/test/0 finding new broker\r\n2016/08/08 15:21:59 client/metadata fetching metadata for [test] from broker 192.168.0.132:9092\r\n2016/08/08 15:21:59 consumer/broker/0 added subscription to test/0\r\n2016/08/08 15:21:59 kafka: error while consuming test/0: kafka: response did not contain all the expected topic/partition blocks\r\n2016/08/08 15:21:59 consumer/broker/0 abandoned subscription to test/0 because kafka: response did not contain all the expected topic/partition blocks\r\n2016/08/08 15:22:01 consumer/test/0 finding new broker\r\n2016/08/08 15:22:01 kafka: error while consuming test/0: kafka: response did not contain all the expected topic/partition blocks\r\n2016/08/08 15:22:01 consumer/broker/0 abandoned subscription to test/0 because kafka: response did not contain all the expected topic/partition blocks\r\n2016/08/08 15:22:03 consumer/test/0 finding new broker\r\n2016/08/08 15:22:03 kafka: error while consuming test/0: kafka: response did not contain all the expected topic/partition blocks\r\n2016/08/08 15:22:03 consumer/broker/0 abandoned subscription to test/0 because kafka: response did not contain all the expected topic/partition blocks\r\n2016/08/08 15:22:05 consumer/test/0 finding new broker\r\n2016/08/08 15:22:05 kafka: error while consuming test/0: kafka: response did not contain all the expected topic/partition blocks\r\n2016/08/08 15:22:05 consumer/broker/0 abandoned subscription to test/0 because kafka: response did not contain all the expected topic/partition blocks\r\n```\r\n(removed the \"ClientID set to sarama\" entries)\r\n##### Problem Description\r\n\r\nCode that reproduces the issue (broker ip should be set to a proper value): https://gist.github.com/dynamix/e7682ff7a76d28bd500ebee52c036b28\r\n\r\nA consumer where Config.Version is set to V0_10_0_0 does not properly consume compressed topics. If the Config.Version is set to any lower value - everything works as it should. \r\n\r\nLooking at what kafka returns - the offsets of messages inside a compressed message are relative and not absolute for 0.10+. The Java Consumer has a special handling for relative offsets:   https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java#L242-L318\r\n\r\nAttempted fix (not sure if this handles all cases): https://github.com/Shopify/sarama/pull/721\r\n"},"comment":{"url":"https://api.github.com/repos/Shopify/sarama/issues/comments/238436191","html_url":"https://github.com/Shopify/sarama/issues/720#issuecomment-238436191","issue_url":"https://api.github.com/repos/Shopify/sarama/issues/720","id":238436191,"user":{"login":"eapache","id":1706826,"avatar_url":"https://avatars.githubusercontent.com/u/1706826?v=3","gravatar_id":"","url":"https://api.github.com/users/eapache","html_url":"https://github.com/eapache","followers_url":"https://api.github.com/users/eapache/followers","following_url":"https://api.github.com/users/eapache/following{/other_user}","gists_url":"https://api.github.com/users/eapache/gists{/gist_id}","starred_url":"https://api.github.com/users/eapache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eapache/subscriptions","organizations_url":"https://api.github.com/users/eapache/orgs","repos_url":"https://api.github.com/users/eapache/repos","events_url":"https://api.github.com/users/eapache/events{/privacy}","received_events_url":"https://api.github.com/users/eapache/received_events","type":"User","site_admin":false},"created_at":"2016-08-09T02:30:18Z","updated_at":"2016-08-09T02:30:18Z","body":"@edenhill does rdkafka handle this properly? Is there documentation on this change anywhere?"}}
{"actor":{"display_login":"eapache"}}