{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/10213","labels_url":"https://api.github.com/repos/apache/spark/issues/10213/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/10213/comments","events_url":"https://api.github.com/repos/apache/spark/issues/10213/events","html_url":"https://github.com/apache/spark/pull/10213","id":121148737,"number":10213,"title":"[SPARK-12222] [Core] Deserialize RoaringBitmap using Kryo serializer throw Buffer underflow exception","user":{"login":"scwf","id":7018048,"avatar_url":"https://avatars.githubusercontent.com/u/7018048?v=3","gravatar_id":"","url":"https://api.github.com/users/scwf","html_url":"https://github.com/scwf","followers_url":"https://api.github.com/users/scwf/followers","following_url":"https://api.github.com/users/scwf/following{/other_user}","gists_url":"https://api.github.com/users/scwf/gists{/gist_id}","starred_url":"https://api.github.com/users/scwf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scwf/subscriptions","organizations_url":"https://api.github.com/users/scwf/orgs","repos_url":"https://api.github.com/users/scwf/repos","events_url":"https://api.github.com/users/scwf/events{/privacy}","received_events_url":"https://api.github.com/users/scwf/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-12-09T02:52:25Z","updated_at":"2016-01-21T06:47:46Z","closed_at":"2015-12-09T05:34:09Z","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/10213","html_url":"https://github.com/apache/spark/pull/10213","diff_url":"https://github.com/apache/spark/pull/10213.diff","patch_url":"https://github.com/apache/spark/pull/10213.patch"},"body":"Jira: https://issues.apache.org/jira/browse/SPARK-12222\r\n\r\nDeserialize RoaringBitmap using Kryo serializer throw Buffer underflow exception:\r\n```\r\ncom.esotericsoftware.kryo.KryoException: Buffer underflow.\r\n\tat com.esotericsoftware.kryo.io.Input.require(Input.java:156)\r\n\tat com.esotericsoftware.kryo.io.Input.skip(Input.java:131)\r\n\tat com.esotericsoftware.kryo.io.Input.skip(Input.java:264)\r\n```\r\n\r\nThis is caused by a bug of kryo's `Input.skip(long count)`(https://github.com/EsotericSoftware/kryo/issues/119) and we call this method in `KryoInputDataInputBridge`.\r\n\r\nInstead of upgrade kryo's version, this pr bypass the  kryo's `Input.skip(long count)` by directly call another `skip` method in kryo's Input.java(https://github.com/EsotericSoftware/kryo/blob/kryo-2.21/src/com/esotericsoftware/kryo/io/Input.java#L124), i.e. write the bug-fixed version of `Input.skip(long count)` in KryoInputDataInputBridge's `skipBytes` method.\r\n\r\nmore detail link to https://github.com/apache/spark/pull/9748#issuecomment-162860246"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/173475873","html_url":"https://github.com/apache/spark/pull/10213#issuecomment-173475873","issue_url":"https://api.github.com/repos/apache/spark/issues/10213","id":173475873,"user":{"login":"highmoutain","id":16812890,"avatar_url":"https://avatars.githubusercontent.com/u/16812890?v=3","gravatar_id":"","url":"https://api.github.com/users/highmoutain","html_url":"https://github.com/highmoutain","followers_url":"https://api.github.com/users/highmoutain/followers","following_url":"https://api.github.com/users/highmoutain/following{/other_user}","gists_url":"https://api.github.com/users/highmoutain/gists{/gist_id}","starred_url":"https://api.github.com/users/highmoutain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/highmoutain/subscriptions","organizations_url":"https://api.github.com/users/highmoutain/orgs","repos_url":"https://api.github.com/users/highmoutain/repos","events_url":"https://api.github.com/users/highmoutain/events{/privacy}","received_events_url":"https://api.github.com/users/highmoutain/received_events","type":"User","site_admin":false},"created_at":"2016-01-21T06:47:46Z","updated_at":"2016-01-21T06:47:46Z","body":"does this problem have been solved in 1.6.0?\r\nBut I encoutered this problem again in 1.6.0 when I use udf.\r\nmy version is spark-1.6.0-bin-hadoop2.6.tgz.\r\n\r\n\r\n(p.mobile_id, 'model', 'MOBILE_MODEL') as mobile_model from analytics_device_profile_zh10 p limit 1;\r\n16/01/21 14:47:29 WARN TaskSetManager: Lost task 0.0 in stage 44.0 (TID 44, 172.30.4.216): com.esotericsoftware.kryo.KryoException: Buffer underflow.\r\nSerialization trace:\r\n\r\nmobileAttributeCahceMap (enterprise.hive.udfs.ConvertDeviceModel)\r\n\tat com.esotericsoftware.kryo.io.Input.require(Input.java:156)\r\n\tat com.esotericsoftware.kryo.io.Input.readAscii_slow(Input.java:580)\r\n\tat com.esotericsoftware.kryo.io.Input.readAscii(Input.java:558)\r\n\tat com.esotericsoftware.kryo.io.Input.readString(Input.java:436)\r\n"}}