{"action":"opened","issue":{"url":"https://api.github.com/repos/edenhill/librdkafka/issues/2121","repository_url":"https://api.github.com/repos/edenhill/librdkafka","labels_url":"https://api.github.com/repos/edenhill/librdkafka/issues/2121/labels{/name}","comments_url":"https://api.github.com/repos/edenhill/librdkafka/issues/2121/comments","events_url":"https://api.github.com/repos/edenhill/librdkafka/issues/2121/events","html_url":"https://github.com/edenhill/librdkafka/issues/2121","id":384840162,"node_id":"MDU6SXNzdWUzODQ4NDAxNjI=","number":2121,"title":"Segfault in round-robin assignor","user":{"login":"ankon","id":1210641,"node_id":"MDQ6VXNlcjEyMTA2NDE=","avatar_url":"https://avatars1.githubusercontent.com/u/1210641?v=4","gravatar_id":"","url":"https://api.github.com/users/ankon","html_url":"https://github.com/ankon","followers_url":"https://api.github.com/users/ankon/followers","following_url":"https://api.github.com/users/ankon/following{/other_user}","gists_url":"https://api.github.com/users/ankon/gists{/gist_id}","starred_url":"https://api.github.com/users/ankon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ankon/subscriptions","organizations_url":"https://api.github.com/users/ankon/orgs","repos_url":"https://api.github.com/users/ankon/repos","events_url":"https://api.github.com/users/ankon/events{/privacy}","received_events_url":"https://api.github.com/users/ankon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-11-27T15:08:22Z","updated_at":"2018-11-27T15:08:22Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Description\r\n===========\r\n\r\nWe're using the 'roundrobin' partition assignment strategy, and sometimes I get segfaults in rdk:main. I looked at one of these in more detail:\r\n~~~~\r\n#0  0x00007fc0f32d6fd3 in rd_kafka_group_member_find_subscription (rk=rk@entry=0x7fc0d4002e50, rkgm=rkgm@entry=0x7fc0b806a380, topic=0x7fc0b8076de0 \"ml3-commands\") at rdkafka_assignor.c:81\r\n#1  0x00007fc0f32d7fb4 in rd_kafka_roundrobin_assignor_assign_cb (rk=0x7fc0d4002e50, member_id=<optimized out>, protocol_name=<optimized out>, metadata=<optimized out>, members=0x7fc0b806a2e0, \r\n    member_cnt=<optimized out>, eligible_topics=0x7fc0b805bfa0, eligible_topic_cnt=12, errstr=0x7fc0d97f9a00 \"\", errstr_size=8, opaque=0x0) at rdkafka_roundrobin_assignor.c:80\r\n#2  0x00007fc0f32d74d8 in rd_kafka_assignor_run (rkcg=rkcg@entry=0x7fc0d4003b20, protocol_name=protocol_name@entry=0x6b8ec40 \"roundrobin\", metadata=0x7fc0b80769f0, members=members@entry=0x7fc0b806a2e0, \r\n    member_cnt=member_cnt@entry=2, errstr=errstr@entry=0x7fc0d97f9a00 \"\", errstr_size=512) at rdkafka_assignor.c:331\r\n#3  0x00007fc0f32be5f7 in rd_kafka_cgrp_assignor_run (rkcg=0x7fc0d4003b20, protocol_name=0x6b8ec40 \"roundrobin\", err=<optimized out>, metadata=<optimized out>, members=0x7fc0b806a2e0, member_cnt=2)\r\n    at rdkafka_cgrp.c:671\r\n#4  0x00007fc0f32be7cf in rd_kafka_cgrp_assignor_handle_Metadata_op (rk=<optimized out>, rkq=<optimized out>, rko=<optimized out>) at rdkafka_cgrp.c:733\r\n#5  0x00007fc0f32ab5fc in rd_kafka_op_call (rk=<optimized out>, rkq=<optimized out>, rko=0x7fc0b802d090) at rdkafka_op.c:519\r\n#6  0x00007fc0f32ac7e5 in rd_kafka_op_handle (rk=0x7fc0d4002e50, rkq=0x7fc0d97f9cc0, rko=0x7fc0b802d090, cb_type=RD_KAFKA_Q_CB_CALLBACK, opaque=0x0, callback=0x0) at rdkafka_op.c:647\r\n#7  0x00007fc0f32a58df in rd_kafka_q_serve (rkq=0x7fc0d40038f0, timeout_ms=<optimized out>, max_cnt=max_cnt@entry=0, cb_type=cb_type@entry=RD_KAFKA_Q_CB_CALLBACK, callback=callback@entry=0x0, \r\n    opaque=opaque@entry=0x0) at rdkafka_queue.c:481\r\n#8  0x00007fc0f327d4e4 in rd_kafka_thread_main (arg=arg@entry=0x7fc0d4002e50) at rdkafka.c:1393\r\n#9  0x00007fc0f32da907 in _thrd_wrapper_function (aArg=<optimized out>) at tinycthread.c:624\r\n#10 0x00007fc0f5d8358e in start_thread () from /lib64/libpthread.so.0\r\n#11 0x00007fc0f5cb0653 in clone () from /lib64/libc.so.6\r\n~~~~\r\n\r\nNote:\r\n1. This is librdkafka via node-rdkafka in NodeJS\r\n2 The commands in gdb below have been trimmed and reordered for readability :D\r\n\r\nIt seems that in `rd_kafka_roundrobin_assignor_assign_cb` the code is running over the length of the members array:\r\n```\r\n(gdb) up\r\n#1  0x00007fc0f32d7fb4 in rd_kafka_roundrobin_assignor_assign_cb (rk=0x7fc0d4002e50, member_id=<optimized out>, protocol_name=<optimized out>, metadata=<optimized out>, members=0x7fc0b806a2e0, \r\n    member_cnt=<optimized out>, eligible_topics=0x7fc0b805bfa0, eligible_topic_cnt=12, errstr=0x7fc0d97f9a00 \"\", errstr_size=8, opaque=0x0) at rdkafka_roundrobin_assignor.c:80\r\n80\t\t\tfor (partition = 0 ;\r\n(gdb) p next\r\n$12 = 2\r\n```\r\n(cf. \"2\", an index, to the member_cnt \"2\" seen in frame 2).\r\n\r\n```\r\n(gdb) p members[2]\r\n$13 = {rkgm_subscription = 0x0, rkgm_assignment = 0x3885, rkgm_eligible = {rl_size = 0, rl_cnt = 0, rl_elems = 0xf4240, rl_free_cb = 0x0, rl_flags = 2, rl_elemsize = 0, rl_p = 0x8000000007}, \r\n  rkgm_member_id = 0xff, rkgm_userdata = 0xd00000100, rkgm_member_metadata = 0x700}\r\n```\r\n\r\nI also looked the subscriptions involved for `members[0]`:\r\n```\r\n(gdb) p *members[0].rkgm_subscription \r\n$21 = {cnt = 12, size = 24, elems = 0x7fc0b80591c0}\r\n(gdb) p members[0].rkgm_subscription->elems[0]\r\n$23 = {topic = 0x7fc0b8017ad0 \"collaborne-commands\", partition = -1, offset = -1001, metadata = 0x0, metadata_size = 0, opaque = 0x0, err = RD_KAFKA_RESP_ERR_NO_ERROR, _private = 0x0}\r\n(gdb) p members[0].rkgm_subscription->elems[1]\r\n$24 = {topic = 0x7fc0b8056e10 \"collaborne-events\", partition = -1, offset = -1001, metadata = 0x0, metadata_size = 0, opaque = 0x0, err = RD_KAFKA_RESP_ERR_NO_ERROR, _private = 0x0}\r\n(gdb) p members[0].rkgm_subscription->elems[2]\r\n$25 = {topic = 0x7fc0b805c110 \"ml1-commands\", partition = -1, offset = -1001, metadata = 0x0, metadata_size = 0, opaque = 0x0, err = RD_KAFKA_RESP_ERR_NO_ERROR, _private = 0x0}\r\n(gdb) p members[0].rkgm_subscription->elems[3]\r\n$26 = {topic = 0x7fc0b8016620 \"ml1-events\", partition = -1, offset = -1001, metadata = 0x0, metadata_size = 0, opaque = 0x0, err = RD_KAFKA_RESP_ERR_NO_ERROR, _private = 0x0}\r\n(gdb) p members[0].rkgm_subscription->elems[4]\r\n$27 = {topic = 0x7fc0b8016670 \"ml2-commands\", partition = -1, offset = -1001, metadata = 0x0, metadata_size = 0, opaque = 0x0, err = RD_KAFKA_RESP_ERR_NO_ERROR, _private = 0x0}\r\n(gdb) p members[0].rkgm_subscription->elems[5]\r\n$28 = {topic = 0x7fc0b8015bf0 \"ml2-events\", partition = -1, offset = -1001, metadata = 0x0, metadata_size = 0, opaque = 0x0, err = RD_KAFKA_RESP_ERR_NO_ERROR, _private = 0x0}\r\n(gdb) p members[0].rkgm_subscription->elems[6]\r\n$29 = {topic = 0x7fc0b803e260 \"ml3-commands\", partition = -1, offset = -1001, metadata = 0x0, metadata_size = 0, opaque = 0x0, err = RD_KAFKA_RESP_ERR_NO_ERROR, _private = 0x0}\r\n(gdb) p members[0].rkgm_subscription->elems[7]\r\n$30 = {topic = 0x7fc0b8058170 \"ml3-events\", partition = -1, offset = -1001, metadata = 0x0, metadata_size = 0, opaque = 0x0, err = RD_KAFKA_RESP_ERR_NO_ERROR, _private = 0x0}\r\n(gdb) p members[0].rkgm_subscription->elems[8]\r\n$31 = {topic = 0x7fc0b803e2b0 \"ml4-commands\", partition = -1, offset = -1001, metadata = 0x0, metadata_size = 0, opaque = 0x0, err = RD_KAFKA_RESP_ERR_NO_ERROR, _private = 0x0}\r\n(gdb) p members[0].rkgm_subscription->elems[9]\r\n$32 = {topic = 0x7fc0b80508f0 \"ml4-events\", partition = -1, offset = -1001, metadata = 0x0, metadata_size = 0, opaque = 0x0, err = RD_KAFKA_RESP_ERR_NO_ERROR, _private = 0x0}\r\n(gdb) p members[0].rkgm_subscription->elems[10]\r\n$33 = {topic = 0x7fc0b805ee50 \"qa-commands\", partition = -1, offset = -1001, metadata = 0x0, metadata_size = 0, opaque = 0x0, err = RD_KAFKA_RESP_ERR_NO_ERROR, _private = 0x0}\r\n(gdb) p members[0].rkgm_subscription->elems[11]\r\n$34 = {topic = 0x7fc0b80378d0 \"qa-events\", partition = -1, offset = -1001, metadata = 0x0, metadata_size = 0, opaque = 0x0, err = RD_KAFKA_RESP_ERR_NO_ERROR, _private = 0x0}\r\n```\r\n... and `members[1]`:\r\n```\r\n(gdb)  p *members[1].rkgm_subscription\r\n$50 = {cnt = 5, size = 10, elems = 0x7fc0b8015660}\r\n(gdb) p members[1].rkgm_subscription->elems[0]\r\n$42 = {topic = 0x7fc0b8057760 \"collaborne-events\", partition = -1, offset = -1001, metadata = 0x0, metadata_size = 0, opaque = 0x0, err = RD_KAFKA_RESP_ERR_NO_ERROR, _private = 0x0}\r\n(gdb) p members[1].rkgm_subscription->elems[1]\r\n$43 = {topic = 0x7fc0b8046c10 \"ml1-events\", partition = -1, offset = -1001, metadata = 0x0, metadata_size = 0, opaque = 0x0, err = RD_KAFKA_RESP_ERR_NO_ERROR, _private = 0x0}\r\n(gdb) p members[1].rkgm_subscription->elems[2]\r\n$44 = {topic = 0x7fc0b805e730 \"ml2-events\", partition = -1, offset = -1001, metadata = 0x0, metadata_size = 0, opaque = 0x0, err = RD_KAFKA_RESP_ERR_NO_ERROR, _private = 0x0}\r\n(gdb) p members[1].rkgm_subscription->elems[3]\r\n$45 = {topic = 0x7fc0b8037340 \"ml3-events\", partition = -1, offset = -1001, metadata = 0x0, metadata_size = 0, opaque = 0x0, err = RD_KAFKA_RESP_ERR_NO_ERROR, _private = 0x0}\r\n(gdb) p members[1].rkgm_subscription->elems[4]\r\n$46 = {topic = 0x7fc0b803d9e0 \"qa-events\", partition = -1, offset = -1001, metadata = 0x0, metadata_size = 0, opaque = 0x0, err = RD_KAFKA_RESP_ERR_NO_ERROR, _private = 0x0}\r\n```\r\n\r\nThe crash doesn't always happen, and looking at the subscriptions there seems to be a good chance that it is caused by some form of concurrent modifications to the subscription list. This would not be impossible, our consumer does see changes in subscribed topics sometimes.\r\n\r\nLooking at https://github.com/edenhill/librdkafka/blob/d7d58e5407852d17485cf8a82841523eb2b8f6d1/src/rdkafka_roundrobin_assignor.c#L85-L90: It seems that `next++` should be guarded over the length of the members, for example:\r\n```diff\r\nnext = (next+1) % member_cnt;\r\n```\r\n\r\nI'm also a bit in doubt about the later part in https://github.com/edenhill/librdkafka/blob/d7d58e5407852d17485cf8a82841523eb2b8f6d1/src/rdkafka_roundrobin_assignor.c#L105: Instead of checking the eligible topic this should probably use `member_cnt` as well.\r\n\r\nChecklist\r\n=========\r\n\r\nPlease provide the following information:\r\n\r\n - [x] librdkafka version (release number or git tag): `v0.11.5` (from pkgconfig)\r\n - [x] Apache Kafka version: `2.0.0`\r\n - [ ] librdkafka client configuration: `partition.assignment.strategy=roundrobin` (and others)\r\n - [x] Operating system: `Fedora 29/x86_64`\r\n - [ ] Provide logs (with `debug=..` as necessary) from librdkafka\r\n - [ ] Provide broker log excerpts\r\n - [ ] Critical issue\r\n\r\n"}}
{"actor":{"display_login":"ankon"}}