{"action":"created","issue":{"url":"https://api.github.com/repos/edenhill/librdkafka/issues/724","repository_url":"https://api.github.com/repos/edenhill/librdkafka","labels_url":"https://api.github.com/repos/edenhill/librdkafka/issues/724/labels{/name}","comments_url":"https://api.github.com/repos/edenhill/librdkafka/issues/724/comments","events_url":"https://api.github.com/repos/edenhill/librdkafka/issues/724/events","html_url":"https://github.com/edenhill/librdkafka/issues/724","id":164407875,"number":724,"title":"core related to broker refcnt and DNS issues","user":{"login":"panarchus","id":2066647,"avatar_url":"https://avatars.githubusercontent.com/u/2066647?v=3","gravatar_id":"","url":"https://api.github.com/users/panarchus","html_url":"https://github.com/panarchus","followers_url":"https://api.github.com/users/panarchus/followers","following_url":"https://api.github.com/users/panarchus/following{/other_user}","gists_url":"https://api.github.com/users/panarchus/gists{/gist_id}","starred_url":"https://api.github.com/users/panarchus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/panarchus/subscriptions","organizations_url":"https://api.github.com/users/panarchus/orgs","repos_url":"https://api.github.com/users/panarchus/repos","events_url":"https://api.github.com/users/panarchus/events{/privacy}","received_events_url":"https://api.github.com/users/panarchus/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-07-07T21:09:04Z","updated_at":"2016-07-08T17:49:11Z","closed_at":null,"body":"Description\r\n===========\r\nWe are running into an interesting issue using rdkafka that results in a core file with this backtrace:\r\n\r\n#0  0x00007f5f4b58bc10 in pthread_mutex_lock () from /lib64/libpthread.so.0\r\n#1  0x00007f5f3036cc49 in mtx_lock (mtx=<optimized out>) at tinycthread.c:135\r\n#2  0x00007f5f3033b386 in rd_refcnt_sub0 (R=0x1e0) at rd.h:266\r\n#3  rd_kafka_broker_op_serve (rko=0x7f5c2c02b690, rkb=0x7f5c100d27b0) at rdkafka_broker.c:2027\r\n#4  rd_kafka_broker_serve (rkb=rkb@entry=0x7f5c100d27b0, timeout_ms=timeout_ms@entry=10) at rdkafka_broker.c:2072\r\n#5  0x00007f5f3033cb91 in rd_kafka_broker_ua_idle (rkb=rkb@entry=0x7f5c100d27b0, timeout_ms=timeout_ms@entry=1000) at rdkafka_broker.c:2136\r\n#6  0x00007f5f3033cd98 in rd_kafka_broker_thread_main (arg=0x7f5c100d27b0) at rdkafka_broker.c:3277\r\n#7  0x00007f5f3036cb99 in _thrd_wrapper_function (aArg=<optimized out>) at tinycthread.c:596\r\n#8  0x00007f5f4b589df3 in start_thread () from /lib64/libpthread.so.0\r\n#9  0x00007f5f4ba983dd in clone () from /lib64/libc.so.\r\n\r\nWe run into this problem on our distributed database (Vertica, 3 physical nodes, each loading and using librdkafka) on nodes in which the Kafka brokers/zookeeper do not have DNS names in /etc/hosts, and we give the IP addresses to rdkafka. It will usually work for about 30 minutes or so before crashing. If we include the DNS names in all /etc/hosts files the crash does not occur.\r\n\r\nThe database attempts to load data from topics every 10 seconds, and each invocation we re-connect/give rdkafka the broker list.\r\n\r\nHave you seen any similar issues? Check below for some environment details. Let me know what other information would be useful for you. Also, is this expected?\r\n\r\nThanks!\r\n\r\nChecklist\r\n=========\r\nPlease provide the following information:\r\n\r\n - [X] librdkafka version (release number or git tag): 0.9.1\r\n - [X] Apache Kafka version: kafka_2.11-0.9.0.1\r\n - [ ] librdkafka client configuration:\r\n - [X] Operating system: CentOS 7\r\n - [X] Using the legacy Consumer\r\n - [ ] Provide logs (with `debug=..` as necessary) from librdkafka\r\n - [ ] Provide broker log excerpts\r\n - [X] Critical issue\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/edenhill/librdkafka/issues/comments/231426470","html_url":"https://github.com/edenhill/librdkafka/issues/724#issuecomment-231426470","issue_url":"https://api.github.com/repos/edenhill/librdkafka/issues/724","id":231426470,"user":{"login":"panarchus","id":2066647,"avatar_url":"https://avatars.githubusercontent.com/u/2066647?v=3","gravatar_id":"","url":"https://api.github.com/users/panarchus","html_url":"https://github.com/panarchus","followers_url":"https://api.github.com/users/panarchus/followers","following_url":"https://api.github.com/users/panarchus/following{/other_user}","gists_url":"https://api.github.com/users/panarchus/gists{/gist_id}","starred_url":"https://api.github.com/users/panarchus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/panarchus/subscriptions","organizations_url":"https://api.github.com/users/panarchus/orgs","repos_url":"https://api.github.com/users/panarchus/repos","events_url":"https://api.github.com/users/panarchus/events{/privacy}","received_events_url":"https://api.github.com/users/panarchus/received_events","type":"User","site_admin":false},"created_at":"2016-07-08T17:49:11Z","updated_at":"2016-07-08T17:49:11Z","body":"I will do this when I load the symbols; in the meantime @ frame 4:\r\n(gdb) p *rkb\r\n$4 = {rkb_link = {tqe_next = 0x7f5c100d2e10, tqe_prev = 0x7f5c100a8620}, rkb_nodeid = 1, rkb_rsal = 0x0, \r\n  rkb_t_rsal_last = 0, rkb_addr_last = 0x0, rkb_transport = 0x0, rkb_corrid = 2, rkb_ops = {rkq_lock = {\r\n      __data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {\r\n          __prev = 0x0, __next = 0x0}}, __size = '\\000' <repeats 39 times>, __align = 0}, rkq_cond = {\r\n      __data = {__lock = 0, __futex = 202, __total_seq = 101, __wakeup_seq = 101, __woken_seq = 101, \r\n        __mutex = 0x7f5c100d27f0, __nwaiters = 0, __broadcast_seq = 0}, \r\n      __size = \"\\000\\000\\000\\000\\312\\000\\000\\000e\\000\\000\\000\\000\\000\\000\\000e\\000\\000\\000\\000\\000\\000\\000e\\000\\000\\000\\000\\000\\000\\000\\360'\\r\\020\\\\\\177\\000\\000\\000\\000\\000\\000\\000\\000\\000\", __align = 867583393792}, \r\n    rkq_fwdq = 0x0, rkq_q = {tqh_first = 0x0, tqh_last = 0x7f5c100d2850}, rkq_qlen = 0, rkq_qsize = 0, \r\n    rkq_refcnt = 1, rkq_flags = 2, rkq_rk = 0x7f5b700fe850}, rkb_toppars = {tqh_first = 0x0, \r\n    tqh_last = 0x7f5c100d2880}, rkb_toppar_cnt = -1, rkb_fetch_toppars = {cqh_first = 0x7f5c100d2898, \r\n    cqh_last = 0x7f5c100d2898}, rkb_fetch_toppar_cnt = 0, rkb_fetch_toppar_next = 0x0, rkb_cgrp = 0x0, \r\n  rkb_ts_fetch_backoff = 0, rkb_fetching = 0, rkb_state = RD_KAFKA_BROKER_STATE_DOWN, \r\n  rkb_ts_state = 3383218937259, rkb_timeout_scan_intvl = {ri_ts_last = 3383218947452, ri_fixed = 0, \r\n    ri_backoff = 0}, rkb_blocking_request_cnt = {val = 0}, rkb_source = RD_KAFKA_LEARNED, rkb_c = {\r\n    tx_bytes = {val = 0}, tx = {val = 0}, tx_err = {val = 0}, tx_retries = {val = 0}, req_timeouts = {\r\n      val = 0}, rx_bytes = {val = 0}, rx = {val = 0}, rx_err = {val = 0}, rx_corrid_err = {val = 0}, \r\n    rx_partial = {val = 0}}, rkb_req_timeouts = 0, rkb_ts_metadata_poll = 3383519913812, \r\n  rkb_metadata_fast_poll_cnt = 0, rkb_lock = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, \r\n      __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\\000' <repeats 39 times>, \r\n    __align = 0}, rkb_thread = 140033717696256, rkb_refcnt = {lock = {__data = {__lock = 0, __count = 0, \r\n        __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, \r\n      __size = '\\000' <repeats 39 times>, __align = 0}, v = 2}, rkb_rk = 0x7f5b700fe850, rkb_recv_buf = 0x0, \r\n  rkb_outbufs = {rkbq_bufs = {tqh_first = 0x7f5c100a7500, tqh_last = 0x7f5c100a7500}, rkbq_cnt = {val = 1}, \r\n    rkbq_msg_cnt = {val = 0}}, rkb_outbuf_msgcnt = {val = 0}, rkb_waitresps = {rkbq_bufs = {tqh_first = 0x0, \r\n      tqh_last = 0x7f5c100d29f0}, rkbq_cnt = {val = 0}, rkbq_msg_cnt = {val = 0}}, rkb_retrybufs = {\r\n    rkbq_bufs = {tqh_first = 0x0, tqh_last = 0x7f5c100d2a08}, rkbq_cnt = {val = 0}, rkbq_msg_cnt = {\r\n      val = 0}}, rkb_avg_rtt = {ra_v = {maxv = 0, minv = 0, avg = 0, sum = 0, cnt = 0, \r\n      start = 3383218936812}, ra_lock = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, \r\n        __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\\000' <repeats 39 times>, \r\n      __align = 0}, ra_type = RD_AVG_GAUGE}, rkb_avg_throttle = {ra_v = {maxv = 0, minv = 0, avg = 0, \r\n      sum = 0, cnt = 0, start = 3383218936812}, ra_lock = {__data = {__lock = 0, __count = 0, __owner = 0, \r\n        __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, \r\n---Type <return> to continue, or q <return> to quit---\r\n      __size = '\\000' <repeats 39 times>, __align = 0}, ra_type = RD_AVG_GAUGE}, \r\n  rkb_name = \"intco64-181:9093/1\", '\\000' <repeats 109 times>, \r\n  rkb_nodename = \"intco64-181:9093\", '\\000' <repeats 111 times>, rkb_port = 9093, \r\n  rkb_origname = 0x7f5c10001290 \"intco64-181\", rkb_ts_connect = 3383219939992, \r\n  rkb_proto = RD_KAFKA_PROTO_PLAINTEXT, rkb_err = {\r\n    **msg = \"intco64-181:9093/1: Failed to resolve 'intco64-181:9093': Name or service not known\", '\\000' <repeats 428 times>, err = 14**}}\r\n\r\nThat msg value shows the DNS name for the IP address of the brokers... could this be related to: https://issues.apache.org/jira/browse/KAFKA-3352?\r\n\r\nI will have the info you asked for later today."}}
{"actor":{"display_login":"panarchus"}}