{"action":"created","issue":{"url":"https://api.github.com/repos/edenhill/librdkafka/issues/1553","repository_url":"https://api.github.com/repos/edenhill/librdkafka","labels_url":"https://api.github.com/repos/edenhill/librdkafka/issues/1553/labels{/name}","comments_url":"https://api.github.com/repos/edenhill/librdkafka/issues/1553/comments","events_url":"https://api.github.com/repos/edenhill/librdkafka/issues/1553/events","html_url":"https://github.com/edenhill/librdkafka/issues/1553","id":276535196,"number":1553,"title":"CPU utilization to 100%   exists always on producing message stress test ","user":{"login":"micweaver","id":4676001,"avatar_url":"https://avatars0.githubusercontent.com/u/4676001?v=4","gravatar_id":"","url":"https://api.github.com/users/micweaver","html_url":"https://github.com/micweaver","followers_url":"https://api.github.com/users/micweaver/followers","following_url":"https://api.github.com/users/micweaver/following{/other_user}","gists_url":"https://api.github.com/users/micweaver/gists{/gist_id}","starred_url":"https://api.github.com/users/micweaver/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/micweaver/subscriptions","organizations_url":"https://api.github.com/users/micweaver/orgs","repos_url":"https://api.github.com/users/micweaver/repos","events_url":"https://api.github.com/users/micweaver/events{/privacy}","received_events_url":"https://api.github.com/users/micweaver/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":23,"created_at":"2017-11-24T07:31:54Z","updated_at":"2018-03-06T12:37:39Z","closed_at":"2018-01-15T10:10:29Z","author_association":"NONE","body":"Description\r\n===========\r\nwe made stress test  on librdkafka( use php-rdkafka base on this library) , produce large numbers of message to kafka cluster,  and there were some connection errors,  like it :   \r\n-192:Local: Message timed out\r\n\r\nstrangely，after a few minutes, the cpu  utilization always be 100%,  occupied by the php process\r\n\r\nI  observed the process hading problem like this:\r\n\r\nroot@tsung02:/home/work# ps -eLf | grep 9698\r\nwork       9698   9661   9698  0    4 11:52 ?        00:00:00 php-fpm: pool php7-www\r\nwork       9698   9661  60420  0    4 11:53 ?        00:00:00 php-fpm: pool php7-www\r\nwork       9698   9661  60426  5    4 11:53 ?        00:00:36 php-fpm: pool php7-www\r\nwork       9698   9661  60430  5    4 11:53 ?        00:00:35 php-fpm: pool php7-www\r\nroot      99109   9239  99109  0    1 12:04 pts/1    00:00:00 grep --color=auto 9698 \r\n\r\nroot@tsung02:/home/work# strace -p 60420\r\nstrace: Process 60420 attached\r\nfutex(0x7f9003fff9d0, FUTEX_WAIT, 60426, NULLstrace: Process 60420 detached\r\n <detached ...>\r\n\r\nroot@tsung02:/home/work# gdb  -p 60426\r\n\r\n(gdb) bt\r\n#0  0x00007f903ad78cf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0\r\n#1  0x00007f9030f42b85 in cnd_timedwait_ms (cnd=cnd@entry=0x168dec8, mtx=mtx@entry=0x168dea0, timeout_ms=timeout_ms@entry=0) at tinycthread.c:501\r\n#2  0x00007f9030f0ee4a in rd_kafka_q_pop_serve (rkq=0x168dea0, timeout_ms=0, version=version@entry=0, cb_type=cb_type@entry=RD_KAFKA_Q_CB_RETURN, \r\n    callback=callback@entry=0x0, opaque=opaque@entry=0x0) at rdkafka_queue.c:364\r\n#3  0x00007f9030f0efa0 in rd_kafka_q_pop (rkq=<optimized out>, timeout_ms=<optimized out>, version=version@entry=0) at rdkafka_queue.c:395\r\n#4  0x00007f9030ef8da4 in rd_kafka_broker_serve (rkb=rkb@entry=0x1861630, abs_timeout=abs_timeout@entry=88121366769) at rdkafka_broker.c:2184\r\n#5  0x00007f9030ef9228 in rd_kafka_broker_ua_idle (rkb=rkb@entry=0x1861630, timeout_ms=<optimized out>, timeout_ms@entry=-1) at rdkafka_broker.c:2270\r\n#6  0x00007f9030ef96b8 in rd_kafka_broker_thread_main (arg=arg@entry=0x1861630) at rdkafka_broker.c:3119\r\n#7  0x00007f9030f42927 in _thrd_wrapper_function (aArg=<optimized out>) at tinycthread.c:624\r\n#8  0x00007f903ad74e25 in start_thread () from /usr/lib64/libpthread.so.0\r\n#9  0x00007f903c0a034d in clone () from /usr/lib64/libc.so.6\r\n\r\n\r\nall the process having problem had the same backtrace, and always be.\r\n\r\n\r\n\r\nHow to reproduce\r\n================\r\n\r\ncan be reproduced when  producing message on stress test \r\n\r\n\r\nChecklist\r\n=========\r\nPlease provide the following information:\r\n\r\n - [x] librdkafka version (release number or git tag):    librdkafka version (build) => 0.11.0.0\r\n - [x] Apache Kafka version:    0.11.0.1 \r\n - [x] librdkafka client configuration:\r\n\r\n        'socket.timeout.ms' => 10000, \r\n        'queue.buffering.max.ms' => 0, \r\n        'socket.blocking.max.ms' => 5,\r\n         'message.timeout.ms' => 1000,\r\n        'request.timeout.ms' => 3000, \r\n\r\n - [x] Operating system:  CentOS Linux 7\r\n - [ ] Using the legacy Consumer\r\n - [ ] Using the high-level KafkaConsumer\r\n - [ ] Provide logs (with `debug=..` as necessary) from librdkafka\r\n - [ ] Provide broker log excerpts\r\n - [ ] Critical issue\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/edenhill/librdkafka/issues/comments/370768277","html_url":"https://github.com/edenhill/librdkafka/issues/1553#issuecomment-370768277","issue_url":"https://api.github.com/repos/edenhill/librdkafka/issues/1553","id":370768277,"user":{"login":"6opuc","id":1191613,"avatar_url":"https://avatars3.githubusercontent.com/u/1191613?v=4","gravatar_id":"","url":"https://api.github.com/users/6opuc","html_url":"https://github.com/6opuc","followers_url":"https://api.github.com/users/6opuc/followers","following_url":"https://api.github.com/users/6opuc/following{/other_user}","gists_url":"https://api.github.com/users/6opuc/gists{/gist_id}","starred_url":"https://api.github.com/users/6opuc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/6opuc/subscriptions","organizations_url":"https://api.github.com/users/6opuc/orgs","repos_url":"https://api.github.com/users/6opuc/repos","events_url":"https://api.github.com/users/6opuc/events{/privacy}","received_events_url":"https://api.github.com/users/6opuc/received_events","type":"User","site_admin":false},"created_at":"2018-03-06T12:37:39Z","updated_at":"2018-03-06T12:37:39Z","author_association":"NONE","body":"It is very strange, but I've encountered the same problem with Confluent.Kafka 0.11.3, but only on linux(docker image microsoft/dotnet:2.0.3-runtime). **On Windows 8.1 everything is fine**.\r\nI used config from https://github.com/confluentinc/confluent-kafka-dotnet/wiki/Producing-messages([\"queue.buffering.max.ms\"] = 0, [\"socket.blocking.max.ms\"] = 1) and CPU usage was 400%(4 cores).\r\nAfter changing [\"socket.blocking.max.ms\"] to 10 CPU usage dropped to 10%([\"queue.buffering.max.ms\"] was the same: 0).\r\nIs it expected behavior or a bug(difference in behavior on windows and linux)?\r\n\r\n\r\n\r\n"}}
{"actor":{"display_login":"6opuc"}}