{"action":"created","issue":{"url":"https://api.github.com/repos/confluentinc/kafka-rest/issues/252","repository_url":"https://api.github.com/repos/confluentinc/kafka-rest","labels_url":"https://api.github.com/repos/confluentinc/kafka-rest/issues/252/labels{/name}","comments_url":"https://api.github.com/repos/confluentinc/kafka-rest/issues/252/comments","events_url":"https://api.github.com/repos/confluentinc/kafka-rest/issues/252/events","html_url":"https://github.com/confluentinc/kafka-rest/issues/252","id":186401452,"number":252,"title":"Crashes with Illegal Character (Producer)","user":{"login":"whithajess","id":2069021,"avatar_url":"https://avatars.githubusercontent.com/u/2069021?v=3","gravatar_id":"","url":"https://api.github.com/users/whithajess","html_url":"https://github.com/whithajess","followers_url":"https://api.github.com/users/whithajess/followers","following_url":"https://api.github.com/users/whithajess/following{/other_user}","gists_url":"https://api.github.com/users/whithajess/gists{/gist_id}","starred_url":"https://api.github.com/users/whithajess/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/whithajess/subscriptions","organizations_url":"https://api.github.com/users/whithajess/orgs","repos_url":"https://api.github.com/users/whithajess/repos","events_url":"https://api.github.com/users/whithajess/events{/privacy}","received_events_url":"https://api.github.com/users/whithajess/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-10-31T21:24:13Z","updated_at":"2016-11-22T23:19:17Z","closed_at":null,"body":"Im seeing pretty consistent crashes with certain messages: (I have a round robin and it crashes all machines)\r\n\r\n```\r\nWARN Illegal character 0x0 in state=START for buffer \r\nWARN badMessage: 400 Illegal character 0x0 for HttpChannelOverHttp@3a3c2e5f{r=0,c=false,a=IDLE,uri=-} (org.eclipse.jetty.http.HttpParser:1344)\r\nHeapByteBuffer@40e111d5[p=1,l=7361,c=16384,r=7360]={\\x00<<<\\x00\\x04C\\x00\\x00\\x00\\x02\\x00\\x00I\\xC9\\x00\\REDACTED}}}\r\n>>>\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00...\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00} (org.eclipse.jetty.http.HttpParser:1719)\r\nUncaught exception in kafka-producer-network-thread | producer-1:  (org.apache.kafka.common.utils.KafkaThread:30)\r\njava.lang.OutOfMemoryError: Java heap space\r\n```\r\n\r\nI thought this could be just a java configuration thing but I am using the suggested tuning from the docs:\r\n```\r\n-Xms1g -Xmx1g -XX:PermSize=48m -XX:MaxPermSize=48m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 \\\r\n       -XX:InitiatingHeapOccupancyPercent=35\r\n```\r\n\r\nI have built the proxy from source according to the docs and using:\r\n```\r\njava -version\r\njava version \"1.8.0_102\"\r\nJava(TM) SE Runtime Environment (build 1.8.0_102-b14)\r\nJava HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)\r\n```\r\n\r\nThe other problem I have is I am doing a health check based on `/` and it still responds 200 after this crash but it does not send `POST` after the crash.\r\n"},"comment":{"url":"https://api.github.com/repos/confluentinc/kafka-rest/issues/comments/262395447","html_url":"https://github.com/confluentinc/kafka-rest/issues/252#issuecomment-262395447","issue_url":"https://api.github.com/repos/confluentinc/kafka-rest/issues/252","id":262395447,"user":{"login":"ewencp","id":47366,"avatar_url":"https://avatars.githubusercontent.com/u/47366?v=3","gravatar_id":"","url":"https://api.github.com/users/ewencp","html_url":"https://github.com/ewencp","followers_url":"https://api.github.com/users/ewencp/followers","following_url":"https://api.github.com/users/ewencp/following{/other_user}","gists_url":"https://api.github.com/users/ewencp/gists{/gist_id}","starred_url":"https://api.github.com/users/ewencp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ewencp/subscriptions","organizations_url":"https://api.github.com/users/ewencp/orgs","repos_url":"https://api.github.com/users/ewencp/repos","events_url":"https://api.github.com/users/ewencp/events{/privacy}","received_events_url":"https://api.github.com/users/ewencp/received_events","type":"User","site_admin":false},"created_at":"2016-11-22T23:19:17Z","updated_at":"2016-11-22T23:19:17Z","body":"@whithajess Have you customized any of the producer settings? Can you get a stack trace and/or heap dump from the OOM error? It looks like the producer is simply running out of memory, but its memory usage should be pretty well bounded since it has limited buffer sizes. But if something else in the proxy is eating up memory, it could just be coincidence that the producer thread is the one that crashes.\r\n\r\nAnd of course, there's the obvious question -- does increasing the max heap help? Even with the recommended settings, we can't guarantee the amount of memory used. For example, you can instantiate as many consumers as you want, each of which may have buffered data, so there can't be a strict upper bound on memory usage."}}
{"actor":{"display_login":"ewencp"}}