{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/ingress-nginx/issues/3314","repository_url":"https://api.github.com/repos/kubernetes/ingress-nginx","labels_url":"https://api.github.com/repos/kubernetes/ingress-nginx/issues/3314/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/ingress-nginx/issues/3314/comments","events_url":"https://api.github.com/repos/kubernetes/ingress-nginx/issues/3314/events","html_url":"https://github.com/kubernetes/ingress-nginx/issues/3314","id":374926925,"node_id":"MDU6SXNzdWUzNzQ5MjY5MjU=","number":3314,"title":"Nginx OOM","user":{"login":"yvespp","id":15231595,"node_id":"MDQ6VXNlcjE1MjMxNTk1","avatar_url":"https://avatars0.githubusercontent.com/u/15231595?v=4","gravatar_id":"","url":"https://api.github.com/users/yvespp","html_url":"https://github.com/yvespp","followers_url":"https://api.github.com/users/yvespp/followers","following_url":"https://api.github.com/users/yvespp/following{/other_user}","gists_url":"https://api.github.com/users/yvespp/gists{/gist_id}","starred_url":"https://api.github.com/users/yvespp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yvespp/subscriptions","organizations_url":"https://api.github.com/users/yvespp/orgs","repos_url":"https://api.github.com/users/yvespp/repos","events_url":"https://api.github.com/users/yvespp/events{/privacy}","received_events_url":"https://api.github.com/users/yvespp/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":20,"created_at":"2018-10-29T09:32:53Z","updated_at":"2018-11-08T20:41:23Z","closed_at":null,"author_association":"NONE","body":"<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.-->\r\n\r\n**Is this a request for help?** (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.): yes\r\n\r\n**What keywords did you search in NGINX Ingress controller issues before filing this one?** (If you have found any duplicates, you should instead reply there.): memory, oom\r\n\r\n---\r\n\r\n**Is this a BUG REPORT or FEATURE REQUEST?** (choose one): BUG REPORT\r\n\r\n<!--\r\nIf this is a BUG REPORT, please:\r\n  - Fill in as much of the template below as you can.  If you leave out\r\n    information, we can't help you as well.\r\n\r\nIf this is a FEATURE REQUEST, please:\r\n  - Describe *in detail* the feature/behavior/change you'd like to see.\r\n\r\nIn both cases, be ready for followup questions, and please respond in a timely\r\nmanner.  If we can't reproduce a bug or think a feature already exists, we\r\nmight close your issue.  If we're wrong, PLEASE feel free to reopen it and\r\nexplain why.\r\n-->\r\n\r\n**NGINX Ingress controller version**: 0.20.0 and 0.19.0\r\n\r\n**Kubernetes version** (use `kubectl version`): v1.11.4\r\n\r\n\r\n**Environment**:\r\n\r\n- **Cloud provider or hardware configuration**: 128 GB RAM; 20 CPUs\r\n- **OS** (e.g. from /etc/os-release): RHEL 7.5\r\n- **Kernel** (e.g. `uname -a`): 3.10.0\r\n- **Install tools**:\r\n- **Others**:\r\n\r\n\r\n**What happened**: The memory of Nginx increases slowly and then suddenly increases rapidly until the Pod gets OOM killed by Kubernetes. In the Pod I can see that the memory is used by nginx itself.\r\n\r\nHere the memory graph of one Pod over the last 24h:\r\n![image](https://user-images.githubusercontent.com/15231595/47636567-14108780-db59-11e8-918c-ea3906090b45.png)\r\n\r\nAll pods last 24h:\r\n![image](https://user-images.githubusercontent.com/15231595/47636673-79647880-db59-11e8-8787-8008fc1c9729.png)\r\n\r\nIn the log of one of the crashed Pods I see this:\r\n```\r\n2018/10/28 23:44:13 [alert] 63#63: worker process 96931 exited on signal 9\r\n2018/10/28 23:44:14 [alert] 63#63: worker process 96964 exited on signal 9\r\n2018/10/28 23:44:14 [alert] 63#63: worker process 96997 exited on signal 9\r\n2018/10/28 23:44:14 [alert] 63#63: worker process 96998 exited on signal 9\r\n2018/10/28 23:44:14 [alert] 63#63: worker process 97064 exited on signal 9\r\n2018/10/28 23:44:15 [error] 97065#97065: *754462 lua entry thread aborted: memory allocation error: not enough memory\r\nstack traceback:\r\ncoroutine 0:\r\n        [C]: in function 'ffi_str'\r\n        /usr/local/lib/lua/resty/core/shdict.lua:242: in function 'get_backends_data'\r\n        /etc/nginx/lua/balancer.lua:100: in function </etc/nginx/lua/balancer.lua:99>, context: ngx.timer\r\n2018/10/28 23:44:15 [alert] 63#63: worker process 97131 exited on signal 9\r\n2018/10/28 23:44:15 [alert] 63#63: worker process 97065 exited on signal 9\r\n2018/10/28 23:44:16 [alert] 63#63: worker process 97229 exited on signal 9\r\n2018/10/28 23:44:16 [alert] 63#63: worker process 97164 exited on signal 9\r\n2018/10/28 23:44:16 [alert] 63#63: worker process 97295 exited on signal 9\r\n2018/10/28 23:44:16 [alert] 63#63: worker process 97130 exited on signal 9\r\n2018/10/28 23:44:17 [alert] 63#63: worker process 97262 exited on signal 9\r\n```\r\n\r\nWith version 0.17.1 in the same cluster and dynamic config we had no such problems. 0.19.0 showed the same OOM behavior. I didn't try 0.18.\r\n\r\n**What you expected to happen**: No OOMs\r\n\r\n**How to reproduce it** (as minimally and precisely as possible): I don't know what causes it, doesn't happen on other, smaller clusters in our env. On the affected cluster we have 1121 Ingresses, 1384 Services and 1808 Pods.\r\n\r\n**Anything else we need to know**:\r\nNginx Config Map:\r\n```\r\n  upstream-max-fails: \"3\"\r\n  upstream-fail-timeout: \"5\"\r\n  proxy-read-timeout: \"300\"\r\n  proxy-send-timeout: \"300\"\r\n  proxy-connect-timeout: \"10\"\r\n  use-gzip: \"false\"\r\n  client-body-buffer-size: \"64k\"\r\n  server-tokens: \"false\"\r\n  proxy-body-size: \"20m\"\r\n  client_max_body_size: \"20m\"\r\n  worker-shutdown-timeout: \"300s\"\r\n  worker-processes: \"4\"\r\n```\r\nController flags:\r\n```\r\n      containers:\r\n      - args:\r\n        - /nginx-ingress-controller\r\n        - --default-ssl-certificate=$(POD_NAMESPACE)/ingress-cert\r\n        - --configmap=$(POD_NAMESPACE)/nginx-ingress-conf\r\n        - --enable-ssl-chain-completion=false\r\n        - --update-status=false\r\n        - --publish-service=$(POD_NAMESPACE)/nginx-ingress\r\n```\r\n\r\nHow can I debug this? Can I somehow see how much memory the lua module uses?"},"comment":{"url":"https://api.github.com/repos/kubernetes/ingress-nginx/issues/comments/437148136","html_url":"https://github.com/kubernetes/ingress-nginx/issues/3314#issuecomment-437148136","issue_url":"https://api.github.com/repos/kubernetes/ingress-nginx/issues/3314","id":437148136,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNzE0ODEzNg==","user":{"login":"evilezh","id":5277173,"node_id":"MDQ6VXNlcjUyNzcxNzM=","avatar_url":"https://avatars0.githubusercontent.com/u/5277173?v=4","gravatar_id":"","url":"https://api.github.com/users/evilezh","html_url":"https://github.com/evilezh","followers_url":"https://api.github.com/users/evilezh/followers","following_url":"https://api.github.com/users/evilezh/following{/other_user}","gists_url":"https://api.github.com/users/evilezh/gists{/gist_id}","starred_url":"https://api.github.com/users/evilezh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/evilezh/subscriptions","organizations_url":"https://api.github.com/users/evilezh/orgs","repos_url":"https://api.github.com/users/evilezh/repos","events_url":"https://api.github.com/users/evilezh/events{/privacy}","received_events_url":"https://api.github.com/users/evilezh/received_events","type":"User","site_admin":false},"created_at":"2018-11-08T20:41:23Z","updated_at":"2018-11-08T20:41:23Z","author_association":"NONE","body":"@yvespp I had similar issues. So - I took time and rewrote deployment for myself.\r\nHere is some stuff I changed in config file:\r\n```\r\nproxy-request-buffering: \"off\"\r\nworker-shutdown-timeout: \"60s\"\r\nuse-geoip: \"false\"\r\n```\r\nand to startup I added:\r\n```\r\n--enable-dynamic-certificates\r\n--enable-ssl-chain-completion=false\r\n```\r\nI did not test which of those extra options solved my issues, I've suspicion about  `proxy-request-buffering: \"off\"`.  I'm running latest 0.20.0.\r\nIf that does not help, I can share all options I've configured."}}
{"actor":{"display_login":"evilezh"}}