{"action":"created","issue":{"url":"https://api.github.com/repos/minio/minio/issues/5994","repository_url":"https://api.github.com/repos/minio/minio","labels_url":"https://api.github.com/repos/minio/minio/issues/5994/labels{/name}","comments_url":"https://api.github.com/repos/minio/minio/issues/5994/comments","events_url":"https://api.github.com/repos/minio/minio/issues/5994/events","html_url":"https://github.com/minio/minio/issues/5994","id":327766856,"node_id":"MDU6SXNzdWUzMjc3NjY4NTY=","number":5994,"title":"Minio backend is killed by OOM under heavy load with errors","user":{"login":"ordahan","id":2500130,"node_id":"MDQ6VXNlcjI1MDAxMzA=","avatar_url":"https://avatars0.githubusercontent.com/u/2500130?v=4","gravatar_id":"","url":"https://api.github.com/users/ordahan","html_url":"https://github.com/ordahan","followers_url":"https://api.github.com/users/ordahan/followers","following_url":"https://api.github.com/users/ordahan/following{/other_user}","gists_url":"https://api.github.com/users/ordahan/gists{/gist_id}","starred_url":"https://api.github.com/users/ordahan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ordahan/subscriptions","organizations_url":"https://api.github.com/users/ordahan/orgs","repos_url":"https://api.github.com/users/ordahan/repos","events_url":"https://api.github.com/users/ordahan/events{/privacy}","received_events_url":"https://api.github.com/users/ordahan/received_events","type":"User","site_admin":false},"labels":[{"id":574010253,"node_id":"MDU6TGFiZWw1NzQwMTAyNTM=","url":"https://api.github.com/repos/minio/minio/labels/priority:%20medium","name":"priority: medium","color":"fbca04","default":false}],"state":"closed","locked":false,"assignee":{"login":"harshavardhana","id":622699,"node_id":"MDQ6VXNlcjYyMjY5OQ==","avatar_url":"https://avatars0.githubusercontent.com/u/622699?v=4","gravatar_id":"","url":"https://api.github.com/users/harshavardhana","html_url":"https://github.com/harshavardhana","followers_url":"https://api.github.com/users/harshavardhana/followers","following_url":"https://api.github.com/users/harshavardhana/following{/other_user}","gists_url":"https://api.github.com/users/harshavardhana/gists{/gist_id}","starred_url":"https://api.github.com/users/harshavardhana/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/harshavardhana/subscriptions","organizations_url":"https://api.github.com/users/harshavardhana/orgs","repos_url":"https://api.github.com/users/harshavardhana/repos","events_url":"https://api.github.com/users/harshavardhana/events{/privacy}","received_events_url":"https://api.github.com/users/harshavardhana/received_events","type":"User","site_admin":false},"assignees":[{"login":"harshavardhana","id":622699,"node_id":"MDQ6VXNlcjYyMjY5OQ==","avatar_url":"https://avatars0.githubusercontent.com/u/622699?v=4","gravatar_id":"","url":"https://api.github.com/users/harshavardhana","html_url":"https://github.com/harshavardhana","followers_url":"https://api.github.com/users/harshavardhana/followers","following_url":"https://api.github.com/users/harshavardhana/following{/other_user}","gists_url":"https://api.github.com/users/harshavardhana/gists{/gist_id}","starred_url":"https://api.github.com/users/harshavardhana/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/harshavardhana/subscriptions","organizations_url":"https://api.github.com/users/harshavardhana/orgs","repos_url":"https://api.github.com/users/harshavardhana/repos","events_url":"https://api.github.com/users/harshavardhana/events{/privacy}","received_events_url":"https://api.github.com/users/harshavardhana/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/minio/minio/milestones/15","html_url":"https://github.com/minio/minio/milestone/15","labels_url":"https://api.github.com/repos/minio/minio/milestones/15/labels","id":2426553,"node_id":"MDk6TWlsZXN0b25lMjQyNjU1Mw==","number":15,"title":"Next Release","description":" Rolling releases","creator":{"login":"deekoder","id":1700901,"node_id":"MDQ6VXNlcjE3MDA5MDE=","avatar_url":"https://avatars1.githubusercontent.com/u/1700901?v=4","gravatar_id":"","url":"https://api.github.com/users/deekoder","html_url":"https://github.com/deekoder","followers_url":"https://api.github.com/users/deekoder/followers","following_url":"https://api.github.com/users/deekoder/following{/other_user}","gists_url":"https://api.github.com/users/deekoder/gists{/gist_id}","starred_url":"https://api.github.com/users/deekoder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deekoder/subscriptions","organizations_url":"https://api.github.com/users/deekoder/orgs","repos_url":"https://api.github.com/users/deekoder/repos","events_url":"https://api.github.com/users/deekoder/events{/privacy}","received_events_url":"https://api.github.com/users/deekoder/received_events","type":"User","site_admin":false},"open_issues":53,"closed_issues":592,"state":"open","created_at":"2017-04-01T00:36:55Z","updated_at":"2018-07-02T08:00:28Z","due_on":"2018-06-30T07:00:00Z","closed_at":null},"comments":9,"created_at":"2018-05-30T14:50:17Z","updated_at":"2018-07-02T08:00:28Z","closed_at":"2018-07-02T08:00:28Z","author_association":"NONE","body":"<!--- Provide a general summary of the issue in the Title above -->\r\n\r\n## Expected Behavior\r\n<!--- If you're describing a bug, tell us what should happen -->\r\n<!--- If you're suggesting a change/improvement, tell us how it should work -->\r\nMinio should keep a pretty stable memory footprint regardless of the user IO pattern, it should not leak / accumulate memory when presented with a lot of requests that subsequently fail due to bad server / client network.\r\n\r\n## Current Behavior\r\n<!--- If describing a bug, tell us what happens instead of the expected behavior -->\r\n<!--- If suggesting a change/improvement, explain the difference from current behavior -->\r\nAfter about a minute in which minio got a lot of invalid PUT requests, the kernel killed the process because it used too much memory (14GB):\r\n```\r\nMay  9 18:35:54 REG-61081-V31MMOBS00-63695-minio-10 kernel: [ 3398.041942] Out of memory: Kill process 23098 (minio) score 934 or sacrifice child\r\nMay  9 18:35:54 REG-61081-V31MMOBS00-63695-minio-10 kernel: [ 3398.042908] Killed process 23098 (minio) total-vm:14506184kB, anon-rss:14365508kB, file-rss:0kB, shmem-rss:0kB\r\n```\r\n\r\nThis was the repeated backtrace for the errors, taken from the syslog:\r\n```\r\nMay  9 18:35:53 REG-61081-V31MMOBS00-63695-minio-10 minio: Error: unexpected EOF\r\nMay  9 18:35:53 REG-61081-V31MMOBS00-63695-minio-10 minio: 1: cmd/erasure-createfile.go:60:cmd.(*ErasureStorage).CreateFile()\r\nMay  9 18:35:53 REG-61081-V31MMOBS00-63695-minio-10 minio: 2: cmd/xl-v1-object.go:644:cmd.xlObjects.putObject()\r\nMay  9 18:35:53 REG-61081-V31MMOBS00-63695-minio-10 minio: 3: cmd/xl-v1-object.go:520:cmd.xlObjects.PutObject()\r\nMay  9 18:35:53 REG-61081-V31MMOBS00-63695-minio-10 minio: 4: cmd/xl-sets.go:560:cmd.(*xlSets).PutObject()\r\nMay  9 18:35:53 REG-61081-V31MMOBS00-63695-minio-10 minio: 5: cmd/object-handlers.go:669:cmd.(ObjectLayer).PutObject-fm()\r\nMay  9 18:35:53 REG-61081-V31MMOBS00-63695-minio-10 minio: 6: cmd/object-handlers.go:746:cmd.objectAPIHandlers.PutObjectHandler()\r\nMay  9 18:35:53 REG-61081-V31MMOBS00-63695-minio-10 minio: 7: cmd/api-router.go:77:cmd.(objectAPIHandlers).PutObjectHandler-fm()\r\nMay  9 18:35:53 REG-61081-V31MMOBS00-63695-minio-10 minio: 8: pkg/handlers/http-tracer.go:149:handlers.TraceReqHandlerFunc.func1()\r\nMay  9 18:35:53 REG-61081-V31MMOBS00-63695-minio-10 minio: 9: net/http/server.go:1947:http.HandlerFunc.ServeHTTP()\r\n```\r\n## Steps to Reproduce (for bugs)\r\n<!--- Provide a link to a live example, or an unambiguous set of steps to -->\r\n<!--- reproduce this bug. Include code to reproduce, if relevant -->\r\n1. Run a distributed minio cluster \r\n2. Send invalid HTTP requests with an incorrect body length (or no body at all, when the content-length is non zero)\r\n3. Wait for the Minio server to get killed by the kernel (oom killer)\r\n\r\n## Your Environment\r\n<!--- Include as many relevant details about the environment you experienced the bug in -->\r\n* Version used (`minio version`):\r\nDon't have the version in my logs, but it was whatever was served on https://dl.minio.io/server/minio/release/linux-amd64/minio at 2018-05-09 18:20:25 (UTC)\r\n* Operating System and version (`uname -a`):\r\n`May  9 18:20:19 REG-61081-V31MMOBS00-63695-minio-10 kernel: [    0.000000] Linux version 3.10.0-514.26.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Re\r\nd Hat 4.8.5-11) (GCC) ) #1 SMP Tue Jul 4 15:04:05 UTC 2017`\r\n"},"comment":{"url":"https://api.github.com/repos/minio/minio/issues/comments/401703500","html_url":"https://github.com/minio/minio/issues/5994#issuecomment-401703500","issue_url":"https://api.github.com/repos/minio/minio/issues/5994","id":401703500,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMTcwMzUwMA==","user":{"login":"ordahan","id":2500130,"node_id":"MDQ6VXNlcjI1MDAxMzA=","avatar_url":"https://avatars0.githubusercontent.com/u/2500130?v=4","gravatar_id":"","url":"https://api.github.com/users/ordahan","html_url":"https://github.com/ordahan","followers_url":"https://api.github.com/users/ordahan/followers","following_url":"https://api.github.com/users/ordahan/following{/other_user}","gists_url":"https://api.github.com/users/ordahan/gists{/gist_id}","starred_url":"https://api.github.com/users/ordahan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ordahan/subscriptions","organizations_url":"https://api.github.com/users/ordahan/orgs","repos_url":"https://api.github.com/users/ordahan/repos","events_url":"https://api.github.com/users/ordahan/events{/privacy}","received_events_url":"https://api.github.com/users/ordahan/received_events","type":"User","site_admin":false},"created_at":"2018-07-02T08:00:28Z","updated_at":"2018-07-02T08:00:28Z","author_association":"NONE","body":"@harshavardhana i'm testing the latest release now\r\n500 errors are definitely in place for these content-length issues, I mistakenly thought that requests with a correct content-length were being rejected as well."}}
{"actor":{"display_login":"ordahan"}}