{"action":"created","issue":{"url":"https://api.github.com/repos/dgraph-io/dgraph/issues/2254","repository_url":"https://api.github.com/repos/dgraph-io/dgraph","labels_url":"https://api.github.com/repos/dgraph-io/dgraph/issues/2254/labels{/name}","comments_url":"https://api.github.com/repos/dgraph-io/dgraph/issues/2254/comments","events_url":"https://api.github.com/repos/dgraph-io/dgraph/issues/2254/events","html_url":"https://github.com/dgraph-io/dgraph/issues/2254","id":307753534,"number":2254,"title":"OOM leading to stuck servers","user":{"login":"Levatius","id":35065284,"avatar_url":"https://avatars0.githubusercontent.com/u/35065284?v=4","gravatar_id":"","url":"https://api.github.com/users/Levatius","html_url":"https://github.com/Levatius","followers_url":"https://api.github.com/users/Levatius/followers","following_url":"https://api.github.com/users/Levatius/following{/other_user}","gists_url":"https://api.github.com/users/Levatius/gists{/gist_id}","starred_url":"https://api.github.com/users/Levatius/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Levatius/subscriptions","organizations_url":"https://api.github.com/users/Levatius/orgs","repos_url":"https://api.github.com/users/Levatius/repos","events_url":"https://api.github.com/users/Levatius/events{/privacy}","received_events_url":"https://api.github.com/users/Levatius/received_events","type":"User","site_admin":false},"labels":[{"id":252336064,"url":"https://api.github.com/repos/dgraph-io/dgraph/labels/bug","name":"bug","color":"fc2929","default":true},{"id":575130195,"url":"https://api.github.com/repos/dgraph-io/dgraph/labels/question","name":"question","color":"bfd4f2","default":true}],"state":"open","locked":false,"assignee":{"login":"pawanrawal","id":7247941,"avatar_url":"https://avatars1.githubusercontent.com/u/7247941?v=4","gravatar_id":"","url":"https://api.github.com/users/pawanrawal","html_url":"https://github.com/pawanrawal","followers_url":"https://api.github.com/users/pawanrawal/followers","following_url":"https://api.github.com/users/pawanrawal/following{/other_user}","gists_url":"https://api.github.com/users/pawanrawal/gists{/gist_id}","starred_url":"https://api.github.com/users/pawanrawal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pawanrawal/subscriptions","organizations_url":"https://api.github.com/users/pawanrawal/orgs","repos_url":"https://api.github.com/users/pawanrawal/repos","events_url":"https://api.github.com/users/pawanrawal/events{/privacy}","received_events_url":"https://api.github.com/users/pawanrawal/received_events","type":"User","site_admin":false},"assignees":[{"login":"pawanrawal","id":7247941,"avatar_url":"https://avatars1.githubusercontent.com/u/7247941?v=4","gravatar_id":"","url":"https://api.github.com/users/pawanrawal","html_url":"https://github.com/pawanrawal","followers_url":"https://api.github.com/users/pawanrawal/followers","following_url":"https://api.github.com/users/pawanrawal/following{/other_user}","gists_url":"https://api.github.com/users/pawanrawal/gists{/gist_id}","starred_url":"https://api.github.com/users/pawanrawal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pawanrawal/subscriptions","organizations_url":"https://api.github.com/users/pawanrawal/orgs","repos_url":"https://api.github.com/users/pawanrawal/repos","events_url":"https://api.github.com/users/pawanrawal/events{/privacy}","received_events_url":"https://api.github.com/users/pawanrawal/received_events","type":"User","site_admin":false}],"milestone":null,"comments":16,"created_at":"2018-03-22T17:59:04Z","updated_at":"2018-04-10T01:29:12Z","closed_at":null,"author_association":"NONE","body":"**Cluster info:**\n- Kubernetes: 8 nodes total, each 32GB memory.\n- Dgraph: 1 zero and 3 servers, each runs on a separate node. Running master image. Memory flag for servers set to 24GB.\n\n**Observed issue:**\nLinear increase in memory usage for each server. When a server hits the request limit (30GB) it restarts. This at first recovers fine but around the 3rd spike it fails to restart and gets _stuck_.\nThere are two ways the pipeline can be stuck:\n1. Incoming mutates timeout\n2. No connection to the servers can be made\n\nThese scenarios seem mutually exclusive (if no connection can be made no mutate can be attempted).\n\n**Questions:**\n- Why does memory usage continue past the 24GB flag?\n- Is 24GB not enough for each server or is it failing to write items to disk?\n\n**Example:**\nServer 0 gets stuck at the 3rd memory spike.\n[logs_server-0.txt](https://github.com/dgraph-io/dgraph/files/1838605/logs_server-0.txt)\n![server-0](https://user-images.githubusercontent.com/35065284/37784696-56b47d74-2df0-11e8-8eb0-8ae232fa3d05.PNG)\n[logs_server-1.txt](https://github.com/dgraph-io/dgraph/files/1838606/logs_server-1.txt)\n![server-1](https://user-images.githubusercontent.com/35065284/37784698-58236f8a-2df0-11e8-984c-58f212a3bf94.PNG)\n[logs_server-2.txt](https://github.com/dgraph-io/dgraph/files/1838607/logs_server-2.txt)\n![server-2](https://user-images.githubusercontent.com/35065284/37784703-59520f1a-2df0-11e8-8490-bdd65323d405.PNG)\n[logs_zero.txt](https://github.com/dgraph-io/dgraph/files/1838610/logs_zero.txt)\n\nI had some trouble getting the heap profile unfortunately, it was returning a malformed HTTP response. It is probably something our end at fault, will try again."},"comment":{"url":"https://api.github.com/repos/dgraph-io/dgraph/issues/comments/379944607","html_url":"https://github.com/dgraph-io/dgraph/issues/2254#issuecomment-379944607","issue_url":"https://api.github.com/repos/dgraph-io/dgraph/issues/2254","id":379944607,"user":{"login":"pawanrawal","id":7247941,"avatar_url":"https://avatars1.githubusercontent.com/u/7247941?v=4","gravatar_id":"","url":"https://api.github.com/users/pawanrawal","html_url":"https://github.com/pawanrawal","followers_url":"https://api.github.com/users/pawanrawal/followers","following_url":"https://api.github.com/users/pawanrawal/following{/other_user}","gists_url":"https://api.github.com/users/pawanrawal/gists{/gist_id}","starred_url":"https://api.github.com/users/pawanrawal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pawanrawal/subscriptions","organizations_url":"https://api.github.com/users/pawanrawal/orgs","repos_url":"https://api.github.com/users/pawanrawal/repos","events_url":"https://api.github.com/users/pawanrawal/events{/privacy}","received_events_url":"https://api.github.com/users/pawanrawal/received_events","type":"User","site_admin":false},"created_at":"2018-04-10T01:29:11Z","updated_at":"2018-04-10T01:29:11Z","author_association":"MEMBER","body":"> 2018/04/09 15:23:56 draft.go:808: Couldn't take snapshot, txn watermark: [673443], applied watermark: [2425895]\r\n\r\nSee if it happens with the latest `dev` image. If it does then please share the debug logs created by Server and Zero."}}
{"actor":{"display_login":"pawanrawal"}}