{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/appengine-mapreduce/issues/95","repository_url":"https://api.github.com/repos/GoogleCloudPlatform/appengine-mapreduce","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/appengine-mapreduce/issues/95/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/appengine-mapreduce/issues/95/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/appengine-mapreduce/issues/95/events","html_url":"https://github.com/GoogleCloudPlatform/appengine-mapreduce/issues/95","id":158573513,"number":95,"title":"out of memory in shuffle stage","user":{"login":"mayankbhagya","id":557860,"avatar_url":"https://avatars.githubusercontent.com/u/557860?v=3","gravatar_id":"","url":"https://api.github.com/users/mayankbhagya","html_url":"https://github.com/mayankbhagya","followers_url":"https://api.github.com/users/mayankbhagya/followers","following_url":"https://api.github.com/users/mayankbhagya/following{/other_user}","gists_url":"https://api.github.com/users/mayankbhagya/gists{/gist_id}","starred_url":"https://api.github.com/users/mayankbhagya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mayankbhagya/subscriptions","organizations_url":"https://api.github.com/users/mayankbhagya/orgs","repos_url":"https://api.github.com/users/mayankbhagya/repos","events_url":"https://api.github.com/users/mayankbhagya/events{/privacy}","received_events_url":"https://api.github.com/users/mayankbhagya/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2016-06-05T20:43:22Z","updated_at":"2016-06-08T22:10:36Z","closed_at":null,"body":"I have ~50M entities stored in datastore. Each item can be of one type out of a total 7 types.\r\n\r\nNext, I have a simple MapReduce job that counts the number of items of each type. The Mapper emits (type, 1). The reducer simply adds the number of 1s received for each type.\r\n\r\nWhen I run this job with 5000 shards, the map-stage runs fine. It uses a total of 20 instances which is maximum possible based on my task-queue configuration.\r\n\r\nHowever, the shuffle-hash stage makes use of only one instance and fails with an out-of-memory error. I am not able to understand why only one instance is being used for hashing and how can I fix this out-of-memory error in /mapreduce/kickoffjob_callback/15714734************.\r\n\r\nI have tried writing a combiner but I never saw a combiner stage on the mapreduce status page or in the logs."},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/appengine-mapreduce/issues/comments/224745444","html_url":"https://github.com/GoogleCloudPlatform/appengine-mapreduce/issues/95#issuecomment-224745444","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/appengine-mapreduce/issues/95","id":224745444,"user":{"login":"ZiglioNZ","id":1219039,"avatar_url":"https://avatars.githubusercontent.com/u/1219039?v=3","gravatar_id":"","url":"https://api.github.com/users/ZiglioNZ","html_url":"https://github.com/ZiglioNZ","followers_url":"https://api.github.com/users/ZiglioNZ/followers","following_url":"https://api.github.com/users/ZiglioNZ/following{/other_user}","gists_url":"https://api.github.com/users/ZiglioNZ/gists{/gist_id}","starred_url":"https://api.github.com/users/ZiglioNZ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ZiglioNZ/subscriptions","organizations_url":"https://api.github.com/users/ZiglioNZ/orgs","repos_url":"https://api.github.com/users/ZiglioNZ/repos","events_url":"https://api.github.com/users/ZiglioNZ/events{/privacy}","received_events_url":"https://api.github.com/users/ZiglioNZ/received_events","type":"User","site_admin":false},"created_at":"2016-06-08T22:10:36Z","updated_at":"2016-06-08T22:10:36Z","body":"Thanks, luckily I use Java then ;-)"}}
{"actor":{"display_login":"ZiglioNZ"}}