{"action":"opened","issue":{"url":"https://api.github.com/repos/tdebatty/spark-knn-graphs/issues/15","repository_url":"https://api.github.com/repos/tdebatty/spark-knn-graphs","labels_url":"https://api.github.com/repos/tdebatty/spark-knn-graphs/issues/15/labels{/name}","comments_url":"https://api.github.com/repos/tdebatty/spark-knn-graphs/issues/15/comments","events_url":"https://api.github.com/repos/tdebatty/spark-knn-graphs/issues/15/events","html_url":"https://github.com/tdebatty/spark-knn-graphs/issues/15","id":343786531,"node_id":"MDU6SXNzdWUzNDM3ODY1MzE=","number":15,"title":"OutOfMemoryError: GC overhead limit exceeded","user":{"login":"fvictorio","id":417134,"node_id":"MDQ6VXNlcjQxNzEzNA==","avatar_url":"https://avatars1.githubusercontent.com/u/417134?v=4","gravatar_id":"","url":"https://api.github.com/users/fvictorio","html_url":"https://github.com/fvictorio","followers_url":"https://api.github.com/users/fvictorio/followers","following_url":"https://api.github.com/users/fvictorio/following{/other_user}","gists_url":"https://api.github.com/users/fvictorio/gists{/gist_id}","starred_url":"https://api.github.com/users/fvictorio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fvictorio/subscriptions","organizations_url":"https://api.github.com/users/fvictorio/orgs","repos_url":"https://api.github.com/users/fvictorio/repos","events_url":"https://api.github.com/users/fvictorio/events{/privacy}","received_events_url":"https://api.github.com/users/fvictorio/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-07-23T20:37:44Z","updated_at":"2018-07-23T20:37:44Z","closed_at":null,"author_association":"NONE","body":"Hi, I'm trying to make a graph of a subset of the EMNIST dataset (10K instances) and I'm getting an `OutOfMemoryError`:\r\n\r\n```\r\njava.lang.OutOfMemoryError: GC overhead limit exceeded\r\n```\r\n\r\nI tried to make a MWE to reproduce this, you can find it here: https://github.com/fvictorio/spark-knn-graphs-outofmemory\r\n\r\nI tried both locally and in google cloud (with four workers with 15GB of RAM each).\r\n\r\nIt's very likely that I'm doing something wrong, since I read in another issue that you tested the library with a dataset with millions of rows. But maybe the large amount of dimensions is causing trouble?\r\n\r\nThanks."}}
{"actor":{"display_login":"fvictorio"}}