{"action":"created","issue":{"url":"https://api.github.com/repos/blazegraph/database/issues/108","repository_url":"https://api.github.com/repos/blazegraph/database","labels_url":"https://api.github.com/repos/blazegraph/database/issues/108/labels{/name}","comments_url":"https://api.github.com/repos/blazegraph/database/issues/108/comments","events_url":"https://api.github.com/repos/blazegraph/database/issues/108/events","html_url":"https://github.com/blazegraph/database/issues/108","id":386989684,"node_id":"MDU6SXNzdWUzODY5ODk2ODQ=","number":108,"title":"Optimizing very large counting query with OOM","user":{"login":"nyurik","id":1641515,"node_id":"MDQ6VXNlcjE2NDE1MTU=","avatar_url":"https://avatars0.githubusercontent.com/u/1641515?v=4","gravatar_id":"","url":"https://api.github.com/users/nyurik","html_url":"https://github.com/nyurik","followers_url":"https://api.github.com/users/nyurik/followers","following_url":"https://api.github.com/users/nyurik/following{/other_user}","gists_url":"https://api.github.com/users/nyurik/gists{/gist_id}","starred_url":"https://api.github.com/users/nyurik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nyurik/subscriptions","organizations_url":"https://api.github.com/users/nyurik/orgs","repos_url":"https://api.github.com/users/nyurik/repos","events_url":"https://api.github.com/users/nyurik/events{/privacy}","received_events_url":"https://api.github.com/users/nyurik/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2018-12-03T20:23:28Z","updated_at":"2018-12-07T19:54:06Z","closed_at":null,"author_association":"NONE","body":"I am trying to count number of items per group. There are about 45,000 groups, and the total number of items is in billions.  Ideally, internally this query should establish a hashmap with counts, thus allocating under a 100,000 integer counters, yet it seems it tries to generate a full list of items for each bucket, and obviously running out of memory. Is there a way to optimize this, or should this be a feature request for Blazegraph? \r\n\r\n```\r\nSELECT ?osmt (count(*) as ?count) where {\r\n  ?s1 osmm:key ?osmt.\r\n  ?s2 ?osmt ?v.\r\n} group by ?osmt\r\n```"},"comment":{"url":"https://api.github.com/repos/blazegraph/database/issues/comments/445346980","html_url":"https://github.com/blazegraph/database/issues/108#issuecomment-445346980","issue_url":"https://api.github.com/repos/blazegraph/database/issues/108","id":445346980,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NTM0Njk4MA==","user":{"login":"thompsonbry","id":2584544,"node_id":"MDQ6VXNlcjI1ODQ1NDQ=","avatar_url":"https://avatars1.githubusercontent.com/u/2584544?v=4","gravatar_id":"","url":"https://api.github.com/users/thompsonbry","html_url":"https://github.com/thompsonbry","followers_url":"https://api.github.com/users/thompsonbry/followers","following_url":"https://api.github.com/users/thompsonbry/following{/other_user}","gists_url":"https://api.github.com/users/thompsonbry/gists{/gist_id}","starred_url":"https://api.github.com/users/thompsonbry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thompsonbry/subscriptions","organizations_url":"https://api.github.com/users/thompsonbry/orgs","repos_url":"https://api.github.com/users/thompsonbry/repos","events_url":"https://api.github.com/users/thompsonbry/events{/privacy}","received_events_url":"https://api.github.com/users/thompsonbry/received_events","type":"User","site_admin":false},"created_at":"2018-12-07T19:54:06Z","updated_at":"2018-12-07T19:54:06Z","author_association":"CONTRIBUTOR","body":"There might be very little native memory available by default. This could\nlead me to an OOM of the native heap unless you explicitly allow direct\nmemory to be used by the JVM\n\nOn Fri, Dec 7, 2018 at 11:52 AM Bryan Thompson <bryan@blazegraph.com> wrote:\n\n> It is out of native memory. The error is from the memory manager. I do not\n> see the option to grant the JVM the ability to use native memory ?  Just an\n> option to limit it to 1G native memory per query.\n>\n> @stas: could you make sure that the best practices are captured for this\n> and for diagnosing the OOM as Java heap, GC OH exceeded, or native heap?\n>\n> Thanks,\n> Bryan\n>\n> On Fri, Dec 7, 2018 at 11:25 AM Yuri Astrakhan <notifications@github.com>\n> wrote:\n>\n>> Also, I was mistaken - distinct ?osmt is only 4,661, not 45,000 i\n>> initially thought, making this issue even stranger.\n>>\n>> SELECT (count(distinct ?osmt) as ?count) where {\n>>   ?s1 osmm:key ?osmt.\n>> }\n>>\n>> —\n>> You are receiving this because you were mentioned.\n>>\n>>\n>> Reply to this email directly, view it on GitHub\n>> <https://github.com/blazegraph/database/issues/108#issuecomment-445338947>,\n>> or mute the thread\n>> <https://github.com/notifications/unsubscribe-auth/ACdv4OMA6KuHwxyABjVHmnVMcj_6tA2Gks5u2sC8gaJpZM4Y_WIs>\n>> .\n>>\n>\n"}}
{"actor":{"display_login":"thompsonbry"}}