{"action":"created","issue":{"url":"https://api.github.com/repos/arangodb/arangodb/issues/2080","repository_url":"https://api.github.com/repos/arangodb/arangodb","labels_url":"https://api.github.com/repos/arangodb/arangodb/issues/2080/labels{/name}","comments_url":"https://api.github.com/repos/arangodb/arangodb/issues/2080/comments","events_url":"https://api.github.com/repos/arangodb/arangodb/issues/2080/events","html_url":"https://github.com/arangodb/arangodb/issues/2080","id":179887786,"number":2080,"title":"Memory \"Pain point\" ;)","user":{"login":"frankmayer","id":825497,"avatar_url":"https://avatars.githubusercontent.com/u/825497?v=3","gravatar_id":"","url":"https://api.github.com/users/frankmayer","html_url":"https://github.com/frankmayer","followers_url":"https://api.github.com/users/frankmayer/followers","following_url":"https://api.github.com/users/frankmayer/following{/other_user}","gists_url":"https://api.github.com/users/frankmayer/gists{/gist_id}","starred_url":"https://api.github.com/users/frankmayer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/frankmayer/subscriptions","organizations_url":"https://api.github.com/users/frankmayer/orgs","repos_url":"https://api.github.com/users/frankmayer/repos","events_url":"https://api.github.com/users/frankmayer/events{/privacy}","received_events_url":"https://api.github.com/users/frankmayer/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-09-28T21:02:29Z","updated_at":"2016-10-07T08:04:19Z","closed_at":null,"body":"I am currently evaluating ArangoDB for a specific use case, which essentially boils down to importing **OpenStreetMap** data into Arango. The final schema has not been decided upon, yet. I am evaluating a simple table-to-collection import schema, as well as a graph schema where osm ways partly become edges - connecting the osm nodes. I have also tested some in-between mixed scenarios. Besides the OSM data, there would also be custom data in the same format as OSM data, but private.\r\n\r\nSo, what's the memory pain point? Well, I grabbed that phrase off the conclusion paragraph of your benchmark post [here](https://www.arangodb.com/2015/10/benchmark-postgresql-mongodb-arangodb/) :smile: \r\n\r\nHere are the issues I am facing:\r\n\r\n1. In OSM, the keys in the various objects are very often repeated and there are a lot of them - see [here](http://taginfo.openstreetmap.org/keys). This leads to a lot of key names wasting a lot of precious space/memory.\r\nFor a test, I did an export of all collections and ran them through the velocypack tool json-to-vpack and achieved a 35%-45% reduction in size, using the `--compress` and `--no-compact` parameters. That is, as I understand it, mostly or only key compression using a lookup table. \r\nOverall the reduction will most possibly be higher because the usage of the keys is below average in this specific dataset, compared to the whole planet. \r\nThe compression capability would even be more important if a graph design would be chosen. To be honest I'd like to go mostly graph, but the storage needs for data and indexes are too much.\r\n\r\n  So, I guess a feature request would be to be able to use the compression capabilities of velocypack.\r\n\r\n2. Also, I would like to ask if there is something on your roadmap to reduce memory consumption for edges (and including index memory reduction in general) a bit. \r\nAn example: for an edge collection with ~16.000.000 documents which has a **data size of 1.89GB**, the **index size is 2.37GB** with only the standard edge indexes _key & _from,_to. That is quiet huge.\r\n\r\n3. Besides the key compression discussed before, it would also be of interest if general compression on data and/or index would be possible - or at least don't have the need to have all data in memory (there was some mention of that approach in said benchmark post).\r\n\r\nThank you\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/arangodb/arangodb/issues/comments/252177880","html_url":"https://github.com/arangodb/arangodb/issues/2080#issuecomment-252177880","issue_url":"https://api.github.com/repos/arangodb/arangodb/issues/2080","id":252177880,"user":{"login":"frankmayer","id":825497,"avatar_url":"https://avatars.githubusercontent.com/u/825497?v=3","gravatar_id":"","url":"https://api.github.com/users/frankmayer","html_url":"https://github.com/frankmayer","followers_url":"https://api.github.com/users/frankmayer/followers","following_url":"https://api.github.com/users/frankmayer/following{/other_user}","gists_url":"https://api.github.com/users/frankmayer/gists{/gist_id}","starred_url":"https://api.github.com/users/frankmayer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/frankmayer/subscriptions","organizations_url":"https://api.github.com/users/frankmayer/orgs","repos_url":"https://api.github.com/users/frankmayer/repos","events_url":"https://api.github.com/users/frankmayer/events{/privacy}","received_events_url":"https://api.github.com/users/frankmayer/received_events","type":"User","site_admin":false},"created_at":"2016-10-07T08:04:19Z","updated_at":"2016-10-07T08:04:19Z","body":"Bump :smile: "}}
{"actor":{"display_login":"frankmayer"}}