{"action":"created","issue":{"url":"https://api.github.com/repos/akheron/jansson/issues/332","repository_url":"https://api.github.com/repos/akheron/jansson","labels_url":"https://api.github.com/repos/akheron/jansson/issues/332/labels{/name}","comments_url":"https://api.github.com/repos/akheron/jansson/issues/332/comments","events_url":"https://api.github.com/repos/akheron/jansson/issues/332/events","html_url":"https://github.com/akheron/jansson/issues/332","id":214027323,"node_id":"MDU6SXNzdWUyMTQwMjczMjM=","number":332,"title":"super high memory usage","user":{"login":"mrexodia","id":2458265,"node_id":"MDQ6VXNlcjI0NTgyNjU=","avatar_url":"https://avatars2.githubusercontent.com/u/2458265?v=4","gravatar_id":"","url":"https://api.github.com/users/mrexodia","html_url":"https://github.com/mrexodia","followers_url":"https://api.github.com/users/mrexodia/followers","following_url":"https://api.github.com/users/mrexodia/following{/other_user}","gists_url":"https://api.github.com/users/mrexodia/gists{/gist_id}","starred_url":"https://api.github.com/users/mrexodia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrexodia/subscriptions","organizations_url":"https://api.github.com/users/mrexodia/orgs","repos_url":"https://api.github.com/users/mrexodia/repos","events_url":"https://api.github.com/users/mrexodia/events{/privacy}","received_events_url":"https://api.github.com/users/mrexodia/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-03-14T10:27:00Z","updated_at":"2018-11-25T11:17:26Z","closed_at":null,"author_association":"NONE","body":"I am testing on [test_mini.json](https://mega.nz/#!js5mmaiZ!7C9lG-8Ok95OOH7hsmuafXPrJ0EsEE6tnrsQJY4fdmo) which is a minified JSON file of 76.3mb in size. Here is my code:\r\n\r\n```c++\r\n#include \"jansson/jansson.h\"\r\n\r\n#pragma comment(lib, \"jansson\\\\jansson_x86.lib\")\r\n\r\nstatic size_t memPool = 0, allocs = 0, frees = 0;\r\n\r\nstatic void* json_malloc(size_t size)\r\n{\r\n    auto ptr = (size_t*)malloc(size + sizeof(size_t));\r\n    *ptr = size;\r\n    memPool += size;\r\n    allocs++;\r\n    return ptr + 1;\r\n}\r\n\r\nstatic void json_free(void* ptr)\r\n{\r\n    auto sptr = (size_t*)ptr;\r\n    sptr--;\r\n    memPool -= *sptr;\r\n    frees++;\r\n    free(sptr);\r\n}\r\n\r\nint main()\r\n{\r\n    json_set_alloc_funcs(json_malloc, json_free);\r\n    auto json = json_load_file(\"test.json\", 0, 0);\r\n    printf(\"json: %p, memPool: %u bytes, allocs: %u, frees: %u\\n\", json, memPool, allocs, frees);\r\n    getchar();\r\n    json_decref(json);\r\n    printf(\"json: %p, memPool: %u bytes, allocs: %u, frees: %u\\n\", json, memPool, allocs, frees);\r\n    getchar();\r\n}\r\n```\r\n\r\nThe output is:\r\n\r\n```\r\njson: 00F638B4, memPool: 433354626 bytes, allocs: 19975350, frees: 4448313\r\njson: 00F638B4, memPool: 0 bytes, allocs: 19975350, frees: 19975350\r\n```\r\n\r\nFor some reason jansson wants `413.3mb` of memory(!) which is is `5.4` times the size of the file.\r\n\r\nWhen removing the `size_t` overhead from the allocation (comment out `json_set_alloc_funcs`) the actual memory usage reported by Windows is `553.8mb` (`76.3+413.3=489.6`, the rest is probably runtime).\r\n\r\n![mem usage](https://i.imgur.com/HaaSuOq.png)\r\n\r\nIs there a way around this?"},"comment":{"url":"https://api.github.com/repos/akheron/jansson/issues/comments/441432866","html_url":"https://github.com/akheron/jansson/issues/332#issuecomment-441432866","issue_url":"https://api.github.com/repos/akheron/jansson/issues/332","id":441432866,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MTQzMjg2Ng==","user":{"login":"mrexodia","id":2458265,"node_id":"MDQ6VXNlcjI0NTgyNjU=","avatar_url":"https://avatars2.githubusercontent.com/u/2458265?v=4","gravatar_id":"","url":"https://api.github.com/users/mrexodia","html_url":"https://github.com/mrexodia","followers_url":"https://api.github.com/users/mrexodia/followers","following_url":"https://api.github.com/users/mrexodia/following{/other_user}","gists_url":"https://api.github.com/users/mrexodia/gists{/gist_id}","starred_url":"https://api.github.com/users/mrexodia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrexodia/subscriptions","organizations_url":"https://api.github.com/users/mrexodia/orgs","repos_url":"https://api.github.com/users/mrexodia/repos","events_url":"https://api.github.com/users/mrexodia/events{/privacy}","received_events_url":"https://api.github.com/users/mrexodia/received_events","type":"User","site_admin":false},"created_at":"2018-11-25T11:17:26Z","updated_at":"2018-11-25T11:17:26Z","author_association":"NONE","body":"I have already solved this problem, just didn’t have time/motivation to\nimplement it...\n\nThe solution is to use a sax style api for reading/writing the json files.\nThis works because the json objects are never actually used while x64dbg is\nrunning.\nOn Sun, 25 Nov 2018 at 07:50, Corey Farrell <notifications@github.com>\nwrote:\n\n> This does not surprise me, the JSON file you referenced has a large number\n> of JSON objects. Each JSON object includes a hash table for lookup of\n> members where rapidjson appears to not use a hashtable. This is a\n> trade-off, hash table provides for faster lookup (read) from JSON objects,\n> but uses more memory. The benefit to hash tables is greater for larger\n> objects, so rapidjson CPU performance might degrade if working with objects\n> that have hundreds of members but the benefit is likely small for objects\n> with only a couple members.\n>\n> I don't think this is something that can be solved.\n>\n> —\n> You are receiving this because you authored the thread.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/akheron/jansson/issues/332#issuecomment-441419574>,\n> or mute the thread\n> <https://github.com/notifications/unsubscribe-auth/ACWCmXii8uN7Gchl_wgm6yqqzPwETwekks5uyj3CgaJpZM4McYWq>\n> .\n>\n"}}
{"actor":{"display_login":"mrexodia"}}