{"action":"created","issue":{"url":"https://api.github.com/repos/tarantool/tarantool/issues/1093","repository_url":"https://api.github.com/repos/tarantool/tarantool","labels_url":"https://api.github.com/repos/tarantool/tarantool/issues/1093/labels{/name}","comments_url":"https://api.github.com/repos/tarantool/tarantool/issues/1093/comments","events_url":"https://api.github.com/repos/tarantool/tarantool/issues/1093/events","html_url":"https://github.com/tarantool/tarantool/issues/1093","id":111349750,"number":1093,"title":"Memory information","user":{"login":"rtsisyk","id":1799054,"avatar_url":"https://avatars.githubusercontent.com/u/1799054?v=3","gravatar_id":"","url":"https://api.github.com/users/rtsisyk","html_url":"https://github.com/rtsisyk","followers_url":"https://api.github.com/users/rtsisyk/followers","following_url":"https://api.github.com/users/rtsisyk/following{/other_user}","gists_url":"https://api.github.com/users/rtsisyk/gists{/gist_id}","starred_url":"https://api.github.com/users/rtsisyk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rtsisyk/subscriptions","organizations_url":"https://api.github.com/users/rtsisyk/orgs","repos_url":"https://api.github.com/users/rtsisyk/repos","events_url":"https://api.github.com/users/rtsisyk/events{/privacy}","received_events_url":"https://api.github.com/users/rtsisyk/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/tarantool/tarantool/labels/feature","name":"feature","color":"bfe5bf"}],"state":"closed","locked":false,"assignee":{"login":"alyapunov","id":5285048,"avatar_url":"https://avatars.githubusercontent.com/u/5285048?v=3","gravatar_id":"","url":"https://api.github.com/users/alyapunov","html_url":"https://github.com/alyapunov","followers_url":"https://api.github.com/users/alyapunov/followers","following_url":"https://api.github.com/users/alyapunov/following{/other_user}","gists_url":"https://api.github.com/users/alyapunov/gists{/gist_id}","starred_url":"https://api.github.com/users/alyapunov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alyapunov/subscriptions","organizations_url":"https://api.github.com/users/alyapunov/orgs","repos_url":"https://api.github.com/users/alyapunov/repos","events_url":"https://api.github.com/users/alyapunov/events{/privacy}","received_events_url":"https://api.github.com/users/alyapunov/received_events","type":"User","site_admin":false},"assignees":[{"login":"alyapunov","id":5285048,"avatar_url":"https://avatars.githubusercontent.com/u/5285048?v=3","gravatar_id":"","url":"https://api.github.com/users/alyapunov","html_url":"https://github.com/alyapunov","followers_url":"https://api.github.com/users/alyapunov/followers","following_url":"https://api.github.com/users/alyapunov/following{/other_user}","gists_url":"https://api.github.com/users/alyapunov/gists{/gist_id}","starred_url":"https://api.github.com/users/alyapunov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alyapunov/subscriptions","organizations_url":"https://api.github.com/users/alyapunov/orgs","repos_url":"https://api.github.com/users/alyapunov/repos","events_url":"https://api.github.com/users/alyapunov/events{/privacy}","received_events_url":"https://api.github.com/users/alyapunov/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/tarantool/tarantool/milestones/7","html_url":"https://github.com/tarantool/tarantool/milestones/wishlist","labels_url":"https://api.github.com/repos/tarantool/tarantool/milestones/7/labels","id":667092,"number":7,"title":"wishlist","description":"Out of scope for 1.8","creator":{"login":"kostja","id":482910,"avatar_url":"https://avatars.githubusercontent.com/u/482910?v=3","gravatar_id":"","url":"https://api.github.com/users/kostja","html_url":"https://github.com/kostja","followers_url":"https://api.github.com/users/kostja/followers","following_url":"https://api.github.com/users/kostja/following{/other_user}","gists_url":"https://api.github.com/users/kostja/gists{/gist_id}","starred_url":"https://api.github.com/users/kostja/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kostja/subscriptions","organizations_url":"https://api.github.com/users/kostja/orgs","repos_url":"https://api.github.com/users/kostja/repos","events_url":"https://api.github.com/users/kostja/events{/privacy}","received_events_url":"https://api.github.com/users/kostja/received_events","type":"User","site_admin":false},"open_issues":183,"closed_issues":2,"state":"open","created_at":"2014-05-22T16:16:45Z","updated_at":"2016-06-29T14:24:20Z","due_on":null,"closed_at":null},"comments":1,"created_at":"2015-10-14T08:38:08Z","updated_at":"2016-06-29T14:24:20Z","closed_at":"2016-06-29T14:24:20Z","body":"\"Why Tarantool uses so much memory?\" is the one of most popular question by our customers.\r\nReally, why? How to understand which subsystem is heavily utilizes memory on the production installation?\r\n\r\nCurrently Tarantool 1.6 totally lacks **useful** memory information utils.\r\nWe even dropped old \"show palloc\", which was last useful tool for investigating memory problems.\r\n\r\nLet's take a look on current tools.\r\n\r\n### box.runtime.info()\r\n\r\n```\r\n- maxalloc: 4398046510080\r\n  used: 12582912\r\n...\r\n```\r\n\r\n* `maxalloc` - a constant distance to a some remote galaxy.\r\n* `used` - indicates how many memory used by some internal subsystems but nobody knowns by which one.\r\n\r\nI don't think that our users want to `grep` our sources to understand that `runtime` means a special **runtime** `arena` for non-data allocations. Moreover, 99.99999% of our current and potential customers don't know WTF is [arena](https://www.google.com/search?q=arena&num=20&safe=off&source=lnms&tbm=isch&sa=X&biw=1278&bih=932). I think that our users should not care about details of Tarantool internal structure..\r\n\r\n###  box.slab.info()        \r\n\r\n```                                                                                                                          \r\n---\r\n- quota_used: 8388608\r\n  arena_used_ratio: 0.8%\r\n  items_used_ratio: 1.9%\r\n  arena_size: 2325384\r\n  quota_size: 1073741824\r\n  arena_used: 1102088\r\n```\r\n\r\nOhh, arena again. Anyway, let's take a look on `arena_size`, `arena_used` and `arena_used_ratio`:\r\n\r\n* `arena_size` is 2325384\r\n* `arena_used` is 1102088\r\n*  `arena_used_ratio` is 0.8%\r\n\r\nArena is [utilized](http://hdwallpaperhub.net/wallpapers/l/1280x1024/26/fight_men_people_arena_bull_football_afl_1280x1024_25743.jpg) by `1102088 / 2325384 = ~ 47%`. But why `arena_used_ratio` is **0.8%** and not **47%**?\r\nLet's move on to `items`. WTF is items? Tuples? Spaces? Whatever?\r\nNext. What is `items_used_ratio`?\r\n\r\n`quota_size` and `quota_used` is mostly OK. I'm just curious, how to change `quota_size`?\r\nI found nothing about quota on http://tarantool.org/doc/singlehtml.html\r\n\r\n`slabs` is mostly OK, except my note about `items` above.\r\n\r\n### box.fiber.info()\r\n\r\n```\r\n---\r\n- 101:\r\n    memory:\r\n      total: 65760\r\n      used: 0\r\n    fid: 101\r\n    name: interactive\r\n    csw: 6\r\n```\r\n\r\nMostly OK. How to understand who used this memory?\r\n\r\n### space.space.index[x]:bsize()\r\n\r\n```\r\ntarantool> box.space._space.index[0]:bsize()\r\n---\r\n- 49152\r\n...\r\n```\r\n\r\nOK. I think that users are also interested in total memory usage by space (e.g. space:bsize()).\r\n\r\n### What should be done\r\n\r\n* Add a new box.memory.info():\r\n\r\n```\r\n- tx: -- per each cord\r\n    - indexes: xxx\r\n    - tuples: xxx\r\n    - fibers:  xxx-- including fiber stack and fiber->gc\r\n    - sessions: xx\r\n    - lua: xx\r\n    - malloc: xxx\r\n    - total:\r\n       - allocated: xxx\r\n       - used: xxx\r\n       - fragmentation: xx%\r\n\r\n# assert(sum(categories) == used)\r\n\r\n- iproto:\r\n    - input_buffers: xxx\r\n    - output_buffers: xxx\r\n    - connections: xxx\r\n    - cbus: xxxx\r\n    - fibers:  xxx-- including fiber stack and fiber->gc\r\n    - malloc: xxx \r\n    - total:\r\n       - allocated: xxx\r\n       - used: xxx\r\n       - fragmentation: xx%\r\n    \r\n\r\n- wal:\r\n    - fibers:  xxx-- including fiber stack and fiber->gc\r\n    - total:\r\n       - allocated: xxx\r\n       - used: xxx\r\n       - fragmentation: xx%\r\n\r\n- sophia_merger:\r\n    - malloc: xxx\r\n    - total:\r\n       - allocated: xxx\r\n       - used: xxx\r\n       - fragmentation: xx%\r\n\r\n- total:\r\n     - allocated: xxx\r\n     - used: xxx\r\n     - fragmentation: xxx\r\n\r\n# assert(used == RSS in top(1))\r\n# assert(allocated == VIRT in top(1))\r\n```\r\n\r\nTake a look on malloc_stats(3) for inspiration:\r\n\r\n```\r\ntarantool> ffi.C.malloc_stats()\r\nArena 0:\r\nsystem bytes     =     860160\r\nin use bytes     =     714704\r\nArena 1:\r\nsystem bytes     =     135168\r\nin use bytes     =     134736\r\nTotal (incl. mmap):\r\nsystem bytes     =     995328\r\nin use bytes     =     849440\r\nmax mmap regions =          1\r\nmax mmap bytes   =     163840\r\n```\r\n\r\n* Fix incorrect (mostly random) numbers in box.slab.info()\r\n* Remove `box.runtime.info()` as totally confusing and uninformative tool.\r\n* Add space:bsize()\r\n\r\nI understand that this ticket is very hard.\r\nPlease implement basic structure of box.memory.info() and then add sections step by step.\r\nSection names provided as for example.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/tarantool/tarantool/issues/comments/229371933","html_url":"https://github.com/tarantool/tarantool/issues/1093#issuecomment-229371933","issue_url":"https://api.github.com/repos/tarantool/tarantool/issues/1093","id":229371933,"user":{"login":"rtsisyk","id":1799054,"avatar_url":"https://avatars.githubusercontent.com/u/1799054?v=3","gravatar_id":"","url":"https://api.github.com/users/rtsisyk","html_url":"https://github.com/rtsisyk","followers_url":"https://api.github.com/users/rtsisyk/followers","following_url":"https://api.github.com/users/rtsisyk/following{/other_user}","gists_url":"https://api.github.com/users/rtsisyk/gists{/gist_id}","starred_url":"https://api.github.com/users/rtsisyk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rtsisyk/subscriptions","organizations_url":"https://api.github.com/users/rtsisyk/orgs","repos_url":"https://api.github.com/users/rtsisyk/repos","events_url":"https://api.github.com/users/rtsisyk/events{/privacy}","received_events_url":"https://api.github.com/users/rtsisyk/received_events","type":"User","site_admin":false},"created_at":"2016-06-29T14:24:20Z","updated_at":"2016-06-29T14:24:20Z","body":"Duplicate of #934"}}
{"actor":{"display_login":"rtsisyk"}}