{"action":"created","issue":{"url":"https://api.github.com/repos/ntop/ntopng/issues/486","repository_url":"https://api.github.com/repos/ntop/ntopng","labels_url":"https://api.github.com/repos/ntop/ntopng/issues/486/labels{/name}","comments_url":"https://api.github.com/repos/ntop/ntopng/issues/486/comments","events_url":"https://api.github.com/repos/ntop/ntopng/issues/486/events","html_url":"https://github.com/ntop/ntopng/issues/486","id":147703760,"number":486,"title":"Autonomous systems page performance","user":{"login":"Queeq","id":3294985,"avatar_url":"https://avatars.githubusercontent.com/u/3294985?v=3","gravatar_id":"","url":"https://api.github.com/users/Queeq","html_url":"https://github.com/Queeq","followers_url":"https://api.github.com/users/Queeq/followers","following_url":"https://api.github.com/users/Queeq/following{/other_user}","gists_url":"https://api.github.com/users/Queeq/gists{/gist_id}","starred_url":"https://api.github.com/users/Queeq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Queeq/subscriptions","organizations_url":"https://api.github.com/users/Queeq/orgs","repos_url":"https://api.github.com/users/Queeq/repos","events_url":"https://api.github.com/users/Queeq/events{/privacy}","received_events_url":"https://api.github.com/users/Queeq/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ntop/ntopng/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":{"login":"simonemainardi","id":3397663,"avatar_url":"https://avatars.githubusercontent.com/u/3397663?v=3","gravatar_id":"","url":"https://api.github.com/users/simonemainardi","html_url":"https://github.com/simonemainardi","followers_url":"https://api.github.com/users/simonemainardi/followers","following_url":"https://api.github.com/users/simonemainardi/following{/other_user}","gists_url":"https://api.github.com/users/simonemainardi/gists{/gist_id}","starred_url":"https://api.github.com/users/simonemainardi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/simonemainardi/subscriptions","organizations_url":"https://api.github.com/users/simonemainardi/orgs","repos_url":"https://api.github.com/users/simonemainardi/repos","events_url":"https://api.github.com/users/simonemainardi/events{/privacy}","received_events_url":"https://api.github.com/users/simonemainardi/received_events","type":"User","site_admin":false},"milestone":null,"comments":23,"created_at":"2016-04-12T10:25:30Z","updated_at":"2016-06-01T16:09:41Z","closed_at":null,"body":"Hi,\r\nI'm running ntopng (Community v.2.2.160411) on a VMWare virtual host (8 CPUs, 8G RAM), Ubuntu 14.04, custom 4.4.6 kernel. Our external traffic during the day is around 25 Gbit/s. Flows exported from border routers are sampled 4096:1.\r\nHosts, Autonomous Systems and Country pages from \"Hosts\" menu take a lot of time to load. I understand that they may need to aggregate a lot of data, but is it okay for it to take tens of seconds to load? If more than one user accesses the system simultaneously it becomes totally unusable for all of them.\r\n\r\nThe following are scenarios I've tried so far to try to resolve the issue (all within the same virtual machine, local connections over TCP):\r\n\r\n1. One nprobe receiving all flows, one ntopng connecting over ZMQ to the nprobe. Observed 100% load on one CPU and <10% load on others. The overloaded CPU was loaded by around 40-60% by Redis server and the rest was system. ntopng was consuming around 40-80% CPU distributed among other CPUs.\r\n2. Six nprobes (one for each FPC carrying uplink traffic on the border router), one ntopng with ZMQ. Pretty much similar picture.\r\n3. Six nprobes + 3 intermediate ntopngs (ZMQ to 2 of the nprobes from each, exporting flows to 1 main ntopng. Separate local Redis server for each of the ntopngs (made this way due to Redis being single-threaded). This way I saw some load distribution among different Redis servers and ntopngs. However main ntopng and the corresponding Redis server kept on eating the majority of CPU (not all though). Subjectively, this lead to some improvement in load time, but it's still big.\r\n4. Same as nr. 3, but six intermediate ntopngs with six Redis servers. The load distributed a bit more. Main ntopng keeps eating 100-150% CPU, main Redis server eats around 30-35% + 30-35% of system load on the same CPU when the page is opened.\r\n5. Setting --dns-mode 3 (i.e. turning it off) makes things better, but it's still unusable by multiple users. Switching between pages on AS page takes more than a minute to complete too.\r\n\r\nI/O is not an issue here, disks are not loaded at all.\r\n\r\nWhat bothers me is that Redis uses too small amount of memory while I'd expect it to cache everything. In scenario nr. 5 main instance consumes 6MB (6 megabytes!) and other instances - 12-25 MB. Main ntopng process, however, eats around 900 MB (others - 100-200MB).\r\nIn scenario nr. 1 ntopng process consumed small amount of memory, but Redis-server was still keeping small (less than 100 MB).\r\n\r\nSomething is wrong here.\r\n\r\nNow scenario nr. 5 is active. Here's `redis-cli` info for the main process:\r\n```\r\n# Server\r\nredis_version:2.8.4\r\nredis_git_sha1:00000000\r\nredis_git_dirty:0\r\nredis_build_id:a44a05d76f06a5d9\r\nredis_mode:standalone\r\nos:Linux 4.4.6 x86_64\r\narch_bits:64\r\nmultiplexing_api:epoll\r\ngcc_version:4.8.2\r\nprocess_id:17544\r\nrun_id:dd4de3e59ff65e0d4274d04245d857cb88d9bd70\r\ntcp_port:6379\r\nuptime_in_seconds:8196\r\nuptime_in_days:0\r\nhz:10\r\nlru_clock:1342164\r\nconfig_file:/etc/redis/redis.conf\r\n  \r\n# Clients\r\nconnected_clients:2\r\nclient_longest_output_list:0\r\nclient_biggest_input_buf:0\r\nblocked_clients:0\r\n\r\n# Memory\r\nused_memory:599728\r\nused_memory_human:585.67K\r\nused_memory_rss:6590464\r\nused_memory_peak:1205296\r\nused_memory_peak_human:1.15M\r\nused_memory_lua:33792\r\nmem_fragmentation_ratio:10.99\r\nmem_allocator:jemalloc-3.4.1\r\n\r\n# Persistence\r\nloading:0\r\nrdb_changes_since_last_save:0\r\nrdb_bgsave_in_progress:0\r\nrdb_last_save_time:1460455907\r\nrdb_last_bgsave_status:ok\r\nrdb_last_bgsave_time_sec:0\r\nrdb_current_bgsave_time_sec:-1\r\naof_enabled:0\r\naof_rewrite_in_progress:0\r\naof_rewrite_scheduled:0\r\naof_last_rewrite_time_sec:-1\r\naof_current_rewrite_time_sec:-1\r\naof_last_bgrewrite_status:ok\r\n\r\n# Stats\r\ntotal_connections_received:5\r\ntotal_commands_processed:68594238\r\ninstantaneous_ops_per_sec:9501\r\nrejected_connections:0\r\nsync_full:0\r\nsync_partial_ok:0\r\nsync_partial_err:0\r\nexpired_keys:3542\r\nevicted_keys:0\r\nkeyspace_hits:234749\r\nkeyspace_misses:68349836\r\npubsub_channels:0\r\npubsub_patterns:0\r\nlatest_fork_usec:928\r\n  \r\n# Replication\r\nrole:master\r\nconnected_slaves:0\r\nmaster_repl_offset:0\r\nrepl_backlog_active:0\r\nrepl_backlog_size:1048576\r\nrepl_backlog_first_byte_offset:0\r\nrepl_backlog_histlen:0\r\n  \r\n# CPU\r\nused_cpu_sys:1955.39\r\nused_cpu_user:441.03\r\nused_cpu_sys_children:0.08\r\nused_cpu_user_children:0.02\r\n  \r\n# Keyspacehad appeared long before that.\r\ndb0:keys=86,expires=4,avg_ttl=33774737\r\n```\r\n**Edit 1**\r\nAdditionally, in scenario nr. 5 some of the intermediate ntopng processes periodically crash without any messages in their log files. There is stderr message (even though they're run in daemon mode):\r\n```\r\n*** Error in `ntopng': double free or corruption (fasttop): 0x0000000002a03b10 ***\r\nBad address (stream_engine.cpp:785)\r\n```\r\n**Edit 2**\r\nI figured that the mentioned message about \"double free or corruption\" has nothing to do with crash. After I've run ntopng instance in foreground, it crashed with the following message:\r\n```\r\nAssertion failed: check () (msg.cpp:220)\r\nAborted\r\n````\r\nThe \"double free or corruption\" message after it had been running for several minutes, but it had been pretty long time before it crashed (an hour or so).\r\n\r\n**Edit 3**\r\n5 out of 6 intermediate ntopng processes crashed over the night. I have noticed two more messages in stderr:\r\n```\r\nSegmentation fault\r\n```\r\n```\r\nAssertion failed: refs_ >= 0 (msg.cpp:272)\r\n```"},"comment":{"url":"https://api.github.com/repos/ntop/ntopng/issues/comments/223043097","html_url":"https://github.com/ntop/ntopng/issues/486#issuecomment-223043097","issue_url":"https://api.github.com/repos/ntop/ntopng/issues/486","id":223043097,"user":{"login":"Queeq","id":3294985,"avatar_url":"https://avatars.githubusercontent.com/u/3294985?v=3","gravatar_id":"","url":"https://api.github.com/users/Queeq","html_url":"https://github.com/Queeq","followers_url":"https://api.github.com/users/Queeq/followers","following_url":"https://api.github.com/users/Queeq/following{/other_user}","gists_url":"https://api.github.com/users/Queeq/gists{/gist_id}","starred_url":"https://api.github.com/users/Queeq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Queeq/subscriptions","organizations_url":"https://api.github.com/users/Queeq/orgs","repos_url":"https://api.github.com/users/Queeq/repos","events_url":"https://api.github.com/users/Queeq/events{/privacy}","received_events_url":"https://api.github.com/users/Queeq/received_events","type":"User","site_admin":false},"created_at":"2016-06-01T16:09:41Z","updated_at":"2016-06-01T16:09:41Z","body":"Last time it was run as `valgrind --tool=memcheck --leak-check=yes --time-stamp=yes ntopng /etc/ntopng/ntopng.conf`\r\n\r\nThe ntopng.conf is the following:\r\n```\r\n-G=/var/tmp/ntopng.pid\r\n--community\r\n--http-port=:3000\r\n--http-prefix=\"/ntopng\"\r\n--dns-mode=3\r\n--max-num-flows=200000\r\n--max-num-hosts=250000\r\n--sticky-hosts=none\r\n--dump-flows=mysql;localhost;ntopng;flows;ntopng;<cut>\r\n--dump-hosts=all\r\n-i=tcp://127.0.0.1:5556\r\n-i=tcp://127.0.0.1:5557\r\n-i=tcp://127.0.0.1:5558\r\n-i=tcp://127.0.0.1:5559\r\n-i=tcp://127.0.0.1:5562\r\n-i=tcp://127.0.0.1:5564\r\n-i=view:tcp://127.0.0.1:5557,tcp://127.0.0.1:5558,tcp://127.0.0.1:5559,tcp://127.0.0.1:5562\r\n-i=view:tcp://127.0.0.1:5556,tcp://127.0.0.1:5557,tcp://127.0.0.1:5558,tcp://127.0.0.1:5559,tcp://127.0.0.1:5562,tcp://127.0.0.1:5564\r\n```\r\n\r\nNtopng freeezes only after UI had been opened for some time. It could run for days without problems when UI is closed.\r\n"}}
{"actor":{"display_login":"Queeq"}}