{"action":"created","issue":{"url":"https://api.github.com/repos/vozlt/nginx-module-vts/issues/35","repository_url":"https://api.github.com/repos/vozlt/nginx-module-vts","labels_url":"https://api.github.com/repos/vozlt/nginx-module-vts/issues/35/labels{/name}","comments_url":"https://api.github.com/repos/vozlt/nginx-module-vts/issues/35/comments","events_url":"https://api.github.com/repos/vozlt/nginx-module-vts/issues/35/events","html_url":"https://github.com/vozlt/nginx-module-vts/issues/35","id":140125835,"number":35,"title":"The value of total connections is wrong","user":{"login":"tkgeng","id":17307367,"avatar_url":"https://avatars.githubusercontent.com/u/17307367?v=3","gravatar_id":"","url":"https://api.github.com/users/tkgeng","html_url":"https://github.com/tkgeng","followers_url":"https://api.github.com/users/tkgeng/followers","following_url":"https://api.github.com/users/tkgeng/following{/other_user}","gists_url":"https://api.github.com/users/tkgeng/gists{/gist_id}","starred_url":"https://api.github.com/users/tkgeng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tkgeng/subscriptions","organizations_url":"https://api.github.com/users/tkgeng/orgs","repos_url":"https://api.github.com/users/tkgeng/repos","events_url":"https://api.github.com/users/tkgeng/events{/privacy}","received_events_url":"https://api.github.com/users/tkgeng/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-03-11T08:53:25Z","updated_at":"2016-03-16T05:17:24Z","closed_at":null,"body":"Hi, I'm back :stuck_out_tongue_closed_eyes: \r\n\r\nThe new problem is that the value of total connections in `Server main` of the status webpage is too big,  it's wrong,it should  be less than 15k,  there's not so many connections in our system. after I restart Nginx, it's back to normal(about 10000 connections), but after about a week, it's into wrong again.\r\n\r\nBTW, when it's in the abnormal state, the data of original module `stub_status_module` of Nginx is also abnormal, of course, they are equal.\r\n\r\n## **_SS:_**\r\n![561818f4-3cce-49e4-8513-2d40d5e475cf](https://cloud.githubusercontent.com/assets/17307367/13697372/0814cbc8-e7a8-11e5-9579-88b1ae0d53af.png)\r\n\r\n## **_nginx-module-vts:_**\r\n![d81a4716-353d-4543-8af1-005cbf410f9c](https://cloud.githubusercontent.com/assets/17307367/13697375/082402d2-e7a8-11e5-9e5f-5fc38620e067.png)\r\n\r\n\r\n## **_stub_status_module:_**\r\n![d55a9022-68e2-4718-9a0a-73427f584f35](https://cloud.githubusercontent.com/assets/17307367/13697373/081d2674-e7a8-11e5-8005-7b5471629113.png)\r\n"},"comment":{"url":"https://api.github.com/repos/vozlt/nginx-module-vts/issues/comments/197157603","html_url":"https://github.com/vozlt/nginx-module-vts/issues/35#issuecomment-197157603","issue_url":"https://api.github.com/repos/vozlt/nginx-module-vts/issues/35","id":197157603,"user":{"login":"vozlt","id":3648408,"avatar_url":"https://avatars.githubusercontent.com/u/3648408?v=3","gravatar_id":"","url":"https://api.github.com/users/vozlt","html_url":"https://github.com/vozlt","followers_url":"https://api.github.com/users/vozlt/followers","following_url":"https://api.github.com/users/vozlt/following{/other_user}","gists_url":"https://api.github.com/users/vozlt/gists{/gist_id}","starred_url":"https://api.github.com/users/vozlt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vozlt/subscriptions","organizations_url":"https://api.github.com/users/vozlt/orgs","repos_url":"https://api.github.com/users/vozlt/repos","events_url":"https://api.github.com/users/vozlt/events{/privacy}","received_events_url":"https://api.github.com/users/vozlt/received_events","type":"User","site_admin":false},"created_at":"2016-03-16T05:17:23Z","updated_at":"2016-03-16T05:17:23Z","body":"The abnormal values probably seem to happen because of the failure to decrease the status variable values. In other words, It seems to due to the terminating of worker process by signal 11 before running decrease process. By the way, more important thing is why worker process exited on signal 11.\r\n\r\n## Status variables\r\nThe status variables used in [ngx_http_stub_status_module](http://nginx.org/en/docs/http/ngx_http_stub_status_module.html) and [nginx-module-vts](https://github.com/vozlt/nginx-module-vts) are a shared memory variables([atomic](https://gcc.gnu.org/onlinedocs/gcc-4.4.5/gcc/Atomic-Builtins.html)). Some variables are only increased without decrease as follows:\r\n\r\n| stats\t\t| increase\t| decrease |\r\n| ----------- | ----------- | ----------- |\r\n| accepted\t| yes\t\t| no |\r\n| handled\t| yes\t\t| no |\r\n| requests\t| yes\t\t| no |\r\n| active\t\t| yes\t\t| yes |\r\n| reading\t| yes\t\t| yes |\r\n| writing\t| yes\t\t| yes |\r\n| waiting\t| yes\t\t| yes |\r\n\r\nIncrease function\r\n```C\r\n(void) ngx_atomic_fetch_add($ngx_stat_variable, 1);\r\n```\r\n\r\nDecrease function\r\n```C\r\n(void) ngx_atomic_fetch_add($ngx_stat_variable, -1);\r\n```\r\n\r\n## Some questions\r\n\r\n1. Does the error(`exited on signal 11`) frequently appears?\r\n\r\n2. How about the system status when logged in error_log?(e.g memory usage, loadavg)\r\n\r\n3. How about the Req/s when logged in error_log if peak than usual?\r\n\r\n4. Is there anything unusual log in /var/log/messages when logged in error_log(e.g. memory...)?\r\n\r\n5. Does it also happen in mainline version(1.9.x)?\r\n\r\n6. Please try again backtrace(core dump)\r\n\r\n## backtrace\r\n### Check max core file size\r\nSee the nginx [debugging](https://www.nginx.com/resources/wiki/start/topics/tutorials/debugging/).\r\n\r\n1. Please check if the soft limit of max core file size is 0 in worker process. \r\n    It can see in /proc/$(worker_process_id)/limits.\r\n\r\n    `$ cat /proc/$(ps aux | awk '/worker/ {print $2}' | sed -n 1p)/limits`\r\n    ```Bash\r\n    Limit                     Soft Limit           Hard Limit           Units\r\n    .\r\n    .\r\n    Max core file size        0                    unlimited            bytes\r\n    .\r\n    .\r\n    ```\r\n\r\n2. If so, fix(> 0 or unlimited) and restart and check it.\r\n    ```Bash\r\n    $ ulimit -S -c unlimited\r\n    $ /usr/sbin/nginx -c /etc/nginx/nginx.conf\r\n    $ cat /proc/$(ps aux | awk '/worker/ {print $2}' | sed -n 1p)/limits\r\n    ```\r\n    ```\r\n    Limit                     Soft Limit           Hard Limit           Units\r\n    .\r\n    .\r\n    Max core file size        unlimited            unlimited            bytes\r\n    .\r\n    .\r\n    ```\r\n    \r\n3. If you succeed, it looks like as follows:\r\n    `2016/... [alert] ... : worker process ... exited on signal 11 (core dumped)`\r\n\r\n\r\n### Script to check max core file size\r\n\r\n`$ vi nginx-core-limit-check.sh`\r\n\r\n```Bash\r\n#! /bin/env bash\r\n\r\necho \"# Max core file size of nginx worker process for core dump(must be soft > 0)\"\r\nprintf \"%-16s%-16s%-16s\\n\" \"PID\" \"Soft Limit\" \"Hard Limit\"\r\nfor pid in $(echo /proc/[0-9]*)\r\ndo\r\n    [ -e \"$pid/cmdline\" ] && cmdline=$(<$pid/cmdline)\r\n    if [[ $cmdline == *\"nginx: worker\"* ]]\r\n    then\r\n        limits=$(<$pid/limits)\r\n        core_soft_limit=$(awk '/core/ {print $5\" \"$6}' $pid/limits)\r\n        printf \"%-16s%-16s%-16s\\n\" $pid $core_soft_limit\r\n    fi  \r\ndone\r\n```\r\n\r\n`$ bash nginx-core-limit-check.sh`\r\n```Bash \r\n# Max core file size of nginx worker process for core dump(must be soft > 0)\r\nPID             Soft Limit      Hard Limit       \r\n/proc/24262     unlimited       unlimited       \r\n/proc/24263     unlimited       unlimited       \r\n```\r\n\r\n## Source trace\r\n\r\nTraced the nginx core source(1.9.12)\r\n\r\n#### The signal 11 is as follows:\r\n```\r\n$ man 7 signal | grep 11\r\nSIGSEGV      11       Core    Invalid memory reference\r\n```\r\n\r\n#### Trace the message `exited on signal 11`\r\n```\r\n2016/03/14 17:25:17 [alert] 2465#0: worker process 31188 exited on signal 11\r\n```\r\n\r\n`$ vi src/os/unix/ngx_process.c`\r\n\r\n```C\r\n452 static void\r\n453 ngx_process_get_status(void)\r\n454 {\r\n.\r\n.\r\n516 #ifdef WCOREDUMP\r\n517             ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, 0,\r\n518                           \"%s %P exited on signal %d%s\",\r\n519                           process, pid, WTERMSIG(status),\r\n520                           WCOREDUMP(status) ? \" (core dumped)\" : \"\");\r\n521 #else\r\n522             ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, 0,\r\n523                           \"%s %P exited on signal %d\",\r\n524                           process, pid, WTERMSIG(status));\r\n525 #endif\r\n```\r\n\r\n#### Trace the message `left in connection`\r\n```\r\n2016/03/11 16:14:27 [alert] 2395#0: *664085514 open socket #2213 left in connection 2186\r\n2016/03/11 16:14:27 [alert] 2395#0: aborting\r\n```\r\n\r\n`$ vi src/os/unix/ngx_process_cycle.c`\r\n\r\n```C\r\n 726 static void\r\n 727 ngx_worker_process_cycle(ngx_cycle_t *cycle, void *data)\r\n 728 {\r\n .\r\n .\r\n 737 \r\n 738     for ( ;; ) {\r\n 739 \r\n 740         if (ngx_exiting) {\r\n 741             ngx_event_cancel_timers();\r\n 742 \r\n 743             if (ngx_event_timer_rbtree.root == ngx_event_timer_rbtree.sentinel)\r\n 744             {\r\n 745                 ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, \"exiting\");\r\n 746 \r\n 747                 ngx_worker_process_exit(cycle);\r\n 748             }\r\n 749         }\r\n 750 \r\n 751         ngx_log_debug0(NGX_LOG_DEBUG_EVENT, cycle->log, 0, \"worker cycle\");\r\n 752 \r\n 753         ngx_process_events_and_timers(cycle);\r\n 754 \r\n 755         if (ngx_terminate) {\r\n 756             ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, \"exiting\");\r\n 757 \r\n 758             ngx_worker_process_exit(cycle);\r\n 759         }\r\n .\r\n .\r\n 946 static void\r\n 947 ngx_worker_process_exit(ngx_cycle_t *cycle)\r\n 948 {\r\n .\r\n .\r\n 958     if (ngx_exiting) {\r\n 959         c = cycle->connections;\r\n 960         for (i = 0; i < cycle->connection_n; i++) {\r\n 961             if (c[i].fd != -1\r\n 962                 && c[i].read\r\n 963                 && !c[i].read->accept\r\n 964                 && !c[i].read->channel\r\n 965                 && !c[i].read->resolver)\r\n 966             {\r\n 967                 ngx_log_error(NGX_LOG_ALERT, cycle->log, 0,\r\n 968                               \"*%uA open socket #%d left in connection %ui\",\r\n 969                               c[i].number, c[i].fd, i);\r\n 970                 ngx_debug_quit = 1;\r\n 971             }\r\n 972         }\r\n 973 \r\n 974         if (ngx_debug_quit) {\r\n 975             ngx_log_error(NGX_LOG_ALERT, cycle->log, 0, \"aborting\");\r\n 976             ngx_debug_point();\r\n 977         }\r\n 978     }\r\n```\r\n"}}