{"action":"closed","issue":{"url":"https://api.github.com/repos/ged/ruby-pg/issues/147","repository_url":"https://api.github.com/repos/ged/ruby-pg","labels_url":"https://api.github.com/repos/ged/ruby-pg/issues/147/labels{/name}","comments_url":"https://api.github.com/repos/ged/ruby-pg/issues/147/comments","events_url":"https://api.github.com/repos/ged/ruby-pg/issues/147/events","html_url":"https://github.com/ged/ruby-pg/issues/147","id":504261081,"node_id":"MDU6SXNzdWU1MDQyNjEwODE=","number":147,"title":"PGconn#async_exec loops forever if socket dies.","user":{"login":"ged","id":13501,"node_id":"MDQ6VXNlcjEzNTAx","avatar_url":"https://avatars1.githubusercontent.com/u/13501?v=4","gravatar_id":"","url":"https://api.github.com/users/ged","html_url":"https://github.com/ged","followers_url":"https://api.github.com/users/ged/followers","following_url":"https://api.github.com/users/ged/following{/other_user}","gists_url":"https://api.github.com/users/ged/gists{/gist_id}","starred_url":"https://api.github.com/users/ged/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ged/subscriptions","organizations_url":"https://api.github.com/users/ged/orgs","repos_url":"https://api.github.com/users/ged/repos","events_url":"https://api.github.com/users/ged/events{/privacy}","received_events_url":"https://api.github.com/users/ged/received_events","type":"User","site_admin":false},"labels":[{"id":1603741623,"node_id":"MDU6TGFiZWwxNjAzNzQxNjIz","url":"https://api.github.com/repos/ged/ruby-pg/labels/0.14.0","name":"0.14.0","color":"ededed","default":false},{"id":1603738800,"node_id":"MDU6TGFiZWwxNjAzNzM4ODAw","url":"https://api.github.com/repos/ged/ruby-pg/labels/Library","name":"Library","color":"ededed","default":false},{"id":48244083,"node_id":"MDU6TGFiZWw0ODI0NDA4Mw==","url":"https://api.github.com/repos/ged/ruby-pg/labels/bug","name":"bug","color":"fc2929","default":true},{"id":1603728890,"node_id":"MDU6TGFiZWwxNjAzNzI4ODkw","url":"https://api.github.com/repos/ged/ruby-pg/labels/major","name":"major","color":"ededed","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2012-12-10T20:39:53Z","updated_at":"2019-10-08T20:45:34Z","closed_at":"2019-10-08T20:45:34Z","author_association":"OWNER","body":"**[Original report](https://bitbucket.org/ged/ruby-pg/issue/147) by Kurt Stephens (Bitbucket: [kstephens](https://bitbucket.org/kstephens), GitHub: [kstephens](https://github.com/kstephens)).**\n\n----------------------------------------\n\nWe are losing DB connectivity in PGconn#async_exec during the PQisBusy() loop in pgconn_block().  \n\nThis condition causes an infinite loop, consuming all CPU resources.\n\nBreak-down of the problem:\n\n* PGconn#async_exec uses pgconn_block() to relinquish control to other Ruby threads while it's waiting for results from the PG connection FD.\n\n* pgconn_block() loops while PQisBusy() is true, asking rb_thread_select() to allow other Ruby threads to work until there is readable data on the PG connection FD:\n\n\n```\n#!none\n\n    (gdb) do\n    #4  0xb6d0ab56 in pgconn_block (argc=0, argv=0x0, self=3023657760) at\n    pg.c:2633\n    2633            ret = rb_thread_select( sd+1, &sd_rset, NULL, NULL, ptimeout );\n    (gdb) l\n    2628        PQconsumeInput( conn );\n    2629   \n    2630        while ( PQisBusy(conn) ) {\n    2631            FD_ZERO( &sd_rset );\n    2632            FD_SET( sd, &sd_rset );\n    2633            ret = rb_thread_select( sd+1, &sd_rset, NULL, NULL,\nptimeout );\n    2634   \n    2635            /* Return false if there was a timeout argument and the\nselect() timed out */\n    2636            if ( ret == 0 && argc )\n    2637                return Qfalse;\n    \n    (gdb) p sd_rset\n    $2 = {__fds_bits = {16, 0 <repeats 31 times>}}\n\n```\n* \"16\" corresponds to the 5th bit, meaning fd 4, which is no longer open:\n\n\n```\n#!none\n\n    kurt@ca10:~$ sudo lsof -p 11892\n    ...\n    ruby    11892 user    0u   CHR       5,1            536 /dev/console\n    ruby    11892 user    1w  FIFO       0,6           5861 pipe\n    ruby    11892 user    2u   REG       8,1  158788 144691 __HIDDEN__\n    ruby    11892 user    3r   REG       8,1      11   3734 __HIDDEN__\n    ruby    11892 user    5u   REG       8,1  158788 144691 __HIDDEN__\n    ruby    11892 user    6u  IPv4 327473100            TCP HIDDEN:HIDDEN (CLOSE_WAIT)\n\n```\n* rb_thread_schedule() is aware of the dead file descriptor (4)\nbecause it calls fcntl() on it to check its validity before calling\nselect():\n\n\n```\n#!none\n\n    $ strace -p 11892\n    fcntl64(4, F_GETFD)                     = -1 EBADF (Bad file descriptor)\n    ...\n    \n    (gdb) up\n    #1  0xb7d9b17d in fcntl () from /lib/i686/nosegneg/libc.so.6\n    (gdb) up\n    #2  0x08065d59 in rb_thread_schedule () at eval.c:11480\n    11480                        if ((FD_ISSET(fd, &readfds) ||\n    (gdb) l\n    11475                if (e == EBADF) {\n    11476                    int badfd = -1;\n    11477                    int fd;\n    11478                    int dummy;\n    11479                    for (fd = 0; fd <= max; fd++) {\n    11480                        if ((FD_ISSET(fd, &readfds) ||\n    11481                             FD_ISSET(fd, &writefds) ||\n    11482                             FD_ISSET(fd, &exceptfds)) &&\n    11483    #ifndef _WIN32\n    11484                            fcntl(fd, F_GETFD, &dummy) == -1 &&\n\n```\n* fcntl() returns -1 and rb_thread_schedule() returns immediately to\npgconn_block().\n\n* PQisBusy() continues to return true; it's unaware that its file\ndescriptor has been closed, thus it hammers the ruby thread scheduler.\n\n* PGconn#async_exec never returns; it spins forever while polling fcntl().\n\n=====================\n\nThis is occurring on CRuby 1.8.7.  I have not yet verified on CRuby 1.9+.\n\nSee the attached patch against gem pg 0.14.1.\n\nA possible fix is to use fcntl(fd, F_GETFD, &dummy) in pgconn_block()\nand raise an error if it == -1, but this is not portable to non-*nix.\n\nI did not find a portable libpq function to do it.\n\nThanks,\nKurt Stephens\n"}}
{"actor":{"display_login":"ged"}}