{"action":"created","issue":{"url":"https://api.github.com/repos/facebook/hhvm/issues/6715","labels_url":"https://api.github.com/repos/facebook/hhvm/issues/6715/labels{/name}","comments_url":"https://api.github.com/repos/facebook/hhvm/issues/6715/comments","events_url":"https://api.github.com/repos/facebook/hhvm/issues/6715/events","html_url":"https://github.com/facebook/hhvm/issues/6715","id":124880305,"number":6715,"title":"madvise MADV_DONTNEED loop in hhvm?","user":{"login":"mqudsi","id":606923,"avatar_url":"https://avatars.githubusercontent.com/u/606923?v=3","gravatar_id":"","url":"https://api.github.com/users/mqudsi","html_url":"https://github.com/mqudsi","followers_url":"https://api.github.com/users/mqudsi/followers","following_url":"https://api.github.com/users/mqudsi/following{/other_user}","gists_url":"https://api.github.com/users/mqudsi/gists{/gist_id}","starred_url":"https://api.github.com/users/mqudsi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mqudsi/subscriptions","organizations_url":"https://api.github.com/users/mqudsi/orgs","repos_url":"https://api.github.com/users/mqudsi/repos","events_url":"https://api.github.com/users/mqudsi/events{/privacy}","received_events_url":"https://api.github.com/users/mqudsi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":57,"created_at":"2016-01-05T02:08:09Z","updated_at":"2016-01-13T17:51:31Z","closed_at":null,"body":"This is going to be a mostly-useless bug report at the start, because hhvm is crashing gdb and I can't get a stacktrace for you.\r\n\r\nRunning hhvm-dbg on Debian (3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u1 (2015-12-14) x86_64 GNU/Linux):\r\n\r\n    HipHop VM 3.11.0 (dbg)\r\n    Compiler: tags/HHVM-3.11.0-0-g3dd564a8cde23e3205a29720d3435c771274085e\r\n    Repo schema: 52047bdda550f21c2ec2fcc295e0e6d02407be51\r\n\r\nRandomly while browsing (but quite often) a Xenforo installation, an hhvm process will go haywire, using 100% CPU. An strace call reveals a storm of `madvise` calls with virtually no end in sight:\r\n\r\n    madvise(0x7f005a3f8000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b03d000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b068000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b090000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b0f6000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b140000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b257000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b02e000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b058000, 36864, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b0b5000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b0be000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b109000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b154000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b24f000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b288000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b293000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b2de000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b35d000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b385000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b3de000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f007fdff000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f007fe0e000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f007fc2c000, 36864, MADV_DONTNEED) = 0\r\n    madvise(0x7f00819de000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f008198e000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f00819c8000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f005a7d8000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005a7e2000, 36864, MADV_DONTNEED) = 0\r\n    madvise(0x7f005a7f8000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f0059815000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f0059829000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f00598cd000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f0059924000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f0059933000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f0059967000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f0059985000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f00599be000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f00599c2000, 36864, MADV_DONTNEED) = 0\r\n    madvise(0x7f00599e6000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f0059b41000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f00594a4000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f005979f000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f00597a8000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f00594cc000, 36864, MADV_DONTNEED) = 0\r\n    madvise(0x7f00594ea000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f0059513000, 36864, MADV_DONTNEED) = 0\r\n    madvise(0x7f0059557000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f0059590000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f00595c8000, 36864, MADV_DONTNEED) = 0\r\n    madvise(0x7f0059615000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f0059643000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f005965d000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f0059678000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f005968f000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f00596b1000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f00596f9000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f0059707000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f0059758000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f0059789000, 36864, MADV_DONTNEED) = 0\r\n    madvise(0x7f00597e1000, 36864, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b166000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b227000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b290000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b3e6000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b097000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b10e000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b11a000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b138000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b170000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b19c000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b1a2000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b1bb000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b202000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b21d000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b26c000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b2fc000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005a11f000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f005a19d000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f005a3d7000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f005a02b000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005a02f000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005a064000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005a06c000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005a095000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005a0e7000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005a0ef000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005a0f5000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005a2c7000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f005a2ce000, 24576, MADV_DONTNEED) = 0\r\n\r\nThey come fast enough to hang my terminal, so that's something. Littered in between them would be valid execution of the PHP code, like this:\r\n\r\n    Trace of process 7815 - /usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemon -vPidFile=/var\r\n\r\n    madvise(0x7f0064c89000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f0064c9e000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f0064e27000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f0064e40000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f0064e87000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f0064f1e000, 24576, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b740000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b759000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b798000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b7cd000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b7e3000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b7ac000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f005b7f1000, 24576, MADV_DONTNEED) = 0\r\n    write(16, \"8c21240 8bd PHP::/var/data/xenfo\"..., 86) = 86\r\n    write(16, \"106b2105 640 PHP::/var/data/xenf\"..., 87) = 87\r\n    madvise(0x7f0081a4a000, 36864, MADV_DONTNEED) = 0\r\n    madvise(0x7f0081a66000, 36864, MADV_DONTNEED) = 0\r\n    madvise(0x7f0081b70000, 36864, MADV_DONTNEED) = 0\r\n    madvise(0x7f0080115000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f008016e000, 12288, MADV_DONTNEED) = 0\r\n    madvise(0x7f007f103000, 12288, MADV_DONTNEED) = 0\r\n\r\nTellingly, this madvise storm continues after the request has been serviced.\r\n\r\nThe logs do not show anything of interest.\r\n\r\nMy configuration file is like this:\r\n\r\n    ; php options\r\n    pid = /var/run/hhvm/pid\r\n    ; hhvm specific \r\n    ;hhvm.server.port = 9000\r\n    hhvm.server.file_socket=/var/run/hhvm/hhvm.sock\r\n    hhvm.server.type = fastcgi\r\n    hhvm.server.default_document = index.php\r\n    hhvm.log.use_log_file = true\r\n    hhvm.log.file = /var/log/hhvm/error.log\r\n    hhvm.repo.central.path = /var/run/hhvm/hhvm.hhbc\r\n    hhvm.log.always_log_unhandled_exceptions = true\r\n    hhvm.log.level = Debug\r\n    hhvm.log.always_log_unhandled_exceptions = true\r\n    hhvm.log.no_silencer = true\r\n    hhvm.log.injectedstacktrace = true\r\n    hhvm.log.nativestacktrace = true\r\n    hhvm.log.MaxMessagesPerRequest = -1\r\n\r\nI'm unable to get a stack trace via gdb because gdb segfaults trying to load hhvm's symbols. This behavior is exhibited in both the regular hhvm package and the hhvm-dbg packages. I can, of course, use gdb just fine with any other process.\r\n\r\nI suppose at this point I should recompile from source instead of using the package with different variations to find out what's going on. Any better advice? "},"comment":{"url":"https://api.github.com/repos/facebook/hhvm/issues/comments/171378235","html_url":"https://github.com/facebook/hhvm/issues/6715#issuecomment-171378235","issue_url":"https://api.github.com/repos/facebook/hhvm/issues/6715","id":171378235,"user":{"login":"fredemmott","id":360927,"avatar_url":"https://avatars.githubusercontent.com/u/360927?v=3","gravatar_id":"","url":"https://api.github.com/users/fredemmott","html_url":"https://github.com/fredemmott","followers_url":"https://api.github.com/users/fredemmott/followers","following_url":"https://api.github.com/users/fredemmott/following{/other_user}","gists_url":"https://api.github.com/users/fredemmott/gists{/gist_id}","starred_url":"https://api.github.com/users/fredemmott/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fredemmott/subscriptions","organizations_url":"https://api.github.com/users/fredemmott/orgs","repos_url":"https://api.github.com/users/fredemmott/repos","events_url":"https://api.github.com/users/fredemmott/events{/privacy}","received_events_url":"https://api.github.com/users/fredemmott/received_events","type":"User","site_admin":false},"created_at":"2016-01-13T17:51:31Z","updated_at":"2016-01-13T17:51:31Z","body":"For both of you, `hhvm.jit_pseudomain=false` looks like it might solve the problem; there are some badly handled cases for code in the top levels of files (not in functions/classes) related to locals not really being local in that case."}}