{"action":"created","issue":{"url":"https://api.github.com/repos/humanmade/Cavalcade-Runner/issues/48","repository_url":"https://api.github.com/repos/humanmade/Cavalcade-Runner","labels_url":"https://api.github.com/repos/humanmade/Cavalcade-Runner/issues/48/labels{/name}","comments_url":"https://api.github.com/repos/humanmade/Cavalcade-Runner/issues/48/comments","events_url":"https://api.github.com/repos/humanmade/Cavalcade-Runner/issues/48/events","html_url":"https://github.com/humanmade/Cavalcade-Runner/issues/48","id":336061900,"node_id":"MDU6SXNzdWUzMzYwNjE5MDA=","number":48,"title":"Proposal: exit runner completely, defer to Systemd for restart","user":{"login":"nathanielks","id":314009,"node_id":"MDQ6VXNlcjMxNDAwOQ==","avatar_url":"https://avatars0.githubusercontent.com/u/314009?v=4","gravatar_id":"","url":"https://api.github.com/users/nathanielks","html_url":"https://github.com/nathanielks","followers_url":"https://api.github.com/users/nathanielks/followers","following_url":"https://api.github.com/users/nathanielks/following{/other_user}","gists_url":"https://api.github.com/users/nathanielks/gists{/gist_id}","starred_url":"https://api.github.com/users/nathanielks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nathanielks/subscriptions","organizations_url":"https://api.github.com/users/nathanielks/orgs","repos_url":"https://api.github.com/users/nathanielks/repos","events_url":"https://api.github.com/users/nathanielks/events{/privacy}","received_events_url":"https://api.github.com/users/nathanielks/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-06-27T03:10:39Z","updated_at":"2018-06-27T23:17:44Z","closed_at":null,"author_association":"MEMBER","body":"## TL;DR:\r\n\r\nI'm concerned about php's ability to be a long running script, so I'm wondering what you would think about the following:\r\n\r\nWhat if, instead of using a while loop and sleeping, we run the script to completion and allow upstart/systemd to manage restarting the process? I'm testing it currently over the last ~3 hours and memory consumption is consistent; it's not growing. \r\n\r\nI added the following to the systemd definition:\r\n\r\n```\r\nRestartSec=1\r\nStartLimitBurst=0\r\n``` \r\n\r\nI realize means we'd have to spawn a connection to the database every second, which I don't think is _terrible_, but not ideal either. This could be changed to 5 second or 10, perhaps.\r\n\r\n---\r\n\r\nWe've been experiencing some issues with memory leaks lately, so I've been doing some testing. I used Xdebug's function tracing ability to see which functions are consuming memory, and here are the top 10 consumers:\r\n\r\n```\r\nPDOStatement->execute                              896  1.2630 33392944  1.2630 33392944\r\nsleep                                              896  895.2948 16176472  895.2948 16176472\r\nPDO->prepare                                       896  1.1511  7825664  1.1511  7825664\r\nHM\\Platform\\boostrap_cavalcade_runner                1  0.0044   531408  0.0013   220472\r\nHM\\Cavalcade\\Runner\\autoload                         4  0.0011    76504  0.0009    76168\r\ninclude                                              7  0.0148   575480  0.0017    34376\r\nPDO->__construct                                     1  0.0389    11392  0.0389    11392\r\nHM\\Cavalcade\\Runner\\Runner->bootstrap                1  0.0551   591656  0.0002     4784\r\nHM\\Cavalcade\\Runner\\Hooks->register                  3  0.0003     2704  0.0001     2704\r\ndefine                                              51  0.0005     2000  0.0005     2000\r\n```\r\n\r\nSeeing as `PDOStatement->execute` was the top consumer, I went through the code and `unset($statement)` after each statement and ran Cavalcade for some time. Even with this, I can see memory usage growing.\r\n\r\nA flaw in my thought process is that the function trace shows the memory consumed over the course of the script, but not necessarily what's being consumed at that moment. I made use of [`https://github.com/BitOne/php-meminfo`](https://github.com/BitOne/php-meminfo) and [`https://github.com/arnaud-lb/php-memory-profiler`](https://github.com/arnaud-lb/php-memory-profiler), and they unfortunately didn't reveal anything. Well, they did reveal that the application isn't leaking anything because the memory allocated was [always _well_ below](https://github.com/BitOne/php-meminfo/issues/63#issuecomment-399576204) what was allocated to the process. \r\n\r\nOne thing that _is_ consistent is that `sleep` seems to consume a lot of memory. Reading around, I've seen that php isn't really designed to be a long running process, so I'm concerned about it in this instance."},"comment":{"url":"https://api.github.com/repos/humanmade/Cavalcade-Runner/issues/comments/400859578","html_url":"https://github.com/humanmade/Cavalcade-Runner/issues/48#issuecomment-400859578","issue_url":"https://api.github.com/repos/humanmade/Cavalcade-Runner/issues/48","id":400859578,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMDg1OTU3OA==","user":{"login":"nathanielks","id":314009,"node_id":"MDQ6VXNlcjMxNDAwOQ==","avatar_url":"https://avatars0.githubusercontent.com/u/314009?v=4","gravatar_id":"","url":"https://api.github.com/users/nathanielks","html_url":"https://github.com/nathanielks","followers_url":"https://api.github.com/users/nathanielks/followers","following_url":"https://api.github.com/users/nathanielks/following{/other_user}","gists_url":"https://api.github.com/users/nathanielks/gists{/gist_id}","starred_url":"https://api.github.com/users/nathanielks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nathanielks/subscriptions","organizations_url":"https://api.github.com/users/nathanielks/orgs","repos_url":"https://api.github.com/users/nathanielks/repos","events_url":"https://api.github.com/users/nathanielks/events{/privacy}","received_events_url":"https://api.github.com/users/nathanielks/received_events","type":"User","site_admin":false},"created_at":"2018-06-27T23:17:44Z","updated_at":"2018-06-27T23:17:44Z","author_association":"MEMBER","body":"I was curious if PDO really was the issue, so I switched PDO out for `mysqli` in the Runner class. This has made a significant difference. Looking at the function traces I can see that, when closing out the cursor, the memory consumed by mysqli is getting cleaned up:\r\n\r\n```\r\nfunction                                 #calls  time     memory  time     memory\r\n---------------------------------------------------------------------------------\r\nmysqli_stmt->get_result                     630  0.0152 12605776  0.0152 12605776\r\nmysqli->prepare                             630  0.7920  5175392  0.7920  5175392\r\n/// a few other functions\r\nmysqli_stmt->close                          630  0.0188 -5019840  0.0188 -5019840\r\nHM\\Cavalcade\\Runner\\Runner->get_next_job    630  1.8087    50864  0.0448 -12746160\r\n```\r\n\r\nUnfortunately I can also see that `sleep` is still accruing memory that's not getting cleaned up:\r\n\r\n```\r\nsleep                                       324  324.1002  5327808  324.1002  5327808\r\n// A few minutes later\r\nsleep                                       629  629.1984 10413016  629.1984 10413016\r\n```\r\n( It grows from 5M to 9M ).\r\n\r\nThis alone is an improvement. With PDO, I saw that memory was increasing 10M each minute. Switching out for `mysqli` reduces that to 2M/minute.\r\n\r\nI'm going to watch the process over the next day and see what it does. It looks like `sleep` is still an issue 🤔 "}}
{"actor":{"display_login":"nathanielks"}}