{"action":"created","issue":{"url":"https://api.github.com/repos/phpbench/phpbench/issues/252","repository_url":"https://api.github.com/repos/phpbench/phpbench","labels_url":"https://api.github.com/repos/phpbench/phpbench/issues/252/labels{/name}","comments_url":"https://api.github.com/repos/phpbench/phpbench/issues/252/comments","events_url":"https://api.github.com/repos/phpbench/phpbench/issues/252/events","html_url":"https://github.com/phpbench/phpbench/issues/252","id":124349034,"number":252,"title":"Provide a benchmark dump + eventually consider committing benchmark dumps automatically","user":{"login":"Ocramius","id":154256,"avatar_url":"https://avatars.githubusercontent.com/u/154256?v=3","gravatar_id":"","url":"https://api.github.com/users/Ocramius","html_url":"https://github.com/Ocramius","followers_url":"https://api.github.com/users/Ocramius/followers","following_url":"https://api.github.com/users/Ocramius/following{/other_user}","gists_url":"https://api.github.com/users/Ocramius/gists{/gist_id}","starred_url":"https://api.github.com/users/Ocramius/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Ocramius/subscriptions","organizations_url":"https://api.github.com/users/Ocramius/orgs","repos_url":"https://api.github.com/users/Ocramius/repos","events_url":"https://api.github.com/users/Ocramius/events{/privacy}","received_events_url":"https://api.github.com/users/Ocramius/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/phpbench/phpbench/labels/discussion","name":"discussion","color":"eb6420"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-12-30T13:39:46Z","updated_at":"2016-02-17T13:51:10Z","closed_at":"2016-02-17T09:52:43Z","body":"While I totally appreciate benchmarking libraries, I found all of them (including those that I wrote myself) useless without historical tracking of the performance changes.\r\n\r\nIf `phpbench` can output (and read) this historical data, then we'd be able to actually track performance improvements.\r\n\r\nWhat is needed for this is following:\r\n\r\n### Baseline performance checking\r\n\r\nWe need a way to define a \"baseline\" performance (for example executing an empty method) to normalize results across different machines. Not very reliable, but better than nothing, and important for CI environments where a dedicated server for performance testing is not viable (99.9% of OSS projects).\r\n\r\nAthletic [has something like that](https://github.com/polyfractal/athletic/blob/ab1f8134ebba46aa08e18f7bb1426e8db474001c/src/Athletic/AthleticEvent.php#L143-L156) via an empty calibration method. /cc @gabrielsch\r\n\r\n### Dump format (versioned)\r\n\r\nThe output of the tests should be dumped, for example:\r\n\r\n```json\r\n{\r\n    \"pwd\": \"/where/did/we/execute/the/tests\",\r\n    \"date\": 123123123,\r\n    \"system-load\": { // add before/after?\r\n        \"15m\": 1.69,\r\n        \"5m\": 1.73,\r\n        \"1s\": 1.79\r\n    },\r\n    \"results\": [ /* loads of stuff here*/ ]\r\n}\r\n```\r\n\r\nThe output must include versioning metadata. If we add or change fields, we should still be able to parse the json:\r\n\r\n```json\r\n{\"phpbench-version\": \"1.2.3\"}\r\n```\r\n\r\nThen we can simply load the data:\r\n\r\n```php\r\n$benchResults = $benchResultsLoader->fromArray(json_decode($someJson, true));\r\n```\r\n\r\n### Saving results\r\n\r\nResults of the benchmarking must be saved somewhere. Since they are part of the project, I suggest saving them in the project itself, in the git tree.\r\n\r\nMy suggestion is to have a `phpbench-history-{$branchName}` branch, where each commit is containing:\r\n * a file with the current benchmark results (see suggested output format above)\r\n * eventually, a merge commit from the parent sha1 (the branch where the tested commit comes from: basically, we force both parent commits to come from `phpbench-history-{$branchName}` and `$branchName` respectively.\r\nNote that there may be more than one benchmark per commit hash: we want to store them all.\r\n\r\nThis is a very simple yet powerful approach, and allows every project to adopt historical performance analysis.\r\n\r\n### Auto-commit in CI environments\r\n\r\nBranches are cheap: let's create a load of them! (users are free to delete them, after all!)\r\n\r\nI suggest taking an approach similar to what @mnapoli did with @CouscousPHP: see https://github.com/CouscousPHP/Couscous/blob/5af53e2346649110f8ce0333bacfec6c2a35ad6e/docs/travis.md\r\n\r\nBasically, we simply push a branch from travis at every build.\r\n\r\n### Analysis\r\n\r\nThis is probably for a separate project, but it is indeed where the results can be seen after all the efforts.\r\n\r\nOnce we have this shit-load (pardon the technical term) of data, we may generate diagrams and web-pages showing the project's health per-environment per-branch per-commit (etc etc).\r\n\r\nThis sort of thing can also live in the git repo (`gh-pages`, y'know), but I am the wrong person to talk to when it comes to data visualization.\r\n\r\nIf the output format and the dumper are to be split into separate projects, then this sort of approach can be reused for a lot of different scenarios (not just PHP-specific)."},"comment":{"url":"https://api.github.com/repos/phpbench/phpbench/issues/comments/185211669","html_url":"https://github.com/phpbench/phpbench/issues/252#issuecomment-185211669","issue_url":"https://api.github.com/repos/phpbench/phpbench/issues/252","id":185211669,"user":{"login":"dantleech","id":530801,"avatar_url":"https://avatars.githubusercontent.com/u/530801?v=3","gravatar_id":"","url":"https://api.github.com/users/dantleech","html_url":"https://github.com/dantleech","followers_url":"https://api.github.com/users/dantleech/followers","following_url":"https://api.github.com/users/dantleech/following{/other_user}","gists_url":"https://api.github.com/users/dantleech/gists{/gist_id}","starred_url":"https://api.github.com/users/dantleech/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dantleech/subscriptions","organizations_url":"https://api.github.com/users/dantleech/orgs","repos_url":"https://api.github.com/users/dantleech/repos","events_url":"https://api.github.com/users/dantleech/events{/privacy}","received_events_url":"https://api.github.com/users/dantleech/received_events","type":"User","site_admin":false},"created_at":"2016-02-17T13:51:10Z","updated_at":"2016-02-17T13:51:10Z","body":"Also lets continue the discussion in #333 "}}