{"action":"created","issue":{"url":"https://api.github.com/repos/realm/realm-core/issues/2368","repository_url":"https://api.github.com/repos/realm/realm-core","labels_url":"https://api.github.com/repos/realm/realm-core/issues/2368/labels{/name}","comments_url":"https://api.github.com/repos/realm/realm-core/issues/2368/comments","events_url":"https://api.github.com/repos/realm/realm-core/issues/2368/events","html_url":"https://github.com/realm/realm-core/pull/2368","id":196529224,"number":2368,"title":"Automatic Performance Tests on Core PRs","user":{"login":"ironage","id":2826060,"avatar_url":"https://avatars.githubusercontent.com/u/2826060?v=3","gravatar_id":"","url":"https://api.github.com/users/ironage","html_url":"https://github.com/ironage","followers_url":"https://api.github.com/users/ironage/followers","following_url":"https://api.github.com/users/ironage/following{/other_user}","gists_url":"https://api.github.com/users/ironage/gists{/gist_id}","starred_url":"https://api.github.com/users/ironage/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ironage/subscriptions","organizations_url":"https://api.github.com/users/ironage/orgs","repos_url":"https://api.github.com/users/ironage/repos","events_url":"https://api.github.com/users/ironage/events{/privacy}","received_events_url":"https://api.github.com/users/ironage/received_events","type":"User","site_admin":false},"labels":[{"id":320979666,"url":"https://api.github.com/repos/realm/realm-core/labels/S:In%20Progress","name":"S:In Progress","color":"dedede","default":false},{"id":320979145,"url":"https://api.github.com/repos/realm/realm-core/labels/T:Internal","name":"T:Internal","color":"c7def8","default":false}],"state":"open","locked":false,"assignee":{"login":"ironage","id":2826060,"avatar_url":"https://avatars.githubusercontent.com/u/2826060?v=3","gravatar_id":"","url":"https://api.github.com/users/ironage","html_url":"https://github.com/ironage","followers_url":"https://api.github.com/users/ironage/followers","following_url":"https://api.github.com/users/ironage/following{/other_user}","gists_url":"https://api.github.com/users/ironage/gists{/gist_id}","starred_url":"https://api.github.com/users/ironage/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ironage/subscriptions","organizations_url":"https://api.github.com/users/ironage/orgs","repos_url":"https://api.github.com/users/ironage/repos","events_url":"https://api.github.com/users/ironage/events{/privacy}","received_events_url":"https://api.github.com/users/ironage/received_events","type":"User","site_admin":false},"assignees":[{"login":"ironage","id":2826060,"avatar_url":"https://avatars.githubusercontent.com/u/2826060?v=3","gravatar_id":"","url":"https://api.github.com/users/ironage","html_url":"https://github.com/ironage","followers_url":"https://api.github.com/users/ironage/followers","following_url":"https://api.github.com/users/ironage/following{/other_user}","gists_url":"https://api.github.com/users/ironage/gists{/gist_id}","starred_url":"https://api.github.com/users/ironage/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ironage/subscriptions","organizations_url":"https://api.github.com/users/ironage/orgs","repos_url":"https://api.github.com/users/ironage/repos","events_url":"https://api.github.com/users/ironage/events{/privacy}","received_events_url":"https://api.github.com/users/ironage/received_events","type":"User","site_admin":false}],"milestone":null,"comments":3,"created_at":"2016-12-19T21:48:53Z","updated_at":"2016-12-20T11:40:56Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/realm/realm-core/pulls/2368","html_url":"https://github.com/realm/realm-core/pull/2368","diff_url":"https://github.com/realm/realm-core/pull/2368.diff","patch_url":"https://github.com/realm/realm-core/pull/2368.patch"},"body":"**Please hold thorough reviews until we have integrated to CI**\r\nI am still looking for guidance from @emanuelez to implement the three fixme's in the new Jenkinsfile.performance file and how to link to the results generated to PRs in GitHub.\r\n\r\nThis will add the ability for CI to generate performance graphs comparing the current HEAD of the PR to previous releases (specified in \"test/bench/revs_to_benchmark.txt\"). The goal is to be able to easily see if a PR has unexpected performance side effects, viewing the relative difference of benchmarks across releases.\r\n\r\nOverview of design for reviewers:\r\nScripts will checkout, build and run the benchmarks for each version of core listed in `revs_to_benchmark.txt`. The results are written to a csv file for that version of core in the specified output directory. The scripts automatically create a different internal directory structure based on a hardware id and benchmark scripts (`test/bench/benchmark_version`). This structure will prevent benchmarks from being used across different machines and also allow us to cache previous results per machine so that instead of running benchmarks for all versions on each PR, we can reuse the previous results of old core versions. If new benchmarks are added or existing benchmarks are modified, the benchmark version should be bumped so that all previous core versions should be rebuilt with the new code.\r\n\r\nOnce the benchmark results of each core version are obtained, the `parse_bench_hist.py` script will read each result and generate graphs by benchmark type across all versions. A `report.html` page is then generated to view all graphs. The graphs are generated with the python matplotlib library (version 1.3.1).\r\n\r\nIt could take around 6 hours on the first run to get benchmark results for all (22) versions of core specified, but with the results caching system in place, it should take no more than 10-15 minutes to get performance results of a normal PR (which doesn't modify benchmark code)."},"comment":{"url":"https://api.github.com/repos/realm/realm-core/issues/comments/268222708","html_url":"https://github.com/realm/realm-core/pull/2368#issuecomment-268222708","issue_url":"https://api.github.com/repos/realm/realm-core/issues/2368","id":268222708,"user":{"login":"realm-ci","id":7657856,"avatar_url":"https://avatars.githubusercontent.com/u/7657856?v=3","gravatar_id":"","url":"https://api.github.com/users/realm-ci","html_url":"https://github.com/realm-ci","followers_url":"https://api.github.com/users/realm-ci/followers","following_url":"https://api.github.com/users/realm-ci/following{/other_user}","gists_url":"https://api.github.com/users/realm-ci/gists{/gist_id}","starred_url":"https://api.github.com/users/realm-ci/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/realm-ci/subscriptions","organizations_url":"https://api.github.com/users/realm-ci/orgs","repos_url":"https://api.github.com/users/realm-ci/repos","events_url":"https://api.github.com/users/realm-ci/events{/privacy}","received_events_url":"https://api.github.com/users/realm-ci/received_events","type":"User","site_admin":false},"created_at":"2016-12-20T11:40:56Z","updated_at":"2016-12-20T11:40:56Z","body":"\n\nPlease check your coverage here: https://ci.realm.io/job/realm/job/realm-core/job/PR-2368/3/Diff_Coverage"}}
{"actor":{"display_login":"realm-ci"}}