{"action":"created","issue":{"url":"https://api.github.com/repos/catapult-project/catapult/issues/2294","repository_url":"https://api.github.com/repos/catapult-project/catapult","labels_url":"https://api.github.com/repos/catapult-project/catapult/issues/2294/labels{/name}","comments_url":"https://api.github.com/repos/catapult-project/catapult/issues/2294/comments","events_url":"https://api.github.com/repos/catapult-project/catapult/issues/2294/events","html_url":"https://github.com/catapult-project/catapult/issues/2294","id":151926643,"number":2294,"title":"The semantics of --pageset-repeat is not clear","user":{"login":"perezju","id":16922344,"avatar_url":"https://avatars.githubusercontent.com/u/16922344?v=3","gravatar_id":"","url":"https://api.github.com/users/perezju","html_url":"https://github.com/perezju","followers_url":"https://api.github.com/users/perezju/followers","following_url":"https://api.github.com/users/perezju/following{/other_user}","gists_url":"https://api.github.com/users/perezju/gists{/gist_id}","starred_url":"https://api.github.com/users/perezju/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/perezju/subscriptions","organizations_url":"https://api.github.com/users/perezju/orgs","repos_url":"https://api.github.com/users/perezju/repos","events_url":"https://api.github.com/users/perezju/events{/privacy}","received_events_url":"https://api.github.com/users/perezju/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/catapult-project/catapult/labels/Telemetry","name":"Telemetry","color":"5319e7"}],"state":"closed","locked":false,"assignee":{"login":"perezju","id":16922344,"avatar_url":"https://avatars.githubusercontent.com/u/16922344?v=3","gravatar_id":"","url":"https://api.github.com/users/perezju","html_url":"https://github.com/perezju","followers_url":"https://api.github.com/users/perezju/followers","following_url":"https://api.github.com/users/perezju/following{/other_user}","gists_url":"https://api.github.com/users/perezju/gists{/gist_id}","starred_url":"https://api.github.com/users/perezju/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/perezju/subscriptions","organizations_url":"https://api.github.com/users/perezju/orgs","repos_url":"https://api.github.com/users/perezju/repos","events_url":"https://api.github.com/users/perezju/events{/privacy}","received_events_url":"https://api.github.com/users/perezju/received_events","type":"User","site_admin":false},"assignees":[{"login":"perezju","id":16922344,"avatar_url":"https://avatars.githubusercontent.com/u/16922344?v=3","gravatar_id":"","url":"https://api.github.com/users/perezju","html_url":"https://github.com/perezju","followers_url":"https://api.github.com/users/perezju/followers","following_url":"https://api.github.com/users/perezju/following{/other_user}","gists_url":"https://api.github.com/users/perezju/gists{/gist_id}","starred_url":"https://api.github.com/users/perezju/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/perezju/subscriptions","organizations_url":"https://api.github.com/users/perezju/orgs","repos_url":"https://api.github.com/users/perezju/repos","events_url":"https://api.github.com/users/perezju/events{/privacy}","received_events_url":"https://api.github.com/users/perezju/received_events","type":"User","site_admin":false}],"milestone":null,"comments":4,"created_at":"2016-04-29T17:20:50Z","updated_at":"2016-07-07T11:19:49Z","closed_at":"2016-07-07T11:19:49Z","body":"Splitting off from [crbug/592380](https://bugs.chromium.org/p/chromium/issues/detail?id=592380)\r\n\r\nThe main point of the discussion is on whether --pageset-repeat should tear down the state (close and reopen a browser) between iterations or not.\r\n\r\nI'll try to summarize the main contention points:\r\n\r\nTearing down the state:\r\n* keeps different measurements of the same page statistically independent\r\n* allows to reduce noise in measurements\r\n\r\n*Not* tearing down the state [current behavior]:\r\n* stresses the browser by creating a longer running story set\r\n* allows to catch memory leaks that are not apparent on a single iteration\r\n\r\nCopying the last comment from @nedn :\r\n\r\n> Anyway, another reason why one should not use pageset-repeat for the purpose of stress testing the browser is telemetry treats page, e.g: google.com, in the first run the same way as in the 10th run. Hence for monitoring purpose, all you see is the summarized value of 10 runs of google.com which are not really statistically independent.\r\n>\r\n> Instead of using pageset-repeat, you should do:\r\n>\r\n>     class MemoryStorSet:\r\n>       def __init__(self):\r\n>          self.AddStory(GooglePage(..))  # Just navigate to google.com\r\n>          ... \r\n>          ...\r\n>          self.AddStory(GoogleHeavyPage())  # Navigate to google.com and make a lot of search queries\r\n>          ...\r\n>          ...\r\n>\r\n> This would allow you to track the memory metrics of the same google.com pages in two very different settings: when the browser is relatively cold & when it's has been around for a long time. I would expect this would also help drive down the noise a bit.\r\n\r\n@primiano @anniesullie @dave-2 @picksi "},"comment":{"url":"https://api.github.com/repos/catapult-project/catapult/issues/comments/231050968","html_url":"https://github.com/catapult-project/catapult/issues/2294#issuecomment-231050968","issue_url":"https://api.github.com/repos/catapult-project/catapult/issues/2294","id":231050968,"user":{"login":"perezju","id":16922344,"avatar_url":"https://avatars.githubusercontent.com/u/16922344?v=3","gravatar_id":"","url":"https://api.github.com/users/perezju","html_url":"https://github.com/perezju","followers_url":"https://api.github.com/users/perezju/followers","following_url":"https://api.github.com/users/perezju/following{/other_user}","gists_url":"https://api.github.com/users/perezju/gists{/gist_id}","starred_url":"https://api.github.com/users/perezju/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/perezju/subscriptions","organizations_url":"https://api.github.com/users/perezju/orgs","repos_url":"https://api.github.com/users/perezju/repos","events_url":"https://api.github.com/users/perezju/events{/privacy}","received_events_url":"https://api.github.com/users/perezju/received_events","type":"User","site_admin":false},"created_at":"2016-07-07T11:19:48Z","updated_at":"2016-07-07T11:19:48Z","body":"This is done. The final implementation is:\r\n\r\n* Benchmarks by default tear down the state between page set repeats; this is what most users expect and yields more repeatable/deterministic results.\r\n* Benchmark authors can, if really needed, switch to the alternative behavior to keep the browser open for longer periods of time, helping to catch leaks, etc.\r\n\r\nThese are now documented in [telemetry.benchmark](https://github.com/catapult-project/catapult/blob/b3e03f627f8f905a5ed63725c07f615dc1389eac/telemetry/telemetry/benchmark.py#L101)."}}
{"actor":{"display_login":"perezju"}}