{"push_id":1414885268,"size":1,"distinct_size":1,"ref":"refs/heads/so-stable-performance-tests","head":"54d691d4780b25e9b989ed04356f755d2b70e973","before":"bd1ceecf95f73196e764f092edbffaf41d52964a","commits":[{"sha":"54d691d4780b25e9b989ed04356f755d2b70e973","author":{"name":"Stefan Oehme","email":"6fe76ede026e27600863a1ff38da4e776dcaecdf@gradle.com"},"message":"Simplify performance measurements\n\nThe many measurements that we injected into the build under test were\nskewing our measurements to the point of making them unreliable or\nunrealistic. We now only measure end to end build time. Here's a breakdown\nwith the rationale for removing each other measurement:\n\n- configuration time: can be done by having a `gradle help` scenario instead\n- execution time: the user does not care whether a long build is stuck in execution or configuration\n- setup/teardown: was ill-defined anyway, basically total - configuration - execution\n- JIT compile time: this is nothing we can influence and thus pointless to measure\n- Memory usage: Was only measured at one point in the build, which doesn't tell us anything about\nany problems at any other point in the build\n- GC CPU time: If this increases we'd see it in total execution time\n\nGenerally, looking at the graphs has never pointed us directly at the problem, we always need to\nprofile anyway. So instead of skewing our measurements with lots of profling code, we should\ninstead use a dedicated profiling job to measure if we actually see a regression.\n\nMemory usage can be tested indirectly by giving each scenario a reasonable amount of memory.\nIf memory usage rises above that reasonable limit, we'd see execution time rise, telling us about\nthe regression. Generally, we do not optimize for smallest memory usage, but for fastest execution\nwith reasonable memory overhead.\n\nThis change also removes all JVM tweaking and wait periods which we introduced in an attempt to\nmake tests more predictable and stable. These tweaks have not really helped us achieve more stable\ntests and have often done the opposite. They also add lots of complexity and make our tests more\nunrealistic. A real users will not add all these JVM options to Gradle.","distinct":true,"url":"https://api.github.com/repos/gradle/gradle/commits/54d691d4780b25e9b989ed04356f755d2b70e973"}]}
{"actor":{"display_login":"oehme"}}