{"action":"created","issue":{"url":"https://api.github.com/repos/ldionne/metabench/issues/108","repository_url":"https://api.github.com/repos/ldionne/metabench","labels_url":"https://api.github.com/repos/ldionne/metabench/issues/108/labels{/name}","comments_url":"https://api.github.com/repos/ldionne/metabench/issues/108/comments","events_url":"https://api.github.com/repos/ldionne/metabench/issues/108/events","html_url":"https://github.com/ldionne/metabench/pull/108","id":148726690,"number":108,"title":"Allow mitigating the base cost of benchmarks","user":{"login":"ldionne","id":700834,"avatar_url":"https://avatars.githubusercontent.com/u/700834?v=3","gravatar_id":"","url":"https://api.github.com/users/ldionne","html_url":"https://github.com/ldionne","followers_url":"https://api.github.com/users/ldionne/followers","following_url":"https://api.github.com/users/ldionne/following{/other_user}","gists_url":"https://api.github.com/users/ldionne/gists{/gist_id}","starred_url":"https://api.github.com/users/ldionne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ldionne/subscriptions","organizations_url":"https://api.github.com/users/ldionne/orgs","repos_url":"https://api.github.com/users/ldionne/repos","events_url":"https://api.github.com/users/ldionne/events{/privacy}","received_events_url":"https://api.github.com/users/ldionne/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ldionne/metabench/labels/feature","name":"feature","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2016-04-15T18:01:25Z","updated_at":"2016-04-15T19:53:45Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/ldionne/metabench/pulls/108","html_url":"https://github.com/ldionne/metabench/pull/108","diff_url":"https://github.com/ldionne/metabench/pull/108.diff","patch_url":"https://github.com/ldionne/metabench/pull/108.patch"},"body":"Right now, benchmarks look like\r\n\r\n```c++\r\n#include <boost/hana/count_if.hpp>\r\n#include <boost/hana/tuple.hpp>\r\n\r\n#include <type_traits>\r\nnamespace hana = boost::hana;\r\n\r\n\r\nstruct is_even {\r\n    template <typename N>\r\n    constexpr bool operator()(N const&) const {\r\n        return N::value % 2 == 0;\r\n    }\r\n};\r\n\r\nint main() {\r\n    auto ints = hana::make_tuple(\r\n        <%= (1..n).map { |i| \"std::integral_constant<int, #{i}>{}\" }.join(', ') %>\r\n    );\r\n    auto result = hana::count_if(ints, is_even{});\r\n    (void)result;\r\n}\r\n```\r\n\r\nThe problem with this is that we're benchmarking both the cost of creating the sequence and that of the `count_if` algorithm. This is misleading, since the cost of creating the sequence is almost always much larger than the cost of performing the algorithm. This causes almost all the benchmarks to look like they have a gentle linear performance, while this really is the creation of the sequence. To mitigate this, it was suggested that we benchmark the cost of creating the sequence itself, and subtract it from the cost of the whole.\r\n\r\nThe goal of this PR is to explore whether this seems useful, and whether subtracting the base cost is a reliable way of achieving our goal. Right now, this is only a WIP. For a dataset with a source file named `xxx.cpp.erb`, if you provide a  `xxx.cpp.erb.base` file, that file will be compiled like the normal `.cpp.erb` file and its compile-time will be subtracted from it. If we go this way, I have something else in mind for the implementation, but for now we should just figure out whether this is desirable.\r\n\r\n@brunocodutra Your thoughts on this would be appreciated."},"comment":{"url":"https://api.github.com/repos/ldionne/metabench/issues/comments/210616767","html_url":"https://github.com/ldionne/metabench/pull/108#issuecomment-210616767","issue_url":"https://api.github.com/repos/ldionne/metabench/issues/108","id":210616767,"user":{"login":"brunocodutra","id":4043663,"avatar_url":"https://avatars.githubusercontent.com/u/4043663?v=3","gravatar_id":"","url":"https://api.github.com/users/brunocodutra","html_url":"https://github.com/brunocodutra","followers_url":"https://api.github.com/users/brunocodutra/followers","following_url":"https://api.github.com/users/brunocodutra/following{/other_user}","gists_url":"https://api.github.com/users/brunocodutra/gists{/gist_id}","starred_url":"https://api.github.com/users/brunocodutra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brunocodutra/subscriptions","organizations_url":"https://api.github.com/users/brunocodutra/orgs","repos_url":"https://api.github.com/users/brunocodutra/repos","events_url":"https://api.github.com/users/brunocodutra/events{/privacy}","received_events_url":"https://api.github.com/users/brunocodutra/received_events","type":"User","site_admin":false},"created_at":"2016-04-15T19:53:45Z","updated_at":"2016-04-15T19:53:45Z","body":"> Our minds seem to converge; this is the \"something else I have in mind\" I hinted about before. \r\n\r\nAwesome; in fact I did suspect it were.\r\n\r\n> This means that whatever base dataset we're using, it's going to have to be generated for all the values of n that we use across all the benchmarks.\r\n\r\nI think we just need to carefully craft ranges to make sure they match each other.\r\nThis relates to #106. \r\n\r\n> Different benchmarks create slightly different sequences. For example, the count_if benchmark uses a hana::make_tuple(std::integral_constant<int, 1>{}...), while the transform benchmark uses a hana::make_tuple(x<1>{}...). \r\n\r\nI don't think it would be too hard to make most benchmarks use the same sequences. Take your example, for instance, we could just as well [define a metafunction for `x<_>`](https://github.com/ldionne/metabench/blob/master/benchmark/type/replace_if/hana.types.cpp.erb). Sure that won't apply to evey case, take `at` for instance which should use a sequence of the same size for all `n`, but perhaps in this case we want a constant OFFSET instead of a BASE_DATASET.\r\n\r\n> Some benchmarks might also #include slightly different files. This could cause the base adjustment to be (perhaps subtly) wrong. Do we care about this? The other solution is to explicitly state the code that should be used as a base for each dataset.\r\n\r\nYou are right, each DATASET has its own base, I mean, it makes no sense to subtract the same base value from all datasets that compose a benchmark. Think MPL which preprocess a whole bunch of nasty macros, if we subtracted it from brigand for example we will most probably see negative values.\r\n\r\n> In summary, the two solutions I see are\r\n> 1. Reuse the same dataset as a base for all benchmarks that want it\r\n\r\nIMO this won't work for the reasons above.\r\n\r\n> 2. Provide a way to exclude base code on a per-dataset basis\r\n\r\nI think we can achieve it by adding a new parameter BASE or BIAS to `metabench_add_dataset`, which should point to a `cpp.erb` template. Naturally several datasets could use the same template as a BASE and it must be guaranteed that a given template will only be compiled once, i.e., during compilation of the first dataset that uses it. The way we setup `metabench_ada_dataset` I believe this would be easy to guarantee by relying on CMake and the makefile generator directly."}}