{"action":"created","issue":{"url":"https://api.github.com/repos/tweag/rules_haskell/issues/35","repository_url":"https://api.github.com/repos/tweag/rules_haskell","labels_url":"https://api.github.com/repos/tweag/rules_haskell/issues/35/labels{/name}","comments_url":"https://api.github.com/repos/tweag/rules_haskell/issues/35/comments","events_url":"https://api.github.com/repos/tweag/rules_haskell/issues/35/events","html_url":"https://github.com/tweag/rules_haskell/issues/35","id":284198323,"node_id":"MDU6SXNzdWUyODQxOTgzMjM=","number":35,"title":"Benchmark support","user":{"login":"Fuuzetsu","id":893115,"node_id":"MDQ6VXNlcjg5MzExNQ==","avatar_url":"https://avatars2.githubusercontent.com/u/893115?v=4","gravatar_id":"","url":"https://api.github.com/users/Fuuzetsu","html_url":"https://github.com/Fuuzetsu","followers_url":"https://api.github.com/users/Fuuzetsu/followers","following_url":"https://api.github.com/users/Fuuzetsu/following{/other_user}","gists_url":"https://api.github.com/users/Fuuzetsu/gists{/gist_id}","starred_url":"https://api.github.com/users/Fuuzetsu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Fuuzetsu/subscriptions","organizations_url":"https://api.github.com/users/Fuuzetsu/orgs","repos_url":"https://api.github.com/users/Fuuzetsu/repos","events_url":"https://api.github.com/users/Fuuzetsu/events{/privacy}","received_events_url":"https://api.github.com/users/Fuuzetsu/received_events","type":"User","site_admin":false},"labels":[{"id":820054968,"node_id":"MDU6TGFiZWw4MjAwNTQ5Njg=","url":"https://api.github.com/repos/tweag/rules_haskell/labels/P3","name":"P3","color":"fbca04","default":false},{"id":741843639,"node_id":"MDU6TGFiZWw3NDE4NDM2Mzk=","url":"https://api.github.com/repos/tweag/rules_haskell/labels/type:%20feature%20request","name":"type: feature request","color":"0052cc","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-12-22T16:09:21Z","updated_at":"2018-11-01T13:56:57Z","closed_at":null,"author_association":"COLLABORATOR","body":"We may want some benchmark support. Some ideas:\r\n\r\n* benchmarks that produce HTML/json/whatever: think criterion outputs\r\n* it could just be a `haskell_binary` with some bazel tag we pick that identifies it as a benchmark: at the very least the user can ask bazel to list/run all rules with the tag then.\r\n\r\nI think we definitely should just start with haskell_binary alias with a rule tag."},"comment":{"url":"https://api.github.com/repos/tweag/rules_haskell/issues/comments/435048452","html_url":"https://github.com/tweag/rules_haskell/issues/35#issuecomment-435048452","issue_url":"https://api.github.com/repos/tweag/rules_haskell/issues/35","id":435048452,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNTA0ODQ1Mg==","user":{"login":"regnat","id":7226587,"node_id":"MDQ6VXNlcjcyMjY1ODc=","avatar_url":"https://avatars2.githubusercontent.com/u/7226587?v=4","gravatar_id":"","url":"https://api.github.com/users/regnat","html_url":"https://github.com/regnat","followers_url":"https://api.github.com/users/regnat/followers","following_url":"https://api.github.com/users/regnat/following{/other_user}","gists_url":"https://api.github.com/users/regnat/gists{/gist_id}","starred_url":"https://api.github.com/users/regnat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/regnat/subscriptions","organizations_url":"https://api.github.com/users/regnat/orgs","repos_url":"https://api.github.com/users/regnat/repos","events_url":"https://api.github.com/users/regnat/events{/privacy}","received_events_url":"https://api.github.com/users/regnat/received_events","type":"User","site_admin":false},"created_at":"2018-11-01T13:56:57Z","updated_at":"2018-11-01T13:56:57Z","author_association":"CONTRIBUTOR","body":"At novadiscovery, we use a `haskell_test` for that, with a special `benchmark` tag which is excluded by default, so that `bazel test //...` won't run the benchmarks and `bazel test //... --test_tags_filter=benchmark` will run all of them.\r\n\r\nTo keep the outputs, we use a custom criterion runner (*i.e.* a wrapper around `Criterion.defaultMain`) which puts all the outputs under `$TEST_UNDECLARED_OUTPUTS_DIR`, so that they are available after under `bazel-testlogs/my/target/test.outputs/result.{html,csv,json}`\r\n\r\nThat woks quite well for us, though that require some specific setup inside the repository (a `test --test_tag_filters=-benchmark` in `.bazelrc` and a library exporting the custom runner)."}}
{"actor":{"display_login":"regnat"}}