{"action":"created","issue":{"url":"https://api.github.com/repos/airspeed-velocity/asv/issues/752","repository_url":"https://api.github.com/repos/airspeed-velocity/asv","labels_url":"https://api.github.com/repos/airspeed-velocity/asv/issues/752/labels{/name}","comments_url":"https://api.github.com/repos/airspeed-velocity/asv/issues/752/comments","events_url":"https://api.github.com/repos/airspeed-velocity/asv/issues/752/events","html_url":"https://github.com/airspeed-velocity/asv/issues/752","id":369278344,"node_id":"MDU6SXNzdWUzNjkyNzgzNDQ=","number":752,"title":"Approximate ratio masks significant benchmark changes","user":{"login":"h-vetinari","id":33685575,"node_id":"MDQ6VXNlcjMzNjg1NTc1","avatar_url":"https://avatars1.githubusercontent.com/u/33685575?v=4","gravatar_id":"","url":"https://api.github.com/users/h-vetinari","html_url":"https://github.com/h-vetinari","followers_url":"https://api.github.com/users/h-vetinari/followers","following_url":"https://api.github.com/users/h-vetinari/following{/other_user}","gists_url":"https://api.github.com/users/h-vetinari/gists{/gist_id}","starred_url":"https://api.github.com/users/h-vetinari/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/h-vetinari/subscriptions","organizations_url":"https://api.github.com/users/h-vetinari/orgs","repos_url":"https://api.github.com/users/h-vetinari/repos","events_url":"https://api.github.com/users/h-vetinari/events{/privacy}","received_events_url":"https://api.github.com/users/h-vetinari/received_events","type":"User","site_admin":false},"labels":[{"id":1036130751,"node_id":"MDU6TGFiZWwxMDM2MTMwNzUx","url":"https://api.github.com/repos/airspeed-velocity/asv/labels/needs-info","name":"needs-info","color":"ffccd8","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-10-11T19:48:44Z","updated_at":"2018-11-18T14:47:18Z","closed_at":null,"author_association":"NONE","body":"ASV sometimes adds a `~` to ratios if there are many sample (with high variance, I'm guessing), but there's a bug somewhere how these results are evaluated when the significant changes are given out.\r\n\r\nAs an example from running the ASV in a pandas PR:\r\n```\r\n(pandas-dev) C:\\[...]\\pddev\\asv_bench>asv continuous -f 1.1 upstream/master 0d3c6d21 -b ^strings.Cat\r\n· Creating environments\r\n· Discovering benchmarks\r\n·· Uninstalling from conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt\r\n·· Building 0d3c6d21 <cln_str_cat~2> for conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt\r\n·· Installing 0d3c6d21 <cln_str_cat~2> into conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt\r\n· Running 2 total benchmarks (2 commits * 1 environments * 1 benchmarks)\r\n[  0.00%] · For pandas commit b28cf5aa <master> (round 1/2):\r\n[  0.00%] ·· Building for conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt\r\n[  0.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt\r\n[ 25.00%] ··· Running (strings.Cat.time_cat--).\r\n[ 25.00%] · For pandas commit 0d3c6d21 <cln_str_cat~2> (round 1/2):\r\n[ 25.00%] ·· Building for conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt\r\n[ 25.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt\r\n[ 50.00%] ··· Running (strings.Cat.time_cat--).\r\n[ 50.00%] · For pandas commit 0d3c6d21 <cln_str_cat~2> (round 2/2):\r\n[ 50.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt\r\n[ 75.00%] ··· strings.Cat.time_cat                                                                                                                                                ok\r\n[ 75.00%] ··· ============ ====== ============ ============== ============= ========== ============ ===========\r\n              --                                                 na_rep / na_frac\r\n              ------------------- -----------------------------------------------------------------------------\r\n               other_cols   sep    None / 0.0   None / 0.001   None / 0.15   - / 0.0    - / 0.001     - / 0.15\r\n              ============ ====== ============ ============== ============= ========== ============ ===========\r\n                   0        None    7.81±2ms     10.9±0.8ms     11.7±0.8ms   9.38±0ms   10.9±0.6ms    12.5±0ms\r\n                   0         ,      9.38±0ms      10.9±0ms      11.7±0.8ms   9.38±0ms    10.9±0ms     12.5±0ms\r\n                   3        None    46.9±8ms      62.5±0ms       62.5±6ms    54.7±8ms    62.5±8ms    78.1±10ms\r\n                   3         ,      78.1±0ms      78.1±0ms       62.5±8ms    78.1±8ms    78.1±0ms     93.8±0ms\r\n              ============ ====== ============ ============== ============= ========== ============ ===========\r\n\r\n[ 75.00%] · For pandas commit b28cf5aa <master> (round 2/2):\r\n[ 75.00%] ·· Building for conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt\r\n[ 75.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt\r\n[100.00%] ··· strings.Cat.time_cat                                                                                                                                                ok\r\n[100.00%] ··· ============ ====== ============ ============== ============= ========== ============ ==========\r\n              --                                                na_rep / na_frac\r\n              ------------------- ----------------------------------------------------------------------------\r\n               other_cols   sep    None / 0.0   None / 0.001   None / 0.15   - / 0.0    - / 0.001    - / 0.15\r\n              ============ ====== ============ ============== ============= ========== ============ ==========\r\n                   0        None    10.9±0ms      9.38±0ms      10.9±0.8ms   9.38±0ms   9.38±0.6ms   10.9±0ms\r\n                   0         ,     9.38±0.6ms    14.1±0.8ms     14.1±0.6ms   9.38±0ms    9.38±0ms    10.9±0ms\r\n                   3        None    62.5±6ms      46.9±6ms       46.9±0ms    54.7±8ms    46.9±8ms    62.5±0ms\r\n                   3         ,      46.9±6ms      46.9±8ms       46.9±6ms    78.1±0ms    70.3±8ms    93.8±8ms\r\n              ============ ====== ============ ============== ============= ========== ============ ==========\r\n\r\n\r\nBENCHMARKS NOT SIGNIFICANTLY CHANGED.\r\n```\r\nCapital letters telling me everything is fine? Not so much unfortunately... (stars added manually)\r\n```\r\n(pandas-dev) C:\\[...]\\pddev\\asv_bench>asv compare b28cf5aa 0d3c6d21\r\n\r\nAll benchmarks:\r\n\r\n       before           after         ratio\r\n     [b28cf5aa]       [0d3c6d21]\r\n     <master>         <cln_str_cat~2>\r\n         9.38±0ms         9.38±0ms       1.00    strings.Cat.time_cat(0, ',', '-', 0.0)\r\n         9.38±0ms         10.9±0ms    **~1.17**  strings.Cat.time_cat(0, ',', '-', 0.001)\r\n         10.9±0ms         12.5±0ms    **~1.14**  strings.Cat.time_cat(0, ',', '-', 0.15)\r\n       9.38±0.6ms         9.38±0ms       1.00    strings.Cat.time_cat(0, ',', None, 0.0)\r\n       14.1±0.8ms         10.9±0ms    **~0.78**  strings.Cat.time_cat(0, ',', None, 0.001)\r\n       14.1±0.6ms       11.7±0.8ms    **~0.83**  strings.Cat.time_cat(0, ',', None, 0.15)\r\n         9.38±0ms         9.38±0ms       1.00    strings.Cat.time_cat(0, None, '-', 0.0)\r\n       9.38±0.6ms       10.9±0.6ms    **~1.17**  strings.Cat.time_cat(0, None, '-', 0.001)\r\n         10.9±0ms         12.5±0ms    **~1.14**  strings.Cat.time_cat(0, None, '-', 0.15)\r\n         10.9±0ms         7.81±2ms    **~0.71**  strings.Cat.time_cat(0, None, None, 0.0)\r\n         9.38±0ms       10.9±0.8ms    **~1.17**  strings.Cat.time_cat(0, None, None, 0.001)\r\n       10.9±0.8ms       11.7±0.8ms       1.07    strings.Cat.time_cat(0, None, None, 0.15)\r\n         78.1±0ms         78.1±8ms       1.00    strings.Cat.time_cat(3, ',', '-', 0.0)\r\n         70.3±8ms         78.1±0ms    **~1.11**  strings.Cat.time_cat(3, ',', '-', 0.001)\r\n         93.8±8ms         93.8±0ms       1.00    strings.Cat.time_cat(3, ',', '-', 0.15)\r\n         46.9±6ms         78.1±0ms    **~1.67**  strings.Cat.time_cat(3, ',', None, 0.0)\r\n         46.9±8ms         78.1±0ms    **~1.67**  strings.Cat.time_cat(3, ',', None, 0.001)\r\n         46.9±6ms         62.5±8ms    **~1.33**  strings.Cat.time_cat(3, ',', None, 0.15)\r\n         54.7±8ms         54.7±8ms       1.00    strings.Cat.time_cat(3, None, '-', 0.0)\r\n         46.9±8ms         62.5±8ms    **~1.33**  strings.Cat.time_cat(3, None, '-', 0.001)\r\n         62.5±0ms        78.1±10ms    **~1.25**  strings.Cat.time_cat(3, None, '-', 0.15)\r\n         62.5±6ms         46.9±8ms    **~0.75**  strings.Cat.time_cat(3, None, None, 0.0)\r\n         46.9±6ms         62.5±0ms    **~1.33**  strings.Cat.time_cat(3, None, None, 0.001)\r\n         46.9±0ms         62.5±6ms    **~1.33**  strings.Cat.time_cat(3, None, None, 0.15)\r\n```\r\n\r\nIt just so happened that *all* the significant changes were hiding behind a tilde...\r\n\r\nPS. I just installed from github a few days ago, I'm on `0.4.dev1533+5f9be296`."},"comment":{"url":"https://api.github.com/repos/airspeed-velocity/asv/issues/comments/439698275","html_url":"https://github.com/airspeed-velocity/asv/issues/752#issuecomment-439698275","issue_url":"https://api.github.com/repos/airspeed-velocity/asv/issues/752","id":439698275,"node_id":"MDEyOklzc3VlQ29tbWVudDQzOTY5ODI3NQ==","user":{"login":"h-vetinari","id":33685575,"node_id":"MDQ6VXNlcjMzNjg1NTc1","avatar_url":"https://avatars1.githubusercontent.com/u/33685575?v=4","gravatar_id":"","url":"https://api.github.com/users/h-vetinari","html_url":"https://github.com/h-vetinari","followers_url":"https://api.github.com/users/h-vetinari/followers","following_url":"https://api.github.com/users/h-vetinari/following{/other_user}","gists_url":"https://api.github.com/users/h-vetinari/gists{/gist_id}","starred_url":"https://api.github.com/users/h-vetinari/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/h-vetinari/subscriptions","organizations_url":"https://api.github.com/users/h-vetinari/orgs","repos_url":"https://api.github.com/users/h-vetinari/repos","events_url":"https://api.github.com/users/h-vetinari/events{/privacy}","received_events_url":"https://api.github.com/users/h-vetinari/received_events","type":"User","site_admin":false},"created_at":"2018-11-18T14:47:18Z","updated_at":"2018-11-18T14:47:18Z","author_association":"NONE","body":"Sorry for the late reply.\r\n\r\n> Because of the noise, you don't know if something increased dramatically: the values printed are only the medians. If there's too much spread in the results to even state with confidence the two measurements gave different results, it's not possible to state there was a change larger than some ratio.\r\n\r\nMy point is that - despite the spread - the increase in the medians might be a real regression, and hiding those cases by default, and without an option to show them gives a false sense of security to the people using ASV. Maybe someone wants to be strict on those noisy cases as well...?\r\n\r\nHow about a parameter that allows to include these tests, either through a boolean, or through setting some threshold for the variance etc.?"}}
{"actor":{"display_login":"h-vetinari"}}