{"action":"created","issue":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/11159","repository_url":"https://api.github.com/repos/pytorch/pytorch","labels_url":"https://api.github.com/repos/pytorch/pytorch/issues/11159/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/pytorch/issues/11159/comments","events_url":"https://api.github.com/repos/pytorch/pytorch/issues/11159/events","html_url":"https://github.com/pytorch/pytorch/pull/11159","id":356119936,"node_id":"MDExOlB1bGxSZXF1ZXN0MjEyNDkxMTUx","number":11159,"title":"[c2][serialized_tests] Refactor tests part 1","user":{"login":"ajyu","id":1071670,"node_id":"MDQ6VXNlcjEwNzE2NzA=","avatar_url":"https://avatars2.githubusercontent.com/u/1071670?v=4","gravatar_id":"","url":"https://api.github.com/users/ajyu","html_url":"https://github.com/ajyu","followers_url":"https://api.github.com/users/ajyu/followers","following_url":"https://api.github.com/users/ajyu/following{/other_user}","gists_url":"https://api.github.com/users/ajyu/gists{/gist_id}","starred_url":"https://api.github.com/users/ajyu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ajyu/subscriptions","organizations_url":"https://api.github.com/users/ajyu/orgs","repos_url":"https://api.github.com/users/ajyu/repos","events_url":"https://api.github.com/users/ajyu/events{/privacy}","received_events_url":"https://api.github.com/users/ajyu/received_events","type":"User","site_admin":false},"labels":[{"id":890282107,"node_id":"MDU6TGFiZWw4OTAyODIxMDc=","url":"https://api.github.com/repos/pytorch/pytorch/labels/caffe2","name":"caffe2","color":"210aa8","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-08-31T20:50:18Z","updated_at":"2018-09-04T20:31:37Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/pytorch/pytorch/pulls/11159","html_url":"https://github.com/pytorch/pytorch/pull/11159","diff_url":"https://github.com/pytorch/pytorch/pull/11159.diff","patch_url":"https://github.com/pytorch/pytorch/pull/11159.patch"},"body":"Followup to [the serialized test framework](https://github.com/pytorch/pytorch/pull/10594)\r\n\r\n\r\nRound 1 for refactoring tests, starting alphabetically. I added some functionality, so I wanted to send out some of these initial changes sooner.\r\n1. Start alphabetically producing serialized outputs for test functions, annotating those we want to include with `@given_and_seeded`. So far I've only added one test per operator, but this already does seem to add quite a few tests.\r\n2. Add functionality to allow us to generate outputs using pytest by adding pytest argument options. This allows us to skip adding a `__main__` function to quite a few tests.\r\n3. Catch any exceptions generating the gradient operator and skip serializing/reading it, since certain operators don't have gradients.\r\n\r\nI also noticed that there is some hypothesis handling that makes `@given_and_seeded` not compatible with adding more hypothesis decorators on top. For example, there are tests that do \r\n```\r\n@settings(...)\r\n@given(...)\r\ndef test_my_stuff(...)\r\n```\r\nBut there is a hypothesis handler that explicitly checks that `@given` is called below `@settings`, so we cannot refactor this to `@given_and_seeded`. I've just avoided decorating these kinds of tests for now, I hope that's alright."},"comment":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/comments/418507839","html_url":"https://github.com/pytorch/pytorch/pull/11159#issuecomment-418507839","issue_url":"https://api.github.com/repos/pytorch/pytorch/issues/11159","id":418507839,"node_id":"MDEyOklzc3VlQ29tbWVudDQxODUwNzgzOQ==","user":{"login":"ajyu","id":1071670,"node_id":"MDQ6VXNlcjEwNzE2NzA=","avatar_url":"https://avatars2.githubusercontent.com/u/1071670?v=4","gravatar_id":"","url":"https://api.github.com/users/ajyu","html_url":"https://github.com/ajyu","followers_url":"https://api.github.com/users/ajyu/followers","following_url":"https://api.github.com/users/ajyu/following{/other_user}","gists_url":"https://api.github.com/users/ajyu/gists{/gist_id}","starred_url":"https://api.github.com/users/ajyu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ajyu/subscriptions","organizations_url":"https://api.github.com/users/ajyu/orgs","repos_url":"https://api.github.com/users/ajyu/repos","events_url":"https://api.github.com/users/ajyu/events{/privacy}","received_events_url":"https://api.github.com/users/ajyu/received_events","type":"User","site_admin":false},"created_at":"2018-09-04T20:31:37Z","updated_at":"2018-09-04T20:31:37Z","author_association":"CONTRIBUTOR","body":"@apaszke, the motivation for these tests mainly comes from a past sev where changes to operator semantics make an operator not backward compatible. There is some more context in this [quip](https://fb.quip.com/tVg1AImdjoKY). While we recognize that implementation differences may cause small differences when compared to an expected test, we think that having such a mechanism will help  us better catch more egregious cases. Backwards incompatible changes are otherwise hard to catch in unit tests since code modifiers would typically change matching tests to pass when making a semantically different change.\r\n\r\nOne possibility is strongly relaxing the tolerance requirements to hopefully prevent failures due to the subtlest implementation updates.\r\n\r\nOther things that we can do that are more ambitious would be to track models run in production and track some serialized inputs matched with either outputs or higher level aggregate statistics which shouldn't differ widely with code change."}}
{"actor":{"display_login":"ajyu"}}