{"action":"opened","issue":{"url":"https://api.github.com/repos/e-mission/e-mission-server/issues/410","repository_url":"https://api.github.com/repos/e-mission/e-mission-server","labels_url":"https://api.github.com/repos/e-mission/e-mission-server/issues/410/labels{/name}","comments_url":"https://api.github.com/repos/e-mission/e-mission-server/issues/410/comments","events_url":"https://api.github.com/repos/e-mission/e-mission-server/issues/410/events","html_url":"https://github.com/e-mission/e-mission-server/issues/410","id":177936509,"number":410,"title":"iOS tests are not robust","user":{"login":"shankari","id":2423263,"avatar_url":"https://avatars.githubusercontent.com/u/2423263?v=3","gravatar_id":"","url":"https://api.github.com/users/shankari","html_url":"https://github.com/shankari","followers_url":"https://api.github.com/users/shankari/followers","following_url":"https://api.github.com/users/shankari/following{/other_user}","gists_url":"https://api.github.com/users/shankari/gists{/gist_id}","starred_url":"https://api.github.com/users/shankari/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shankari/subscriptions","organizations_url":"https://api.github.com/users/shankari/orgs","repos_url":"https://api.github.com/users/shankari/repos","events_url":"https://api.github.com/users/shankari/events{/privacy}","received_events_url":"https://api.github.com/users/shankari/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-09-20T00:57:54Z","updated_at":"2016-09-20T00:57:54Z","closed_at":null,"body":"Ever since we checked in a56adddc5dc8c94cbe98964aafb17df3bc3f724c, and 391a85a058fa21570ee8ad4cb80fadddc9c3e6ce, we've been seeing a lot of volatility in our pull requests, although they do eventually pass.\r\n\r\nSee https://github.com/e-mission/e-mission-server/pull/405, https://github.com/e-mission/e-mission-server/pull/406, https://github.com/e-mission/e-mission-server/pull/409.\r\n\r\nThis happened so frequently that I re-ran locally to see what was going on. And the answer is -- the randomization!\r\n\r\nRecall that in a56adddc5dc8c94cbe98964aafb17df3bc3f724c, we added some noise into the smoothing data on iOS. That noise causes subtle changes in the max speed, which causes subtle changes in the segmentation, which causes subtle changes in the total distance.\r\n\r\nThe solution is to either rollback the change or use a fixed random seed. Fixed random seed sounds better for now.\r\n\r\n\r\n## So far, so identical. ##\r\n\r\n### Working instance ###\r\n\r\n```\r\n2016-09-19 16:25:49,577:INFO:140735159468032:^^^^^^^^^^^^^^^^^^^^Smoothing section 57e073fdf6858f49f941bfeb for user 4d9e4e89-809b-4b85-bad4-c2d7ac215a13^^^^^^^^^^^^^^^^^^^^\r\n2016-09-19 16:25:49,577:DEBUG:140735159468032:filter_jumps(4d9e4e89-809b-4b85-bad4-c2d7ac215a13, 57e073fdf6858f49f941bfeb) called\r\n2016-09-19 16:25:49,577:DEBUG:140735159468032:curr_query = {'$or': [{'metadata.key': 'background/filtered_location'}], 'user_id': UUID('4d9e4e89-809b-4b85-bad4-c2d7ac215a13'), 'data.ts': {'$lte': 1469068274.999965, '$gte': 1469065859.779821}}, sort_key = data.ts\r\n2016-09-19 16:25:49,577:DEBUG:140735159468032:orig_ts_db_keys = ['background/filtered_location'], analysis_ts_db_keys = []\r\n2016-09-19 16:25:49,578:DEBUG:140735159468032:finished querying values for ['background/filtered_location']\r\n2016-09-19 16:25:49,578:DEBUG:140735159468032:finished querying values for []\r\n2016-09-19 16:25:49,587:DEBUG:140735159468032:Found 45 results\r\n2016-09-19 16:25:49,589:DEBUG:140735159468032:After de-duping, converted 45 points to 45 \r\n2016-09-19 16:25:49,590:DEBUG:140735159468032:Found iOS section, filling in gaps with fake data\r\n2016-09-19 16:25:49,590:DEBUG:140735159468032:Found large gap ending at 28, filling it\r\n2016-09-19 16:25:49,591:DEBUG:140735159468032:start = 27, end = 28, generating entries between 1469066924.0 and 1469066990.0\r\n2016-09-19 16:25:49,591:DEBUG:140735159468032:Fill lengths are: dist 1, angle 1, lat 1, lng 1\r\n```\r\n\r\n### Not working instance ###\r\n\r\n```\r\n2016-09-19 16:26:06,309:INFO:140735159468032:^^^^^^^^^^^^^^^^^^^^Smoothing section 57e0740df6858f4a1bfbfefd for user 858af1b4-d0b5-4b85-b3f3-986104c71b06^^^^^^^^^^^^^^^^^^^^\r\n2016-09-19 16:26:06,309:DEBUG:140735159468032:filter_jumps(858af1b4-d0b5-4b85-b3f3-986104c71b06, 57e0740df6858f4a1bfbfefd) called\r\n2016-09-19 16:26:06,310:DEBUG:140735159468032:curr_query = {'$or': [{'metadata.key': 'background/filtered_location'}], 'user_id': UUID('858af1b4-d0b5-4b85-b3f3-986104c71b06'), 'data.ts': {'$lte': 1469068274.999965, '$gte': 1469065859.779821}}, sort_key = data.ts\r\n2016-09-19 16:26:06,310:DEBUG:140735159468032:orig_ts_db_keys = ['background/filtered_location'], analysis_ts_db_keys = []\r\n2016-09-19 16:26:06,311:DEBUG:140735159468032:finished querying values for ['background/filtered_location']\r\n2016-09-19 16:26:06,311:DEBUG:140735159468032:finished querying values for []\r\n2016-09-19 16:26:06,321:DEBUG:140735159468032:Found 45 results\r\n2016-09-19 16:26:06,324:DEBUG:140735159468032:After de-duping, converted 45 points to 45 \r\n2016-09-19 16:26:06,324:DEBUG:140735159468032:Found iOS section, filling in gaps with fake data\r\n2016-09-19 16:26:06,325:DEBUG:140735159468032:Found large gap ending at 28, filling it\r\n2016-09-19 16:26:06,326:DEBUG:140735159468032:start = 27, end = 28, generating entries between 1469066924.0 and 1469066990.0\r\n2016-09-19 16:26:06,327:DEBUG:140735159468032:Fill lengths are: dist 1, angle 1, lat 1, lng 1\r\n```\r\n\r\n## Difference in quartile values ##\r\n\r\n### Working instance ###\r\n\r\nSo after filling in, the number of points is the same but the quartile values,\r\nand thus, the IQR, are subtly different.\r\n\r\n```\r\n2016-09-19 16:25:49,609:DEBUG:140735159468032:len(section_points_df) = 46\r\n2016-09-19 16:25:49,632:DEBUG:140735159468032:section_points_df.shape = (46, 24), with_speeds_df.shape = (46, 27)\r\n2016-09-19 16:25:49,652:DEBUG:140735159468032:quartile values are 0.25    1.173621\r\n0.75    1.348601\r\nName: speed, dtype: float64\r\n2016-09-19 16:25:49,652:DEBUG:140735159468032:iqr 0.174979405349\r\n2016-09-19 16:25:49,652:DEBUG:140735159468032:maxSpeed = 1.87353876802\r\n```\r\n\r\n### Not working instance ###\r\n\r\n```\r\n2016-09-19 16:26:06,353:DEBUG:140735159468032:len(section_points_df) = 46\r\n2016-09-19 16:26:06,376:DEBUG:140735159468032:section_points_df.shape = (46, 24), with_speeds_df.shape = (46, 27)\r\n2016-09-19 16:26:06,404:DEBUG:140735159468032:quartile values are 0.25    1.173621\r\n0.75    1.353456\r\nName: speed, dtype: float64\r\n2016-09-19 16:26:06,404:DEBUG:140735159468032:iqr 0.179834642608\r\n2016-09-19 16:26:06,404:DEBUG:140735159468032:maxSpeed = 1.89295971706\r\n```\r\n\r\n## Splits are different ##\r\n\r\nAnd so the splits are different. And then its pretty obvious why the values are\r\ndifferent.\r\n\r\n```\r\n2016-09-19 16:25:49,655:DEBUG:140735159468032:For cluster 0 - 2, distance = 54.0145036842, is_cluster = True\r\n2016-09-19 16:25:49,656:DEBUG:140735159468032:For cluster 2 - 29, distance = 796.349531704, is_cluster = False\r\n2016-09-19 16:25:49,657:DEBUG:140735159468032:For cluster 29 - 46, distance = 605.501923938, is_cluster = False\r\n```\r\n\r\n```\r\n2016-09-19 16:26:06,408:DEBUG:140735159468032:For cluster 0 - 2, distance = 54.0145036842, is_cluster = True\r\n2016-09-19 16:26:06,409:DEBUG:140735159468032:For cluster 2 - 28, distance = 754.868801101, is_cluster = False\r\n2016-09-19 16:26:06,410:DEBUG:140735159468032:For cluster 28 - 29, distance = 0.0, is_cluster = True\r\n2016-09-19 16:26:06,410:DEBUG:140735159468032:For cluster 29 - 46, distance = 605.501923938, is_cluster = False\r\n```\r\n"}}
{"actor":{"display_login":"shankari"}}