{"action":"created","issue":{"url":"https://api.github.com/repos/e-mission/e-mission-server/issues/439","repository_url":"https://api.github.com/repos/e-mission/e-mission-server","labels_url":"https://api.github.com/repos/e-mission/e-mission-server/issues/439/labels{/name}","comments_url":"https://api.github.com/repos/e-mission/e-mission-server/issues/439/comments","events_url":"https://api.github.com/repos/e-mission/e-mission-server/issues/439/events","html_url":"https://github.com/e-mission/e-mission-server/issues/439","id":184713472,"number":439,"title":"AssertionError while processing _fix_squished_place_mismatch","user":{"login":"shankari","id":2423263,"avatar_url":"https://avatars.githubusercontent.com/u/2423263?v=3","gravatar_id":"","url":"https://api.github.com/users/shankari","html_url":"https://github.com/shankari","followers_url":"https://api.github.com/users/shankari/followers","following_url":"https://api.github.com/users/shankari/following{/other_user}","gists_url":"https://api.github.com/users/shankari/gists{/gist_id}","starred_url":"https://api.github.com/users/shankari/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shankari/subscriptions","organizations_url":"https://api.github.com/users/shankari/orgs","repos_url":"https://api.github.com/users/shankari/repos","events_url":"https://api.github.com/users/shankari/events{/privacy}","received_events_url":"https://api.github.com/users/shankari/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":14,"created_at":"2016-10-23T20:26:26Z","updated_at":"2016-12-04T21:44:50Z","closed_at":null,"body":"We find raw -> filtered\n\n```\n2016-10-23 03:38:13,117:DEBUG:139657654888256:For raw trip 580bf83888f66356dad2d40a, found filtered trip 580c30a488f6635b29ea55be\n```\n\nThis is a trip, not untracked\n\n```\n2016-10-23 03:38:13,121:DEBUG:139657654888256:_is_squished_untracked: 580bb32e88f663518b966b23 is a segmentation/raw_trip, not segmentation/raw_untracked, returning False\n```\n\nThere's a minor difference between the two points (18 meters apart).\n\n> > ecc.calDistance([-122.2588476074135, 37.87331113595317], [-122.2588238866276, 37.87347466688405])\n> > 18.30262099474195\n\n```\n2016-10-23 03:38:13,124:DEBUG:139657654888256:squishing mismatch: resetting trip start_loc [-122.2588476074135, 37.87331113595317] to cleaned_start_place.location [-122.2588238866276, 37.87347466688405]\n```\n\nNote that we attempt to fix the mismatch if the coordinates are not identical\n\n```\n    if (cleaned_start_place_data.location.coordinates !=\n                       raw_start_place.data.location.coordinates):\n        _fix_squished_place_mismatch(cleaned_trip.user_id, cleaned_trip.get_id(),\n                                     ts, cleaned_trip_data, cleaned_start_place_data)\n\n```\n\nWe look for a location at ts = 1477155094.000022. Why? Do we find it?\n\n```\n2016-10-23 03:38:13,175:DEBUG:139657654888256:get_entry_at_ts query = {'metadata.key': 'background/filtered_location', 'user_id': UUID('8f1a6737-e8ee-4769-a1a4-da1948d886b6'), 'data.ts': 1477155094.000022}\n```\n\nUnsure, but we insert an entry at 1477154950.1033349\n\n```\n2016-10-23 03:38:13\n178:DEBUG:139657654888256:Inserting entry Entry({'data': Recreatedlocation(\n{u'loc': {u'type': u'Point' u'coordinates': [-122.2588238866276 37.87347466688405]}\n u'fmt_time': '2016-10-22T09:49:10.103335-07:00'\n u'ts': 1477154950.1033349\n 'mode': 7\n 'section': ObjectId('580c30a388f6635b29ea54ff')\n u'sensed_speed': 3.579999923706055\n 'distance': 0})\n '_id': ObjectId('580c30a588f6635b29ea55dd')\n 'key': 'analysis/recreated_location'})}) into timeseries\n```\n\nThen, we query for the entries for this section (1477154950.103335 -> 1477160591.999968). We find 1906 entries, which are no de-duped. and then the speeds mismatch.\n\n```\n2016-10-23 03:38:13,194:DEBUG:139657654888256:curr_query = {'$or': [{'metadata.key': 'analysis/recreated_location'}], 'user_id': UUID('8f1a6737-e8ee-4769-a1a4-da1948d886b6'), 'data.ts': {'$lte': 1477160591.999968, '$gte': 1477154950.103335}}, sort_key = data.ts\n2016-10-23 03:38:13,194:DEBUG:139657654888256:orig_ts_db_keys = [], analysis_ts_db_keys = ['analysis/recreated_location']\n2016-10-23 03:38:13,194:DEBUG:139657654888256:finished querying values for []\n2016-10-23 03:38:13,194:DEBUG:139657654888256:finished querying values for ['analysis/recreated_location']\n2016-10-23 03:38:14,055:DEBUG:139657654888256:Found 1906 results\n2016-10-23 03:38:14,061:DEBUG:139657654888256:After de-duping, converted 1906 points to 1906 \n2016-10-23 03:38:14,062:DEBUG:139657654888256:Returning entry with length 1906 result\n2016-10-23 03:38:14,589:ERROR:139657654888256:[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] != [0.0, 0.6100873664913984, 0.05965627862873103, 0.059656275796728137, 0.05965627291921605, 0.05965627008539246, 0.059656267209699104, 0.05965626437769427, 0.059656261500180226, 0.05965625864542004]\n2016-10-23 03:38:14,590:ERROR:139657654888256:Cleaning and resampling failed for user 8f1a6737-e8ee-4769-a1a4-da1948d886b6\nTraceback (most recent call last):\n  File \"/mnt/e-mission/e-mission-server/emission/analysis/intake/cleaning/clean_and_resample.py\", line 72, in clean_and_resample\n    last_raw_place = save_cleaned_segments_for_ts(user_id, time_query.startTs, time_query.endTs)\n  File \"/mnt/e-mission/e-mission-server/emission/analysis/intake/cleaning/clean_and_resample.py\", line 90, in save_cleaned_segments_for_ts\n    return save_cleaned_segments_for_timeline(user_id, tl)\n  File \"/mnt/e-mission/e-mission-server/emission/analysis/intake/cleaning/clean_and_resample.py\", line 116, in save_cleaned_segments_for_timeline\n    (last_cleaned_place, filtered_tl) = create_and_link_timeline(tl, user_id, trip_map)\n  File \"/mnt/e-mission/e-mission-server/emission/analysis/intake/cleaning/clean_and_resample.py\", line 878, in create_and_link_timeline\n    link_trip_start(ts, curr_cleaned_trip, curr_cleaned_start_place, raw_start_place)\n  File \"/mnt/e-mission/e-mission-server/emission/analysis/intake/cleaning/clean_and_resample.py\", line 932, in link_trip_start\n    ts, cleaned_trip_data, cleaned_start_place_data)\n  File \"/mnt/e-mission/e-mission-server/emission/analysis/intake/cleaning/clean_and_resample.py\", line 1055, in _fix_squished_place_mismatch\n    assert False\nAssertionError\n```\n"},"comment":{"url":"https://api.github.com/repos/e-mission/e-mission-server/issues/comments/264733921","html_url":"https://github.com/e-mission/e-mission-server/issues/439#issuecomment-264733921","issue_url":"https://api.github.com/repos/e-mission/e-mission-server/issues/439","id":264733921,"user":{"login":"shankari","id":2423263,"avatar_url":"https://avatars.githubusercontent.com/u/2423263?v=3","gravatar_id":"","url":"https://api.github.com/users/shankari","html_url":"https://github.com/shankari","followers_url":"https://api.github.com/users/shankari/followers","following_url":"https://api.github.com/users/shankari/following{/other_user}","gists_url":"https://api.github.com/users/shankari/gists{/gist_id}","starred_url":"https://api.github.com/users/shankari/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shankari/subscriptions","organizations_url":"https://api.github.com/users/shankari/orgs","repos_url":"https://api.github.com/users/shankari/repos","events_url":"https://api.github.com/users/shankari/events{/privacy}","received_events_url":"https://api.github.com/users/shankari/received_events","type":"User","site_admin":false},"created_at":"2016-12-04T21:44:50Z","updated_at":"2016-12-04T21:44:50Z","body":"This has been happening for so long that it is going to be hard to track down the root cause. Resetting the pipeline and re-running again.\r\nWe also need to keep a careful eye on the errors and investigate more quickly next time."}}
{"actor":{"display_login":"shankari"}}