{"action":"created","issue":{"url":"https://api.github.com/repos/scivision/georinex/issues/23","repository_url":"https://api.github.com/repos/scivision/georinex","labels_url":"https://api.github.com/repos/scivision/georinex/issues/23/labels{/name}","comments_url":"https://api.github.com/repos/scivision/georinex/issues/23/comments","events_url":"https://api.github.com/repos/scivision/georinex/issues/23/events","html_url":"https://github.com/scivision/georinex/issues/23","id":347975808,"node_id":"MDU6SXNzdWUzNDc5NzU4MDg=","number":23,"title":"Proposal: Speed up data storing processing in xarray","user":{"login":"VOMAY","id":32896877,"node_id":"MDQ6VXNlcjMyODk2ODc3","avatar_url":"https://avatars2.githubusercontent.com/u/32896877?v=4","gravatar_id":"","url":"https://api.github.com/users/VOMAY","html_url":"https://github.com/VOMAY","followers_url":"https://api.github.com/users/VOMAY/followers","following_url":"https://api.github.com/users/VOMAY/following{/other_user}","gists_url":"https://api.github.com/users/VOMAY/gists{/gist_id}","starred_url":"https://api.github.com/users/VOMAY/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/VOMAY/subscriptions","organizations_url":"https://api.github.com/users/VOMAY/orgs","repos_url":"https://api.github.com/users/VOMAY/repos","events_url":"https://api.github.com/users/VOMAY/events{/privacy}","received_events_url":"https://api.github.com/users/VOMAY/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-08-06T15:37:41Z","updated_at":"2018-08-10T04:29:22Z","closed_at":null,"author_association":"NONE","body":"Dear scivision,\r\n\r\nI ran a profiler on georinex (why did you change the name), while reading a 24h@30s Multi-GNSS Rinex 3 (LEICA GR25) file and found that the xarray.merge is responsible for >90% of the time consumption. This is not a surprise, but annoying nonetheless. I checked the code and saw that you are using xarray.concat instead of xarray.merge for single constellation cases. So I made a test:\r\n\r\n``` python\r\n#Galileo only: 57.44 sec\r\ne=gr.readrinex('C:/Users/Volker Mayer/Desktop/ABMFxxGLP_R_20181330000_01D_30S_MO.rnx', use=['E'])\r\n#GPS only: 59.79 sec\r\ng=gr.readrinex('C:/Users/Volker Mayer/Desktop/ABMFxxGLP_R_20181330000_01D_30S_MO.rnx', use=['G'])\r\n#GPS+Galileo: 2.89 min\r\neg=gr.readrinex('C:/Users/Volker Mayer/Desktop/ABMFxxGLP_R_20181330000_01D_30S_MO.rnx', use=['E', 'G'])\r\n```\r\nAs you can see, reading GPS+Galileo at once takes about 50% longer than doing them one after another. This shows that concat is substantially quicker than merge. So here is my proposal:\r\n\r\nCreate one xarray dataset for each constellation and merge them **only once** at the very end. I am optimistic that this will allow at least 50% of time savings. Probably even quicker would be to use one xarray dataset per satellite and again merge them only once!\r\n\r\nBest Regards\r\nVolker"},"comment":{"url":"https://api.github.com/repos/scivision/georinex/issues/comments/411971908","html_url":"https://github.com/scivision/georinex/issues/23#issuecomment-411971908","issue_url":"https://api.github.com/repos/scivision/georinex/issues/23","id":411971908,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMTk3MTkwOA==","user":{"login":"scivision","id":10931741,"node_id":"MDQ6VXNlcjEwOTMxNzQx","avatar_url":"https://avatars2.githubusercontent.com/u/10931741?v=4","gravatar_id":"","url":"https://api.github.com/users/scivision","html_url":"https://github.com/scivision","followers_url":"https://api.github.com/users/scivision/followers","following_url":"https://api.github.com/users/scivision/following{/other_user}","gists_url":"https://api.github.com/users/scivision/gists{/gist_id}","starred_url":"https://api.github.com/users/scivision/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scivision/subscriptions","organizations_url":"https://api.github.com/users/scivision/orgs","repos_url":"https://api.github.com/users/scivision/repos","events_url":"https://api.github.com/users/scivision/events{/privacy}","received_events_url":"https://api.github.com/users/scivision/received_events","type":"User","site_admin":false},"created_at":"2018-08-10T04:29:22Z","updated_at":"2018-08-10T04:29:22Z","author_association":"OWNER","body":"Do you have a public example of a large RINEX file you'd like to compare on?\r\nI think from `TimeRinex.py` that it can be fast to:\r\n1. quick read RINEX 3 OBS files for SV & time only\r\n2. allocate the empty `xarrayDataset`, sizing by time and SV read in step 1\r\n3. populate the array with data as the file is fully read\r\nThis could be even faster and significantly cleaner/simpler to implement than the linked lists.\r\n\r\nBy one-time I actually meant off-line, that is, convert say 10,000 RINEX files before the human actually works with them. "}}
{"actor":{"display_login":"scivision"}}