{"action":"created","issue":{"url":"https://api.github.com/repos/gammapy/gammapy/issues/1819","repository_url":"https://api.github.com/repos/gammapy/gammapy","labels_url":"https://api.github.com/repos/gammapy/gammapy/issues/1819/labels{/name}","comments_url":"https://api.github.com/repos/gammapy/gammapy/issues/1819/comments","events_url":"https://api.github.com/repos/gammapy/gammapy/issues/1819/events","html_url":"https://github.com/gammapy/gammapy/issues/1819","id":362583615,"node_id":"MDU6SXNzdWUzNjI1ODM2MTU=","number":1819,"title":"DataStore from file","user":{"login":"cdeil","id":852409,"node_id":"MDQ6VXNlcjg1MjQwOQ==","avatar_url":"https://avatars3.githubusercontent.com/u/852409?v=4","gravatar_id":"","url":"https://api.github.com/users/cdeil","html_url":"https://github.com/cdeil","followers_url":"https://api.github.com/users/cdeil/followers","following_url":"https://api.github.com/users/cdeil/following{/other_user}","gists_url":"https://api.github.com/users/cdeil/gists{/gist_id}","starred_url":"https://api.github.com/users/cdeil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cdeil/subscriptions","organizations_url":"https://api.github.com/users/cdeil/orgs","repos_url":"https://api.github.com/users/cdeil/repos","events_url":"https://api.github.com/users/cdeil/events{/privacy}","received_events_url":"https://api.github.com/users/cdeil/received_events","type":"User","site_admin":false},"labels":[{"id":39740769,"node_id":"MDU6TGFiZWwzOTc0MDc2OQ==","url":"https://api.github.com/repos/gammapy/gammapy/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/gammapy/gammapy/milestones/11","html_url":"https://github.com/gammapy/gammapy/milestone/11","labels_url":"https://api.github.com/repos/gammapy/gammapy/milestones/11/labels","id":2487784,"node_id":"MDk6TWlsZXN0b25lMjQ4Nzc4NA==","number":11,"title":"0.8","description":"","creator":{"login":"cdeil","id":852409,"node_id":"MDQ6VXNlcjg1MjQwOQ==","avatar_url":"https://avatars3.githubusercontent.com/u/852409?v=4","gravatar_id":"","url":"https://api.github.com/users/cdeil","html_url":"https://github.com/cdeil","followers_url":"https://api.github.com/users/cdeil/followers","following_url":"https://api.github.com/users/cdeil/following{/other_user}","gists_url":"https://api.github.com/users/cdeil/gists{/gist_id}","starred_url":"https://api.github.com/users/cdeil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cdeil/subscriptions","organizations_url":"https://api.github.com/users/cdeil/orgs","repos_url":"https://api.github.com/users/cdeil/repos","events_url":"https://api.github.com/users/cdeil/events{/privacy}","received_events_url":"https://api.github.com/users/cdeil/received_events","type":"User","site_admin":false},"open_issues":12,"closed_issues":496,"state":"open","created_at":"2017-04-28T09:17:03Z","updated_at":"2018-09-21T11:58:03Z","due_on":null,"closed_at":null},"comments":0,"created_at":"2018-09-21T11:58:03Z","updated_at":"2018-09-21T12:12:31Z","closed_at":null,"author_association":"MEMBER","body":"@lmohrmann @adonath - I have now produced an example for the case where the index files are in one FITS file: https://github.com/gammapy/gammapy-extra/tree/master/datasets/hess-dl3-dr1\r\n\r\nThere was discussion in  #1793 already, but the `from_dir` doesn't work well for that case, and also it has a bug - `Path.exists` is called as a static method, passing a string as `self`. The `make_path` utils function in Gammapy should be used to resolve env vars and create path objects instead. Also since no default HDU names are tried, I don't even think it's possible to load the `$GAMMAPY_DATA/hess-dl3-dr1/hess-dl3-dr3-with-background.fits.gz` via a DataStore method in a simple way?\r\n\r\nOverall my recommendation would be (which I think is also what I suggested in #1793):\r\n\r\n- Simplify `from_dir` again. The dir passed should be the dir to look for index files. The base_dir is something that's usually set in the HDU index file header (see spec) and can be completely different. Only in the case where it's not set we could default to the dir given for the index file location. I think that's the previous behaviour.\r\n\r\n- Add a separate `DataStore.from_file` for the use case of having both index files in one FITS file, don't try to shoehorn that into `from_dir`. A `read` method could be used that checks if it's a file or a dir and then either calls `from_dir` or `from_file`. By looking for the default HDU names, the warnings from Table.read would not appear for users when everything is OK.\r\n\r\nOf course, @lmohrmann and @adonath - If you don't want that, there are other ways to do it. But at the very least the `Path.exists(some string)` issue should be fixed and covered by a test.\r\n\r\nIt's a bit time-critical since @adonath  will do a tutorial at the CTA meeting on Monday and this needs to be ready. So either one of you does this today or tomorrow latest, or I could do it tonight. Let me know.\r\n\r\n```\r\nIn [3]: from gammapy.data import DataStore\r\n   ...: data_store = DataStore.from_dir(\r\n   ...:     base_dir=\"\",\r\n   ...:     hdu_table_filename=\"$GAMMAPY_DATA/hess-dl3-dr1/hess-dl3-dr3-with-background.fits.gz\",\r\n   ...:     obs_table_filename=\"$GAMMAPY_DATA/hess-dl3-dr1/hess-dl3-dr3-with-background.fits.gz\",\r\n   ...: )\r\n   ...: \r\n   ...: \r\n---------------------------------------------------------------------------\r\nAttributeError                            Traceback (most recent call last)\r\n<ipython-input-3-efedae7e2a01> in <module>()\r\n      3     base_dir=\"\",\r\n      4     hdu_table_filename=\"$GAMMAPY_DATA/hess-dl3-dr1/hess-dl3-dr3-with-background.fits.gz\",\r\n----> 5     obs_table_filename=\"$GAMMAPY_DATA/hess-dl3-dr1/hess-dl3-dr3-with-background.fits.gz\",\r\n      6 )\r\n\r\n~/work/code/gammapy-docs/build/dev/gammapy/gammapy/data/data_store.py in from_dir(cls, base_dir, hdu_table_filename, obs_table_filename)\r\n     83             obs_table_filename = base_dir / cls.DEFAULT_OBS_TABLE\r\n     84 \r\n---> 85         if not Path.exists(hdu_table_filename):\r\n     86             raise IOError(\"File not found: {}\".format(hdu_table_filename))\r\n     87         log.debug(\"Reading {}\".format(hdu_table_filename))\r\n\r\n~/work/code/gammapy-docs/build/dev/gammapy/gammapy/extern/pathlib.py in exists(self)\r\n   1445         \"\"\"\r\n   1446         try:\r\n-> 1447             self.stat()\r\n   1448         except OSError as e:\r\n   1449             if e.errno not in (ENOENT, ENOTDIR):\r\n\r\nAttributeError: 'str' object has no attribute 'stat'\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/gammapy/gammapy/issues/comments/423510683","html_url":"https://github.com/gammapy/gammapy/issues/1819#issuecomment-423510683","issue_url":"https://api.github.com/repos/gammapy/gammapy/issues/1819","id":423510683,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMzUxMDY4Mw==","user":{"login":"lmohrmann","id":15608734,"node_id":"MDQ6VXNlcjE1NjA4NzM0","avatar_url":"https://avatars3.githubusercontent.com/u/15608734?v=4","gravatar_id":"","url":"https://api.github.com/users/lmohrmann","html_url":"https://github.com/lmohrmann","followers_url":"https://api.github.com/users/lmohrmann/followers","following_url":"https://api.github.com/users/lmohrmann/following{/other_user}","gists_url":"https://api.github.com/users/lmohrmann/gists{/gist_id}","starred_url":"https://api.github.com/users/lmohrmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lmohrmann/subscriptions","organizations_url":"https://api.github.com/users/lmohrmann/orgs","repos_url":"https://api.github.com/users/lmohrmann/repos","events_url":"https://api.github.com/users/lmohrmann/events{/privacy}","received_events_url":"https://api.github.com/users/lmohrmann/received_events","type":"User","site_admin":false},"created_at":"2018-09-21T12:12:31Z","updated_at":"2018-09-21T12:12:31Z","author_association":"CONTRIBUTOR","body":"> Also since no default HDU names are tried, I don't even think it's possible to load the $GAMMAPY_DATA/hess-dl3-dr1/hess-dl3-dr3-with-background.fits.gz via a DataStore method in a simple way?\r\n\r\nThe default file names _are_ being tried, so that should work.\r\n\r\n> The dir passed should be the dir to look for index files.\r\n\r\nAgreed!\r\n\r\n> I think that's the previous behaviour.\r\n\r\nNo, it's not, which was the main reason for #1793.\r\n\r\n> it has a bug - `Path.exists` is called as a static method, passing a string as `self`.\r\n\r\nUps, sorry, I didn't know this must be called with a `Path` as argument. I can fix this but don't have time to implement anything else.\r\n\r\nLet me know whether I should make a PR to fix the bug or whether you want to do something more fancy. In any case, I'd appreciate if the option to specify non-standard filenames for the obs and hdu files in the `base_dir` remains."}}
{"actor":{"display_login":"lmohrmann"}}