{"action":"created","issue":{"url":"https://api.github.com/repos/NSLS-II/suitcase/issues/9","repository_url":"https://api.github.com/repos/NSLS-II/suitcase","labels_url":"https://api.github.com/repos/NSLS-II/suitcase/issues/9/labels{/name}","comments_url":"https://api.github.com/repos/NSLS-II/suitcase/issues/9/comments","events_url":"https://api.github.com/repos/NSLS-II/suitcase/issues/9/events","html_url":"https://github.com/NSLS-II/suitcase/issues/9","id":144080626,"number":9,"title":"Suitcase export error at CHX","user":{"login":"ericdill","id":6223243,"avatar_url":"https://avatars.githubusercontent.com/u/6223243?v=3","gravatar_id":"","url":"https://api.github.com/users/ericdill","html_url":"https://github.com/ericdill","followers_url":"https://api.github.com/users/ericdill/followers","following_url":"https://api.github.com/users/ericdill/following{/other_user}","gists_url":"https://api.github.com/users/ericdill/gists{/gist_id}","starred_url":"https://api.github.com/users/ericdill/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ericdill/subscriptions","organizations_url":"https://api.github.com/users/ericdill/orgs","repos_url":"https://api.github.com/users/ericdill/repos","events_url":"https://api.github.com/users/ericdill/events{/privacy}","received_events_url":"https://api.github.com/users/ericdill/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":46,"created_at":"2016-03-28T20:53:59Z","updated_at":"2016-03-29T22:16:40Z","closed_at":null,"body":"@ordirules reported this export error to me via email:\r\n\r\n```\r\n---------------------------------------------------------------------------\r\nKeyError                                  Traceback (most recent call last)\r\n<ipython-input-4-2f4252d32139> in <module>()\r\n----> 1 suitcase.export(cfndb,\"/home/lhermitte/test.hd5\")\r\n\r\n/opt/conda_envs/analysis/lib/python3.4/site-packages/suitcase-0.2.2-py3.4.egg/suitcase.py in export(headers, filename)\r\n     46                 data_group = desc_group.create_group('data')\r\n     47                 ts_group = desc_group.create_group('timestamps')\r\n---> 48                 [fill_event(e) for e in events]\r\n     49                 for key, value in data_keys.items():\r\n     50                     value = dict(value)\r\n\r\n/opt/conda_envs/analysis/lib/python3.4/site-packages/suitcase-0.2.2-py3.4.egg/suitcase.py in <listcomp>(.0)\r\n     46                 data_group = desc_group.create_group('data')\r\n     47                 ts_group = desc_group.create_group('timestamps')\r\n---> 48                 [fill_event(e) for e in events]\r\n     49                 for key, value in data_keys.items():\r\n     50                     value = dict(value)\r\n\r\n/opt/conda_envs/analysis/lib/python3.4/site-packages/databroker/databroker.py in fill_event(event, handler_registry, handler_overrides)\r\n    280         if is_external.get(data_key, False):\r\n    281             if data_key not in handler_overrides:\r\n--> 282                 event.data[data_key] = fs.retrieve(value, handler_registry)\r\n    283             else:\r\n    284                 mock_registry = mock_registries[data_key]\r\n\r\n/opt/conda_envs/analysis/lib/python3.4/site-packages/filestore/api.py in get_data(eid, handle_registry)\r\n    154         handle_registry = {}\r\n    155     with _FS_SINGLETON.handler_context(handle_registry) as fs:\r\n--> 156         return fs.get_datum(eid)\r\n    157\r\n    158\r\n\r\n/opt/conda_envs/analysis/lib/python3.4/site-packages/filestore/fs.py in get_datum(self, eid)\r\n    113         return _get_datum(self._datum_col, eid,\r\n    114                           self._datum_cache, self.get_spec_handler,\r\n--> 115                           logger)\r\n    116\r\n    117     def register_handler(self, key, handler, overwrite=False):\r\n\r\n/opt/conda_envs/analysis/lib/python3.4/site-packages/filestore/core.py in get_datum(col, eid, datum_cache, get_spec_handler, logger)\r\n     36                         \"datum cache can hold.\")\r\n     37\r\n---> 38     handler = get_spec_handler(datum['resource'])\r\n     39     return handler(**datum['datum_kwargs'])\r\n     40\r\n\r\n/opt/conda_envs/analysis/lib/python3.4/site-packages/filestore/fs.py in get_spec_handler(self, resource)\r\n    205\r\n    206         spec = resource['spec']\r\n--> 207         handler = self.handler_reg[spec]\r\n    208         key = (str(resource['_id']), handler.__name__)\r\n    209\r\n\r\n/opt/conda_envs/analysis/lib/python3.4/collections/__init__.py in __getitem__(self, key)\r\n    803             except KeyError:\r\n    804                 pass\r\n--> 805         return self.__missing__(key)            # support subclasses that define __missing__\r\n    806\r\n    807     def get(self, key, default=None):\r\n\r\n/opt/conda_envs/analysis/lib/python3.4/collections/__init__.py in __missing__(self, key)\r\n    795\r\n    796     def __missing__(self, key):\r\n--> 797         raise KeyError(key)\r\n    798\r\n    799     def __getitem__(self, key):\r\n\r\nKeyError: 'AD_EIGER'\r\n```"},"comment":{"url":"https://api.github.com/repos/NSLS-II/suitcase/issues/comments/203135944","html_url":"https://github.com/NSLS-II/suitcase/issues/9#issuecomment-203135944","issue_url":"https://api.github.com/repos/NSLS-II/suitcase/issues/9","id":203135944,"user":{"login":"ordirules","id":705366,"avatar_url":"https://avatars.githubusercontent.com/u/705366?v=3","gravatar_id":"","url":"https://api.github.com/users/ordirules","html_url":"https://github.com/ordirules","followers_url":"https://api.github.com/users/ordirules/followers","following_url":"https://api.github.com/users/ordirules/following{/other_user}","gists_url":"https://api.github.com/users/ordirules/gists{/gist_id}","starred_url":"https://api.github.com/users/ordirules/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ordirules/subscriptions","organizations_url":"https://api.github.com/users/ordirules/orgs","repos_url":"https://api.github.com/users/ordirules/repos","events_url":"https://api.github.com/users/ordirules/events{/privacy}","received_events_url":"https://api.github.com/users/ordirules/received_events","type":"User","site_admin":false},"created_at":"2016-03-29T22:16:40Z","updated_at":"2016-03-29T22:16:40Z","body":"ok great. I just wanted to give feedback to try to help somewhat with meeting users' needs but sounds like you guys have already considered all this and are working on it. :-)\r\n\r\nAbout the file downloading. I can explain. Currently, what I'm doing is extracting the metadata and then rsync'ing the folders with the data on CHX we've taken onto our local servers. When I read the files, I locally set a parent directory in my routines and extract the relative paths from the filenames saved.\r\nHowever, when we're taking data in real time, what I will sometimes do is create a mount point with sshfs directly into CHX's file structure. Running our code from my laptop/workstation/athome or wherever is then a matter of changing the parent directory. What we're doing requires a bit of tweaking and playing around sometimes so simply using existing notebooks on the nsls2 server might not be enough.\r\n\r\nMy case might be a little extreme and maybe it's not as common. However, I think it'd be nice for databroker to support it by allowing users to select whether or not to download the large files (like detector files) with the metadata or not. If they opt out, then they're responsible for retrieving these larger files from the beamline. \r\n\r\nThere is just one issue I'm worried about from my point of view. I'm a little bit worried about the idea of dumping all data into the same hdf5 file, even if it ends up saving space.\r\nThe way I see it is that as a user coming to a beamline, I expect to extract some data that I need. It could be a processed result, or raw files. With this data, there's also metadata (time, location, sample name etc). I think both these quantities should be kept separate and not be abstracted into the same grab bag of data.\r\n\r\nAnyway, that's my point of view, but I'm honestly flexible. You've heard my comment and I trust your actions. I definitely like the overall structure of storing metadata in some general database. It helps reduce confusion. :)\r\n\r\nthanks for the info :-)"}}