{"action":"closed","issue":{"url":"https://api.github.com/repos/dials/dials/issues/199","repository_url":"https://api.github.com/repos/dials/dials","labels_url":"https://api.github.com/repos/dials/dials/issues/199/labels{/name}","comments_url":"https://api.github.com/repos/dials/dials/issues/199/comments","events_url":"https://api.github.com/repos/dials/dials/issues/199/events","html_url":"https://github.com/dials/dials/issues/199","id":175925129,"number":199,"title":"Eiger imageset slicing","user":{"login":"phyy-nx","id":13471434,"avatar_url":"https://avatars.githubusercontent.com/u/13471434?v=3","gravatar_id":"","url":"https://api.github.com/users/phyy-nx","html_url":"https://github.com/phyy-nx","followers_url":"https://api.github.com/users/phyy-nx/followers","following_url":"https://api.github.com/users/phyy-nx/following{/other_user}","gists_url":"https://api.github.com/users/phyy-nx/gists{/gist_id}","starred_url":"https://api.github.com/users/phyy-nx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/phyy-nx/subscriptions","organizations_url":"https://api.github.com/users/phyy-nx/orgs","repos_url":"https://api.github.com/users/phyy-nx/repos","events_url":"https://api.github.com/users/phyy-nx/events{/privacy}","received_events_url":"https://api.github.com/users/phyy-nx/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/dials/dials/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/dials/dials/labels/dxtbx","name":"dxtbx","color":"0052cc"}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-09-09T04:21:20Z","updated_at":"2016-10-26T10:48:19Z","closed_at":"2016-10-26T10:48:19Z","body":"# Background\r\nThe bugs listed below are preventing users from running dials.combine_experiments on the experiment.json files created from processing Eiger data as stills using dials.stills_process.  Therefore, these users cannot refine metrology using joint refinement.\r\n\r\nThese tests were performed at dials.lbl.gov using the eiger16MNov2015 nearly nexus dataset.\r\n\r\n# Problem 1: Slice a datablock.json's imageset\r\n$ cd /net/dials/raid1/aaron/dectris/eiger16MNov2015/2015_11_10/test\r\n$ dials.import ../insu6_1_master.h5\r\n$ cat show_bug_1.py\r\n```\r\nfrom dxtbx.datablock import DataBlockFactory\r\nfrom dxtbx.imageset import ImageSet\r\nimport sys\r\ndatablocks = DataBlockFactory.from_json_file(sys.argv[1], check_format=False)\r\nimageset = datablocks[0].extract_imagesets()[0]\r\nprint imageset._indices\r\nsubset = imageset[100:101]\r\nprint subset._indices\r\n```\r\n\r\n$ libtbx.python show_bug_1.py datablock.json\r\n[0]\r\n[0]\r\n\r\nExpected result: a list of available images numbers in the imageset\r\n\r\n# Problem 2: slice an imageset created by DataBlockFactory.from_filenames\r\n$ cd /net/dials/raid1/aaron/dectris/eiger16MNov2015/2015_11_10/test\r\n$ cat show_bug_2.py\r\n```\r\nfrom dxtbx.datablock import DataBlockFactory\r\nfrom dxtbx.imageset import ImageSet\r\nimport sys\r\ndatablocks = DataBlockFactory.from_filenames([sys.argv[1]])\r\nimageset = datablocks[0].extract_imagesets()[0]\r\nprint imageset._indices\r\nsubset = imageset[100:101]\r\nprint subset._indices\r\n\r\nsliced_datablock = DataBlockFactory.from_imageset(subset)[0]\r\n\r\nfrom dxtbx.datablock import DataBlockDumper\r\ndump = DataBlockDumper(sliced_datablock)\r\ndump.as_json(\"sliced_datablock.json\")\r\n\r\nroundtrip_datablock = DataBlockFactory.from_json_file(\"sliced_datablock.json\")[0]\r\nprint roundtrip_datablock.extract_imagesets()[0]._indices\r\ndump = DataBlockDumper(roundtrip_datablock)\r\ndump.as_json(\"roundtrip_datablock.json\")\r\n```\r\n$ libtbx.python show_bug_2.py ../insu6_1_master.h5\r\n[0, \\<snip\\>, 899]\r\n[100]\r\n[0]\r\n\r\nExpected result:\r\n[0, \\<snip\\>, 899]\r\n[100]\r\n[100]\r\n\r\n$ diff sliced_datablock.json roundtrip_datablock.json\r\nFiles are identical, as expected\r\n\r\n# Problem 3: remove scan and gonio from datablock\r\n$ cd /net/dials/raid1/aaron/dectris/eiger16MNov2015/2015_11_10/test\r\n$ cat show_bug_3.py\r\n```\r\nfrom dxtbx.datablock import DataBlockFactory\r\nfrom dxtbx.imageset import ImageSet\r\nimport sys\r\ndatablocks = DataBlockFactory.from_filenames([sys.argv[1]])\r\nimageset = datablocks[0].extract_imagesets()[0]\r\nprint imageset._indices\r\nsubset = imageset[100:101]\r\nprint subset._indices\r\nsubset = ImageSet(subset.reader(), subset.indices())\r\nsubset.set_scan(None)\r\nsubset.set_goniometer(None)\r\nprint subset._indices\r\n\r\nsliced_datablock = DataBlockFactory.from_imageset(subset)[0]\r\n\r\nfrom dxtbx.datablock import DataBlockDumper\r\ndump = DataBlockDumper(sliced_datablock)\r\ndump.as_json(\"sliced_datablock.json\")\r\n\r\nroundtrip_datablock = DataBlockFactory.from_json_file(\"sliced_datablock.json\")[0]\r\nprint roundtrip_datablock.extract_imagesets()[0]._indices\r\ndump = DataBlockDumper(roundtrip_datablock)\r\ndump.as_json(\"roundtrip_datablock.json\")\r\n```\r\n$ libtbx.python show_bug_3.py ../insu6_1_master.h5\r\n[0, \\<snip\\>, 899]\r\n[100]\r\n[100]\r\n[0, \\<snip\\>, 899]\r\n\r\nExpected result:\r\n[0, \\<snip\\>, 899]\r\n[100]\r\n[100]\r\n[100]\r\n\r\n$ diff sliced_datablock.json roundtrip_datablock.json\r\nFiles are very different. Expected result: files are identical\r\n\r\n# Problem 4: sliced experiment list\r\n$ cd /net/dials/raid1/aaron/dectris/eiger16MNov2015/2015_11_10/test\r\n$ cat show_bug_4.py\r\n```\r\nfrom dxtbx.datablock import DataBlockFactory\r\nfrom dxtbx.model.experiment.experiment_list import ExperimentListFactory\r\nfrom dxtbx.imageset import ImageSet\r\n\r\nimport sys\r\ndatablocks = DataBlockFactory.from_filenames([sys.argv[1]])\r\nimageset = datablocks[0].extract_imagesets()[0]\r\nprint imageset._indices\r\nsubset = imageset[100:101]\r\nprint subset._indices\r\n\r\nfrom dxtbx.model.crystal import crystal_model\r\ncrystal = crystal_model((1,0,0),(0,1,0),(0,0,1),'P1')\r\nexperiments = ExperimentListFactory.from_imageset_and_crystal(subset, crystal)\r\nprint experiments[0].imageset._indices\r\n\r\nfrom dxtbx.model.experiment.experiment_list import ExperimentListDumper\r\ndump = ExperimentListDumper(experiments)\r\ndump.as_json(\"sliced_experiments.json\")\r\n\r\nexperiments = ExperimentListFactory.from_json_file(\"sliced_experiments.json\")\r\nprint experiments[0].imageset._indices\r\ndump = ExperimentListDumper(experiments)\r\ndump.as_json(\"roundtrip_experiments.json\")\r\n```\r\n$ libtbx.python show_bug_4.py ../insu6_1_master.h5\r\n[0, \\<snip\\>, 899]\r\n[100]\r\n[100]\r\n[0]\r\n\r\nExpected result:\r\n[0, \\<snip\\>, 899]\r\n[100]\r\n[100]\r\n[100]\r\n\r\n$ diff sliced_experiments.json roundtrip_experiments.json\r\n```\r\n17c17\r\n<       \"template\": \"../insu6_1_master.h5\",\r\n---\r\n>       \"template\": \"/net/dials/raid1/aaron/dectris/eiger16MNov2015/2015_11_10/insu6_1_master.h5\",\r\n```\r\nExpected result: files are identical\r\n\r\n# Problem 5: remove scan and gonio from experiment list with sliced imageset\r\n$ cd /net/dials/raid1/aaron/dectris/eiger16MNov2015/2015_11_10/test\r\n$ cat show_bug_5.py\r\n```\r\nfrom dxtbx.datablock import DataBlockFactory\r\nfrom dxtbx.model.experiment.experiment_list import ExperimentListFactory\r\nfrom dxtbx.imageset import ImageSet\r\n\r\nimport sys\r\ndatablocks = DataBlockFactory.from_filenames([sys.argv[1]])\r\nimageset = datablocks[0].extract_imagesets()[0]\r\nprint imageset._indices\r\nsubset = imageset[100:101]\r\nprint subset._indices\r\nsubset = ImageSet(subset.reader(), subset.indices())\r\nsubset.set_scan(None)\r\nsubset.set_goniometer(None)\r\nprint subset._indices\r\n\r\nfrom dxtbx.model.crystal import crystal_model\r\ncrystal = crystal_model((1,0,0),(0,1,0),(0,0,1),'P1')\r\nexperiments = ExperimentListFactory.from_imageset_and_crystal(subset, crystal)\r\nprint experiments[0].imageset._indices\r\n\r\nfrom dxtbx.model.experiment.experiment_list import ExperimentListDumper\r\ndump = ExperimentListDumper(experiments)\r\ndump.as_json(\"sliced_experiments.json\")\r\n\r\nexperiments = ExperimentListFactory.from_json_file(\"sliced_experiments.json\")\r\nprint experiments[0].imageset._indices\r\ndump = ExperimentListDumper(experiments)\r\ndump.as_json(\"roundtrip_experiments.json\")\r\n```\r\n$ libtbx.python show_bug_5.py ../insu6_1_master.h5\r\n[0, \\<snip\\>, 899]\r\n[100]\r\n[100]\r\n[100]\r\nTraceback (most recent call last):\r\n  File \"show_bug_5.py\", line 25, in <module>\r\n    experiments = ExperimentListFactory.from_json_file(\"sliced_experiments.json\")\r\n  File \"/net/cci-filer2/raid1/home/aaron/projects/phenix-src/cctbx_project/dxtbx/model/experiment/experiment_list.py\", line 979, in from_json_file\r\n    return ExperimentListFactory.from_json(infile.read(), check_format)\r\n  File \"/net/cci-filer2/raid1/home/aaron/projects/phenix-src/cctbx_project/dxtbx/model/experiment/experiment_list.py\", line 969, in from_json\r\n    check_format)\r\n  File \"/net/cci-filer2/raid1/home/aaron/projects/phenix-src/cctbx_project/dxtbx/model/experiment/experiment_list.py\", line 954, in from_dict\r\n    experiments = ExperimentListDict(obj, check_format).decode()\r\n  File \"/net/cci-filer2/raid1/home/aaron/projects/phenix-src/cctbx_project/dxtbx/model/experiment/experiment_list.py\", line 439, in decode\r\n    self._ilist = self._extract_imagesets()\r\n  File \"/net/cci-filer2/raid1/home/aaron/projects/phenix-src/cctbx_project/dxtbx/model/experiment/experiment_list.py\", line 502, in _extract_imagesets\r\n    raise TypeError('expected int or str, got %s' % type(value))\r\nTypeError: expected int or str, got <type 'list'>\r\n\r\nExpected result: no crash.  Cannot test if the roundtrip json is identical to the sliced json.\r\n\r\n"}}
{"actor":{"display_login":"jmp1985"}}