{"action":"created","issue":{"url":"https://api.github.com/repos/ga4gh/server/issues/1104","repository_url":"https://api.github.com/repos/ga4gh/server","labels_url":"https://api.github.com/repos/ga4gh/server/issues/1104/labels{/name}","comments_url":"https://api.github.com/repos/ga4gh/server/issues/1104/comments","events_url":"https://api.github.com/repos/ga4gh/server/issues/1104/events","html_url":"https://github.com/ga4gh/server/pull/1104","id":148696310,"number":1104,"title":"Updates on repo manager","user":{"login":"jeromekelleher","id":2664569,"avatar_url":"https://avatars.githubusercontent.com/u/2664569?v=3","gravatar_id":"","url":"https://api.github.com/users/jeromekelleher","html_url":"https://github.com/jeromekelleher","followers_url":"https://api.github.com/users/jeromekelleher/followers","following_url":"https://api.github.com/users/jeromekelleher/following{/other_user}","gists_url":"https://api.github.com/users/jeromekelleher/gists{/gist_id}","starred_url":"https://api.github.com/users/jeromekelleher/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeromekelleher/subscriptions","organizations_url":"https://api.github.com/users/jeromekelleher/orgs","repos_url":"https://api.github.com/users/jeromekelleher/repos","events_url":"https://api.github.com/users/jeromekelleher/events{/privacy}","received_events_url":"https://api.github.com/users/jeromekelleher/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ga4gh/server/labels/in%20progress","name":"in progress","color":"ededed"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2016-04-15T15:46:38Z","updated_at":"2016-04-19T14:53:14Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/ga4gh/server/pulls/1104","html_url":"https://github.com/ga4gh/server/pull/1104","diff_url":"https://github.com/ga4gh/server/pull/1104.diff","patch_url":"https://github.com/ga4gh/server/pull/1104.patch"},"body":"This is a PR updating the repo manager. Changes:\r\n\r\n1. Changed the SQL schema so that it enforces the integrity of the data model at a schema level using foreign key contraints.\r\n2. Added default name for objects added in the CLI to be derived from the input file.\r\n3. add-readgroupset now supports both local files and remote URLs, along with optionally specifying an index file. The index file is provided via an optional positional argument, so we do ``add-readgroupset repoFile datasetName bam_url index_file``. When a URL is provided, the index is mandatory.\r\n4. Changed the ``check`` command to ``verify``, and reimplemented it. Since the DB schema now guarantees the integrity of the data model, the focus of this command is changed to making sure that the various containers we're pointing at work as they are supposed to.  At the moment it goes through all of the various containers and makes sure that we can actually read some data back from them. This is an essential debug tool I think. The current implementation is crude, but it is still useful. \r\n5. Changed error handling in the CLI to throw exceptions of the relevant type all the way back to the top level, where they are caught. For example when a ReadGroupSet cannot be found with a specific name, this exception is thrown, and we get a message like \"Error: ReadGroupSet with name 'HG000sd' not found\". This is printed to stderr and we exit with status 1. In general, we should aim to handle errors in the repo manager CLI like this, where we just follow the nominal case and expect the exception handling to do the right thing.\r\n6. Re-enabled some repo manager unit tests, and added unit tests for the parts of the CLI that have been implemented.\r\n\r\nDataset, ReferenceSet and ReadGroupSet handling is now in quite good shape. Follow up PRs will add VariantSet, VariantAnnotationSet, and FeatureSet."},"comment":{"url":"https://api.github.com/repos/ga4gh/server/issues/comments/211961593","html_url":"https://github.com/ga4gh/server/pull/1104#issuecomment-211961593","issue_url":"https://api.github.com/repos/ga4gh/server/issues/1104","id":211961593,"user":{"login":"diekhans","id":110053,"avatar_url":"https://avatars.githubusercontent.com/u/110053?v=3","gravatar_id":"","url":"https://api.github.com/users/diekhans","html_url":"https://github.com/diekhans","followers_url":"https://api.github.com/users/diekhans/followers","following_url":"https://api.github.com/users/diekhans/following{/other_user}","gists_url":"https://api.github.com/users/diekhans/gists{/gist_id}","starred_url":"https://api.github.com/users/diekhans/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/diekhans/subscriptions","organizations_url":"https://api.github.com/users/diekhans/orgs","repos_url":"https://api.github.com/users/diekhans/repos","events_url":"https://api.github.com/users/diekhans/events{/privacy}","received_events_url":"https://api.github.com/users/diekhans/received_events","type":"User","site_admin":false},"created_at":"2016-04-19T14:53:14Z","updated_at":"2016-04-19T14:53:14Z","body":"\nCompletely agree that containers with the same name is a\ndisaster.  I would suggest this be the restriction.\n\nIt's gets a bit more complex for the contents of some containers\nfor items that don't naturally have one name.  For instance,\nrepeat annotations.\n\nJerome Kelleher <notifications@github.com> writes:\n> OK, let's consider this from the perspective of a repo administrator. I have a\n> ten thousand BAM files and a few hundred VCFs, spread across a dozen datasets.\n> I want to add a new dataset called 'study_xyz', so I run\n> \n> $ ga4gh_repo add-dataset repo.db study_xyy\n> \n> As it happens, I made a mistake and I typed 'study_xyy' instead, which is the\n> name of an already existing dataset. What is the expected behaviour here? I\n> would say, overwhelmingly,\n> \n> $ ga4gh_repo add-dataset repo.db study_xyy\n> Error: Name 'study_xyy' already exists\n> \n> What would be the purpose of allowing the administrator create a dataset with\n> the same name as an existing dataset? Why would an administrator want to do\n> this? This is surely an error, or there is no value in the concept of a dataset\n> name at all.\n> \n> You might argue that we should use IDs to refer to datasets instead, and that\n> this will allow us to avoid this problem. OK, but I think this gives makes the\n> lives of our repository administrator really hard. For example,\n> \n> $ ga4gh_repo add-dataset repo.db study_xyy\n> Created dataset with ID '123423542134523'\n> $ ga4gh_repo add-dataset repo.db study_xyy\n> Created dataset with ID '123423532134523'\n> \n> Now, which dataset is which? How do I remember which dataset I want to put a\n> particular BAM file into? To me, the whole purpose of names is to provide the\n> repository administrator a way of attaching meaningful (local) identifiers to\n> objects. Otherwise, I think trying to manage a large data repository would be a\n> nightmare.\n> \n> I've concentrated on datasets here, but this generalises. For example, suppose\n> I run the following commands:\n> \n> $  ga4gh_repo add-readgroupset repo.db study_xyz ga4gh-example-data/HG00096.bam\n> $  ga4gh_repo add-readgroupset repo.db study_xyz ga4gh-example-data/HG00096.bam\n> \n> What's the expected behaviour here? Again, I would argue that the user\n> expectations would be, overwhelmingly, that we exit with error status. Why\n> would I want to add the same ReadGroupSet to a dataset? OK, if I really want to\n> I can do\n> \n> $  ga4gh_repo add-readgroupset repo.db study_xyz ga4gh-example-data/HG00096.bam --name=HG00096_copy\n> \n> but this just emphasises the point that we're doing something odd. Unless we\n> use name as a local identifier, we have no way of catching these common user\n> errors beyond hashing the VCF or BAM file.\n> \n> This sort of thing will really matter to an admin who is managing a large\n> repository. Allowing for duplicates names opens up all sorts of possibilities\n> for weird behaviour for zero benefit. Can you think of a concrete case where\n> allowing a duplicate name for a Dataset, ReadGroupSet, ReadGroup, ReferenceSet,\n> Reference, FeatureSet, VariantSet, VariantAnnotationSet or CallSet would be\n> useful? The schema is there to allow a variety of implementations, and some\n> have thought it important that allowing duplicate names will allow freedom of\n> implementation. Fine. But we shouldn't cripple our implementation just for the\n> sake of following the full possible generality of the schema. Our users would\n> not thank us for it.\n> \n> —\n> You are receiving this because you were mentioned.\n> Reply to this email directly or view it on GitHub*\n> \n"}}