{"action":"created","comment":{"url":"https://api.github.com/repos/broadinstitute/rawls/pulls/comments/58540142","id":58540142,"diff_hunk":"@@ -63,23 +64,53 @@ trait AttributeComponent {\n       }\n     }\n \n-    private def insertAttributeRef(name: String, workspaceId: UUID, ref: AttributeEntityReference, listIndex: Option[Int] = None): ReadWriteAction[Long] = {\n+    private def insertAttributeRef(name: String, workspaceId: UUID, ref: AttributeEntityReference, listIndex: Option[Int] = None): ReadWriteAction[UUID] = {\n+      val uuid = UUID.randomUUID()\n       entityQuery.findEntityByName(workspaceId, ref.entityType, ref.entityName).result.flatMap {\n         case Seq() => throw new RawlsException(s\"$ref not found in workspace $workspaceId\")\n-        case Seq(entityRecord) => (attributeQuery returning attributeQuery.map(_.id)) += marshalAttributeEntityReference(name, listIndex, entityRecord)\n+        case Seq(entityRecord) => (attributeQuery += marshalAttributeEntityReference(uuid, name, listIndex, entityRecord.id)).map(_ => uuid)\n       }\n     }\n \n     private def insertAttributeValue(name: String, value: AttributeValue, listIndex: Option[Int] = None) = {\n-      (attributeQuery returning attributeQuery.map(_.id)) += marshalAttributeValue(name, value, listIndex)\n+      val uuid = UUID.randomUUID()\n+      (attributeQuery += marshalAttributeValue(uuid, name, value, listIndex)).map(_ => uuid)\n     }\n \n-    private def marshalAttributeEntityReference(name: String, listIndex: Option[Int], entityRecord: EntityRecord): AttributeRecord = {\n-      AttributeRecord(0, name, None, None, None, Option(entityRecord.id), listIndex)\n+    def marshalAttribute(name: String, attribute: Attribute, pregeneratedEntityIds: Map[(String, String), UUID]): Seq[AttributeRecord] = {","path":"src/main/scala/org/broadinstitute/dsde/rawls/dataaccess/slick/AttributeComponent.scala","position":69,"original_position":69,"commit_id":"ae241744292e2d6bdc54d7cd439f9bb4feeaac02","original_commit_id":"ae241744292e2d6bdc54d7cd439f9bb4feeaac02","user":{"login":"dvoet","id":1521333,"avatar_url":"https://avatars.githubusercontent.com/u/1521333?v=3","gravatar_id":"","url":"https://api.github.com/users/dvoet","html_url":"https://github.com/dvoet","followers_url":"https://api.github.com/users/dvoet/followers","following_url":"https://api.github.com/users/dvoet/following{/other_user}","gists_url":"https://api.github.com/users/dvoet/gists{/gist_id}","starred_url":"https://api.github.com/users/dvoet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dvoet/subscriptions","organizations_url":"https://api.github.com/users/dvoet/orgs","repos_url":"https://api.github.com/users/dvoet/repos","events_url":"https://api.github.com/users/dvoet/events{/privacy}","received_events_url":"https://api.github.com/users/dvoet/received_events","type":"User","site_admin":false},"body":"rename pregeneratedEntityIds to entityIdsByName","created_at":"2016-04-05T13:46:45Z","updated_at":"2016-04-05T13:46:45Z","html_url":"https://github.com/broadinstitute/rawls/pull/367#discussion_r58540142","pull_request_url":"https://api.github.com/repos/broadinstitute/rawls/pulls/367","_links":{"self":{"href":"https://api.github.com/repos/broadinstitute/rawls/pulls/comments/58540142"},"html":{"href":"https://github.com/broadinstitute/rawls/pull/367#discussion_r58540142"},"pull_request":{"href":"https://api.github.com/repos/broadinstitute/rawls/pulls/367"}}},"pull_request":{"url":"https://api.github.com/repos/broadinstitute/rawls/pulls/367","id":65021812,"html_url":"https://github.com/broadinstitute/rawls/pull/367","diff_url":"https://github.com/broadinstitute/rawls/pull/367.diff","patch_url":"https://github.com/broadinstitute/rawls/pull/367.patch","issue_url":"https://api.github.com/repos/broadinstitute/rawls/issues/367","number":367,"state":"open","locked":false,"title":"Bulk insert entities, attributes and entity_attribute records when cloning workspaces","user":{"login":"MatthewBemis","id":7257391,"avatar_url":"https://avatars.githubusercontent.com/u/7257391?v=3","gravatar_id":"","url":"https://api.github.com/users/MatthewBemis","html_url":"https://github.com/MatthewBemis","followers_url":"https://api.github.com/users/MatthewBemis/followers","following_url":"https://api.github.com/users/MatthewBemis/following{/other_user}","gists_url":"https://api.github.com/users/MatthewBemis/gists{/gist_id}","starred_url":"https://api.github.com/users/MatthewBemis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MatthewBemis/subscriptions","organizations_url":"https://api.github.com/users/MatthewBemis/orgs","repos_url":"https://api.github.com/users/MatthewBemis/repos","events_url":"https://api.github.com/users/MatthewBemis/events{/privacy}","received_events_url":"https://api.github.com/users/MatthewBemis/received_events","type":"User","site_admin":false},"body":"GAWB-249\r\n\r\nThis reduces the number of queries needed to clone the TCGA-BRCA workspace from about 134,000 queries to about 250 (with batch size set to 1000)\r\n\r\nThis also changes the id type of entities and attributes to UUIDs, which allows us to know the id of everything that we're inserting before we insert it, making bulk insert possible.\r\n\r\n~~8 tests still fail, they're all related to h2 not knowing what this batch insert syntax is and thinking it's invalid. I've done some sanity tests in place of those by cloning the TCGA-BRCA workspace and then diffing the entity list responses, and that has looked good so far.~~\r\n\r\nThings that still need to be done:\r\n\r\n~~-Add the finished Liquibase migration script from David Mohs [(see here)](https://github.com/broadinstitute/rawls/blob/dmohs-bulk-clone/src/main/resources/org/broadinstitute/dsde/rawls/liquibase/changesets/20160330_refactor_entities_attributes.xml)~~\r\n~~-Fix unit tests, and do more testing~~\r\n~~-Rebase~~\r\n\r\n- [x] **Submitter**: Rebase to develop. DO NOT SQUASH\r\n- [x] **Submitter**: Make sure Swagger is updated if API changes\r\n- [x] **Submitter**: Make sure documentation for code is complete\r\n- [x] **Submitter**: Make sure liquibase is updated if appropriate\r\n- [x] **Submitter**: Review code comments; remove done TODOs, create stories for remaining TODOs\r\n- [x] **Submitter**: Include the JIRA issue number in the PR description\r\n- [x] **Submitter**: Add description or comments on the PR explaining the hows/whys (if not obvious)\r\n- [x] Tell ![](http://i.imgur.com/9dLzbPd.png) that the PR exists if he wants to look at it\r\n- [x] Anoint a lead reviewer (LR). **Assign PR to LR**\r\n- [x] **LR**: Initial review by LR and others.\r\n- [ ] Comment / review / update cycle:\r\n  * Rest of team may comments on PR at will\r\n  * **LR assigns to submitter** for feedback fixes\r\n  * Submitter updates documentation as needed\r\n  * Submitter rebases to develop again if necessary\r\n  * Submitter makes further commits. DO NOT SQUASH. **Reassign to LR** for further feedback\r\n- [ ] ![](http://i.imgur.com/9dLzbPd.png) sign off\r\n- [ ] **LR** sign off\r\n- [ ] **Assign to submitter** to finalize\r\n- [ ] **Submitter**: Squash commits, rebase if necessary\r\n- [ ] **Submitter**: Verify all tests go green, including CI tests\r\n- [ ] **Submitter**: Merge to develop \r\n- [ ] **Submitter**: Delete branch after merge\r\n- [ ] **Submitter**: Check configuration files in Jenkins in case they need changes\r\n- [ ] **Submitter**: **Test this change works on dev environment after deployment**. YOU own getting it fixed if dev isn't working for ANY reason!\r\n- [ ] **Submitter**: Verify swagger UI on dev environment still works after deployment\r\n- [ ] **Submitter**: Inform other teams of any API changes via hipchat and/or email\r\n- [ ] **Submitter**: Mark JIRA issue as resolved once this checklist is completed\r\n","created_at":"2016-04-02T00:31:50Z","updated_at":"2016-04-05T13:46:45Z","closed_at":null,"merged_at":null,"merge_commit_sha":"d70f32c83277806213b9ecb95b6ae7cadb27b63b","assignee":{"login":"dvoet","id":1521333,"avatar_url":"https://avatars.githubusercontent.com/u/1521333?v=3","gravatar_id":"","url":"https://api.github.com/users/dvoet","html_url":"https://github.com/dvoet","followers_url":"https://api.github.com/users/dvoet/followers","following_url":"https://api.github.com/users/dvoet/following{/other_user}","gists_url":"https://api.github.com/users/dvoet/gists{/gist_id}","starred_url":"https://api.github.com/users/dvoet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dvoet/subscriptions","organizations_url":"https://api.github.com/users/dvoet/orgs","repos_url":"https://api.github.com/users/dvoet/repos","events_url":"https://api.github.com/users/dvoet/events{/privacy}","received_events_url":"https://api.github.com/users/dvoet/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/broadinstitute/rawls/pulls/367/commits","review_comments_url":"https://api.github.com/repos/broadinstitute/rawls/pulls/367/comments","review_comment_url":"https://api.github.com/repos/broadinstitute/rawls/pulls/comments{/number}","comments_url":"https://api.github.com/repos/broadinstitute/rawls/issues/367/comments","statuses_url":"https://api.github.com/repos/broadinstitute/rawls/statuses/ae241744292e2d6bdc54d7cd439f9bb4feeaac02","head":{"label":"broadinstitute:mb_bulk_clone2","ref":"mb_bulk_clone2","sha":"ae241744292e2d6bdc54d7cd439f9bb4feeaac02","user":{"login":"broadinstitute","id":393552,"avatar_url":"https://avatars.githubusercontent.com/u/393552?v=3","gravatar_id":"","url":"https://api.github.com/users/broadinstitute","html_url":"https://github.com/broadinstitute","followers_url":"https://api.github.com/users/broadinstitute/followers","following_url":"https://api.github.com/users/broadinstitute/following{/other_user}","gists_url":"https://api.github.com/users/broadinstitute/gists{/gist_id}","starred_url":"https://api.github.com/users/broadinstitute/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/broadinstitute/subscriptions","organizations_url":"https://api.github.com/users/broadinstitute/orgs","repos_url":"https://api.github.com/users/broadinstitute/repos","events_url":"https://api.github.com/users/broadinstitute/events{/privacy}","received_events_url":"https://api.github.com/users/broadinstitute/received_events","type":"Organization","site_admin":false},"repo":{"id":34129770,"name":"rawls","full_name":"broadinstitute/rawls","owner":{"login":"broadinstitute","id":393552,"avatar_url":"https://avatars.githubusercontent.com/u/393552?v=3","gravatar_id":"","url":"https://api.github.com/users/broadinstitute","html_url":"https://github.com/broadinstitute","followers_url":"https://api.github.com/users/broadinstitute/followers","following_url":"https://api.github.com/users/broadinstitute/following{/other_user}","gists_url":"https://api.github.com/users/broadinstitute/gists{/gist_id}","starred_url":"https://api.github.com/users/broadinstitute/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/broadinstitute/subscriptions","organizations_url":"https://api.github.com/users/broadinstitute/orgs","repos_url":"https://api.github.com/users/broadinstitute/repos","events_url":"https://api.github.com/users/broadinstitute/events{/privacy}","received_events_url":"https://api.github.com/users/broadinstitute/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/broadinstitute/rawls","description":"Rawls service for DSDE","fork":false,"url":"https://api.github.com/repos/broadinstitute/rawls","forks_url":"https://api.github.com/repos/broadinstitute/rawls/forks","keys_url":"https://api.github.com/repos/broadinstitute/rawls/keys{/key_id}","collaborators_url":"https://api.github.com/repos/broadinstitute/rawls/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/broadinstitute/rawls/teams","hooks_url":"https://api.github.com/repos/broadinstitute/rawls/hooks","issue_events_url":"https://api.github.com/repos/broadinstitute/rawls/issues/events{/number}","events_url":"https://api.github.com/repos/broadinstitute/rawls/events","assignees_url":"https://api.github.com/repos/broadinstitute/rawls/assignees{/user}","branches_url":"https://api.github.com/repos/broadinstitute/rawls/branches{/branch}","tags_url":"https://api.github.com/repos/broadinstitute/rawls/tags","blobs_url":"https://api.github.com/repos/broadinstitute/rawls/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/broadinstitute/rawls/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/broadinstitute/rawls/git/refs{/sha}","trees_url":"https://api.github.com/repos/broadinstitute/rawls/git/trees{/sha}","statuses_url":"https://api.github.com/repos/broadinstitute/rawls/statuses/{sha}","languages_url":"https://api.github.com/repos/broadinstitute/rawls/languages","stargazers_url":"https://api.github.com/repos/broadinstitute/rawls/stargazers","contributors_url":"https://api.github.com/repos/broadinstitute/rawls/contributors","subscribers_url":"https://api.github.com/repos/broadinstitute/rawls/subscribers","subscription_url":"https://api.github.com/repos/broadinstitute/rawls/subscription","commits_url":"https://api.github.com/repos/broadinstitute/rawls/commits{/sha}","git_commits_url":"https://api.github.com/repos/broadinstitute/rawls/git/commits{/sha}","comments_url":"https://api.github.com/repos/broadinstitute/rawls/comments{/number}","issue_comment_url":"https://api.github.com/repos/broadinstitute/rawls/issues/comments{/number}","contents_url":"https://api.github.com/repos/broadinstitute/rawls/contents/{+path}","compare_url":"https://api.github.com/repos/broadinstitute/rawls/compare/{base}...{head}","merges_url":"https://api.github.com/repos/broadinstitute/rawls/merges","archive_url":"https://api.github.com/repos/broadinstitute/rawls/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/broadinstitute/rawls/downloads","issues_url":"https://api.github.com/repos/broadinstitute/rawls/issues{/number}","pulls_url":"https://api.github.com/repos/broadinstitute/rawls/pulls{/number}","milestones_url":"https://api.github.com/repos/broadinstitute/rawls/milestones{/number}","notifications_url":"https://api.github.com/repos/broadinstitute/rawls/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/broadinstitute/rawls/labels{/name}","releases_url":"https://api.github.com/repos/broadinstitute/rawls/releases{/id}","deployments_url":"https://api.github.com/repos/broadinstitute/rawls/deployments","created_at":"2015-04-17T17:16:07Z","updated_at":"2016-03-04T19:30:59Z","pushed_at":"2016-04-05T13:35:58Z","git_url":"git://github.com/broadinstitute/rawls.git","ssh_url":"git@github.com:broadinstitute/rawls.git","clone_url":"https://github.com/broadinstitute/rawls.git","svn_url":"https://github.com/broadinstitute/rawls","homepage":null,"size":3661,"stargazers_count":5,"watchers_count":5,"language":"Scala","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":2,"forks":1,"open_issues":2,"watchers":5,"default_branch":"develop"}},"base":{"label":"broadinstitute:develop","ref":"develop","sha":"ff734903e62745201ee3c02052450ad8342556a6","user":{"login":"broadinstitute","id":393552,"avatar_url":"https://avatars.githubusercontent.com/u/393552?v=3","gravatar_id":"","url":"https://api.github.com/users/broadinstitute","html_url":"https://github.com/broadinstitute","followers_url":"https://api.github.com/users/broadinstitute/followers","following_url":"https://api.github.com/users/broadinstitute/following{/other_user}","gists_url":"https://api.github.com/users/broadinstitute/gists{/gist_id}","starred_url":"https://api.github.com/users/broadinstitute/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/broadinstitute/subscriptions","organizations_url":"https://api.github.com/users/broadinstitute/orgs","repos_url":"https://api.github.com/users/broadinstitute/repos","events_url":"https://api.github.com/users/broadinstitute/events{/privacy}","received_events_url":"https://api.github.com/users/broadinstitute/received_events","type":"Organization","site_admin":false},"repo":{"id":34129770,"name":"rawls","full_name":"broadinstitute/rawls","owner":{"login":"broadinstitute","id":393552,"avatar_url":"https://avatars.githubusercontent.com/u/393552?v=3","gravatar_id":"","url":"https://api.github.com/users/broadinstitute","html_url":"https://github.com/broadinstitute","followers_url":"https://api.github.com/users/broadinstitute/followers","following_url":"https://api.github.com/users/broadinstitute/following{/other_user}","gists_url":"https://api.github.com/users/broadinstitute/gists{/gist_id}","starred_url":"https://api.github.com/users/broadinstitute/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/broadinstitute/subscriptions","organizations_url":"https://api.github.com/users/broadinstitute/orgs","repos_url":"https://api.github.com/users/broadinstitute/repos","events_url":"https://api.github.com/users/broadinstitute/events{/privacy}","received_events_url":"https://api.github.com/users/broadinstitute/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/broadinstitute/rawls","description":"Rawls service for DSDE","fork":false,"url":"https://api.github.com/repos/broadinstitute/rawls","forks_url":"https://api.github.com/repos/broadinstitute/rawls/forks","keys_url":"https://api.github.com/repos/broadinstitute/rawls/keys{/key_id}","collaborators_url":"https://api.github.com/repos/broadinstitute/rawls/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/broadinstitute/rawls/teams","hooks_url":"https://api.github.com/repos/broadinstitute/rawls/hooks","issue_events_url":"https://api.github.com/repos/broadinstitute/rawls/issues/events{/number}","events_url":"https://api.github.com/repos/broadinstitute/rawls/events","assignees_url":"https://api.github.com/repos/broadinstitute/rawls/assignees{/user}","branches_url":"https://api.github.com/repos/broadinstitute/rawls/branches{/branch}","tags_url":"https://api.github.com/repos/broadinstitute/rawls/tags","blobs_url":"https://api.github.com/repos/broadinstitute/rawls/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/broadinstitute/rawls/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/broadinstitute/rawls/git/refs{/sha}","trees_url":"https://api.github.com/repos/broadinstitute/rawls/git/trees{/sha}","statuses_url":"https://api.github.com/repos/broadinstitute/rawls/statuses/{sha}","languages_url":"https://api.github.com/repos/broadinstitute/rawls/languages","stargazers_url":"https://api.github.com/repos/broadinstitute/rawls/stargazers","contributors_url":"https://api.github.com/repos/broadinstitute/rawls/contributors","subscribers_url":"https://api.github.com/repos/broadinstitute/rawls/subscribers","subscription_url":"https://api.github.com/repos/broadinstitute/rawls/subscription","commits_url":"https://api.github.com/repos/broadinstitute/rawls/commits{/sha}","git_commits_url":"https://api.github.com/repos/broadinstitute/rawls/git/commits{/sha}","comments_url":"https://api.github.com/repos/broadinstitute/rawls/comments{/number}","issue_comment_url":"https://api.github.com/repos/broadinstitute/rawls/issues/comments{/number}","contents_url":"https://api.github.com/repos/broadinstitute/rawls/contents/{+path}","compare_url":"https://api.github.com/repos/broadinstitute/rawls/compare/{base}...{head}","merges_url":"https://api.github.com/repos/broadinstitute/rawls/merges","archive_url":"https://api.github.com/repos/broadinstitute/rawls/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/broadinstitute/rawls/downloads","issues_url":"https://api.github.com/repos/broadinstitute/rawls/issues{/number}","pulls_url":"https://api.github.com/repos/broadinstitute/rawls/pulls{/number}","milestones_url":"https://api.github.com/repos/broadinstitute/rawls/milestones{/number}","notifications_url":"https://api.github.com/repos/broadinstitute/rawls/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/broadinstitute/rawls/labels{/name}","releases_url":"https://api.github.com/repos/broadinstitute/rawls/releases{/id}","deployments_url":"https://api.github.com/repos/broadinstitute/rawls/deployments","created_at":"2015-04-17T17:16:07Z","updated_at":"2016-03-04T19:30:59Z","pushed_at":"2016-04-05T13:35:58Z","git_url":"git://github.com/broadinstitute/rawls.git","ssh_url":"git@github.com:broadinstitute/rawls.git","clone_url":"https://github.com/broadinstitute/rawls.git","svn_url":"https://github.com/broadinstitute/rawls","homepage":null,"size":3661,"stargazers_count":5,"watchers_count":5,"language":"Scala","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":2,"forks":1,"open_issues":2,"watchers":5,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/broadinstitute/rawls/pulls/367"},"html":{"href":"https://github.com/broadinstitute/rawls/pull/367"},"issue":{"href":"https://api.github.com/repos/broadinstitute/rawls/issues/367"},"comments":{"href":"https://api.github.com/repos/broadinstitute/rawls/issues/367/comments"},"review_comments":{"href":"https://api.github.com/repos/broadinstitute/rawls/pulls/367/comments"},"review_comment":{"href":"https://api.github.com/repos/broadinstitute/rawls/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/broadinstitute/rawls/pulls/367/commits"},"statuses":{"href":"https://api.github.com/repos/broadinstitute/rawls/statuses/ae241744292e2d6bdc54d7cd439f9bb4feeaac02"}}}}