{"action":"created","comment":{"url":"https://api.github.com/repos/isper3at/incubator-rya/pulls/comments/80528493","id":80528493,"diff_hunk":"@@ -0,0 +1,250 @@\n+<!-- Licensed to the Apache Software Foundation (ASF) under one\n+or more contributor license agreements.  See the NOTICE file\n+distributed with this work for additional information\n+regarding copyright ownership.  The ASF licenses this file\n+to you under the Apache License, Version 2.0 (the\n+\"License\"); you may not use this file except in compliance\n+with the License.  You may obtain a copy of the License at\n+\n+  http://www.apache.org/licenses/LICENSE-2.0\n+\n+Unless required by applicable law or agreed to in writing,\n+software distributed under the License is distributed on an\n+\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n+KIND, either express or implied.  See the License for the\n+specific language governing permissions and limitations\n+under the License. -->\n+\n+# Rya Export\n+\n+___\n+\n+## Purpose\n+\n+Rya Export is a java application that can be used to export from a parent Rya datastore to a child Rya datastore. The changes to the child can then be synced back to the parent when the process is run again after the child datastore has collected/modified its data set. The datastores currently supported are Accumulo and MongoDB.\n+\n+![Figure 1: Child to Parent Merge](resources/images/child_to_parent_merge.png)\n+\n+## Overview\n+\n+At deployment time, a Rya child instance will be created and sent out where its users will create, modify, and delete data. This data from the child will then need to be synced with its parent after deployment. By default, the data only needs to be handled at the datastore level and so will not need to know the logical consistency of the higher level models it's storing (i.e. Rya logic). Some decisions that were made about the tool:\n+\n+  * When copying, the parent is copying its data to its child and can optionally filter its data to be after a specified timestamp.\n+\n+  * When merging, the data is going only ONE way: the child is re-integrating its new data with the parent Rya instance (the parent does NOT have to merge its new data into the child).\n+\n+  * For Accumulo datastores, since an update to a row value is just a delete followed by an add, row value updates aren't needed.\n+\n+  * For Accumulo datastores, special care must be taken when there's a difference between the parent's and child's column visibility in the same row. In this case, the two visibility values should be OR'ed together to produce the new visibility value to use.\n+  \n+  * For Accumulo to MongoDB merging, column visibility is ignored when updating.\n+\n+  * The tool should support merging data from after a specific time (start time). This will merge in a smaller subset of data modifications from the child to the parent.\n+  \n+  * The Rya Export Client can handle copying/merging from Accumulo to Accumulo, Accumulo to MongoDB, MongoDB to Accumulo, and MongoDB to MongoDB.\n+\n+## Rya Export Client Process\n+\n+The Rya Export Client can be ran through the command line. It can either be used to copy a child instance from a parent or it can be used to sync an already exported child instance by merging its data over to the parent instance. The user can configure a timestamp for the data to be included so only data coming after that time is replicated. The command line will make calls to an uber jar containing the tool and execute the job. This will also allow extra config parameters (such as the location of the parent and child instance) to be used when called.\n+\n+## Implementation Details\n+\n+The Rya Export Client can be run on any machine connected to both the parent instance machine and the child instance machine. It determines which machine is the parent and child by storing metadata that marks the parent machine after the initial copy.\n+\n+The Accumulo side export uses a TimestampFilter iterator setting to only include the data that fits the time range. This time criteria is added to the Scanner that fetch the Rya statements to copy. The desired statements are fetched from the parent and then added to the child table using a BatchWriter. The process job also checks a parent's authorization list so they can be copied to the child user.\n+\n+The MongoDB side export uses a DB object to search the collection for a Rya statement while sorting them by timestamp.\n+\n+By default the tool will copy all the tables from the parent.\n+\n+When merging, the Rya Export Client Accumulo statement store utilizes its internal DAO to handle the processing of data. It is implemented using Java with a Scanner running over the child Accumulo instance to find the statements to copy and then queries for the same statements in the parent instance. If the statement key is found then it doesn't need to merge the statement and it proceeds to the next statement. The column visibility is combined with an OR operation if only the visibilities are different between the child and parent.","path":"extras/rya.export/README.md","position":60,"original_position":60,"commit_id":"f1c905419fc3d0d6750bd22062e5da28087d61a8","original_commit_id":"f1c905419fc3d0d6750bd22062e5da28087d61a8","user":{"login":"ejwhite922","id":16158321,"avatar_url":"https://avatars.githubusercontent.com/u/16158321?v=3","gravatar_id":"","url":"https://api.github.com/users/ejwhite922","html_url":"https://github.com/ejwhite922","followers_url":"https://api.github.com/users/ejwhite922/followers","following_url":"https://api.github.com/users/ejwhite922/following{/other_user}","gists_url":"https://api.github.com/users/ejwhite922/gists{/gist_id}","starred_url":"https://api.github.com/users/ejwhite922/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ejwhite922/subscriptions","organizations_url":"https://api.github.com/users/ejwhite922/orgs","repos_url":"https://api.github.com/users/ejwhite922/repos","events_url":"https://api.github.com/users/ejwhite922/events{/privacy}","received_events_url":"https://api.github.com/users/ejwhite922/received_events","type":"User","site_admin":false},"body":"There is another bullet for MongoDB.","created_at":"2016-09-26T17:34:16Z","updated_at":"2016-09-26T17:34:16Z","html_url":"https://github.com/isper3at/incubator-rya/pull/19#discussion_r80528493","pull_request_url":"https://api.github.com/repos/isper3at/incubator-rya/pulls/19","_links":{"self":{"href":"https://api.github.com/repos/isper3at/incubator-rya/pulls/comments/80528493"},"html":{"href":"https://github.com/isper3at/incubator-rya/pull/19#discussion_r80528493"},"pull_request":{"href":"https://api.github.com/repos/isper3at/incubator-rya/pulls/19"}}},"pull_request":{"url":"https://api.github.com/repos/isper3at/incubator-rya/pulls/19","id":86011642,"html_url":"https://github.com/isper3at/incubator-rya/pull/19","diff_url":"https://github.com/isper3at/incubator-rya/pull/19.diff","patch_url":"https://github.com/isper3at/incubator-rya/pull/19.patch","issue_url":"https://api.github.com/repos/isper3at/incubator-rya/issues/19","number":19,"state":"open","locked":false,"title":"RYA-191 Added README for rya.export.","user":{"login":"ejwhite922","id":16158321,"avatar_url":"https://avatars.githubusercontent.com/u/16158321?v=3","gravatar_id":"","url":"https://api.github.com/users/ejwhite922","html_url":"https://github.com/ejwhite922","followers_url":"https://api.github.com/users/ejwhite922/followers","following_url":"https://api.github.com/users/ejwhite922/following{/other_user}","gists_url":"https://api.github.com/users/ejwhite922/gists{/gist_id}","starred_url":"https://api.github.com/users/ejwhite922/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ejwhite922/subscriptions","organizations_url":"https://api.github.com/users/ejwhite922/orgs","repos_url":"https://api.github.com/users/ejwhite922/repos","events_url":"https://api.github.com/users/ejwhite922/events{/privacy}","received_events_url":"https://api.github.com/users/ejwhite922/received_events","type":"User","site_admin":false},"body":"## Description\r\nAdded README file for rya.export that explains how to use the client.\r\n\r\n### Tests\r\nViewed README.\r\n\r\n### Links\r\n[Jira](https://issues.apache.org/jira/browse/RYA191)\r\n\r\n### Checklist\r\n- [ ] Code Review\r\n- [x] Squash Commits\r\n\r\n#### People To Review\r\n@isper3at, @pujav65\r\n\r\n","created_at":"2016-09-20T16:35:27Z","updated_at":"2016-09-26T17:34:16Z","closed_at":null,"merged_at":null,"merge_commit_sha":"f3c1f7a44f1dd5052cdc727150131ba6111ed40e","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/isper3at/incubator-rya/pulls/19/commits","review_comments_url":"https://api.github.com/repos/isper3at/incubator-rya/pulls/19/comments","review_comment_url":"https://api.github.com/repos/isper3at/incubator-rya/pulls/comments{/number}","comments_url":"https://api.github.com/repos/isper3at/incubator-rya/issues/19/comments","statuses_url":"https://api.github.com/repos/isper3at/incubator-rya/statuses/f1c905419fc3d0d6750bd22062e5da28087d61a8","head":{"label":"ejwhite922:RYA-191_RyaExportREADME","ref":"RYA-191_RyaExportREADME","sha":"f1c905419fc3d0d6750bd22062e5da28087d61a8","user":{"login":"ejwhite922","id":16158321,"avatar_url":"https://avatars.githubusercontent.com/u/16158321?v=3","gravatar_id":"","url":"https://api.github.com/users/ejwhite922","html_url":"https://github.com/ejwhite922","followers_url":"https://api.github.com/users/ejwhite922/followers","following_url":"https://api.github.com/users/ejwhite922/following{/other_user}","gists_url":"https://api.github.com/users/ejwhite922/gists{/gist_id}","starred_url":"https://api.github.com/users/ejwhite922/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ejwhite922/subscriptions","organizations_url":"https://api.github.com/users/ejwhite922/orgs","repos_url":"https://api.github.com/users/ejwhite922/repos","events_url":"https://api.github.com/users/ejwhite922/events{/privacy}","received_events_url":"https://api.github.com/users/ejwhite922/received_events","type":"User","site_admin":false},"repo":{"id":47428209,"name":"incubator-rya","full_name":"ejwhite922/incubator-rya","owner":{"login":"ejwhite922","id":16158321,"avatar_url":"https://avatars.githubusercontent.com/u/16158321?v=3","gravatar_id":"","url":"https://api.github.com/users/ejwhite922","html_url":"https://github.com/ejwhite922","followers_url":"https://api.github.com/users/ejwhite922/followers","following_url":"https://api.github.com/users/ejwhite922/following{/other_user}","gists_url":"https://api.github.com/users/ejwhite922/gists{/gist_id}","starred_url":"https://api.github.com/users/ejwhite922/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ejwhite922/subscriptions","organizations_url":"https://api.github.com/users/ejwhite922/orgs","repos_url":"https://api.github.com/users/ejwhite922/repos","events_url":"https://api.github.com/users/ejwhite922/events{/privacy}","received_events_url":"https://api.github.com/users/ejwhite922/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/ejwhite922/incubator-rya","description":"Mirror of Apache Rya (Incubating)","fork":true,"url":"https://api.github.com/repos/ejwhite922/incubator-rya","forks_url":"https://api.github.com/repos/ejwhite922/incubator-rya/forks","keys_url":"https://api.github.com/repos/ejwhite922/incubator-rya/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ejwhite922/incubator-rya/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ejwhite922/incubator-rya/teams","hooks_url":"https://api.github.com/repos/ejwhite922/incubator-rya/hooks","issue_events_url":"https://api.github.com/repos/ejwhite922/incubator-rya/issues/events{/number}","events_url":"https://api.github.com/repos/ejwhite922/incubator-rya/events","assignees_url":"https://api.github.com/repos/ejwhite922/incubator-rya/assignees{/user}","branches_url":"https://api.github.com/repos/ejwhite922/incubator-rya/branches{/branch}","tags_url":"https://api.github.com/repos/ejwhite922/incubator-rya/tags","blobs_url":"https://api.github.com/repos/ejwhite922/incubator-rya/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ejwhite922/incubator-rya/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ejwhite922/incubator-rya/git/refs{/sha}","trees_url":"https://api.github.com/repos/ejwhite922/incubator-rya/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ejwhite922/incubator-rya/statuses/{sha}","languages_url":"https://api.github.com/repos/ejwhite922/incubator-rya/languages","stargazers_url":"https://api.github.com/repos/ejwhite922/incubator-rya/stargazers","contributors_url":"https://api.github.com/repos/ejwhite922/incubator-rya/contributors","subscribers_url":"https://api.github.com/repos/ejwhite922/incubator-rya/subscribers","subscription_url":"https://api.github.com/repos/ejwhite922/incubator-rya/subscription","commits_url":"https://api.github.com/repos/ejwhite922/incubator-rya/commits{/sha}","git_commits_url":"https://api.github.com/repos/ejwhite922/incubator-rya/git/commits{/sha}","comments_url":"https://api.github.com/repos/ejwhite922/incubator-rya/comments{/number}","issue_comment_url":"https://api.github.com/repos/ejwhite922/incubator-rya/issues/comments{/number}","contents_url":"https://api.github.com/repos/ejwhite922/incubator-rya/contents/{+path}","compare_url":"https://api.github.com/repos/ejwhite922/incubator-rya/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ejwhite922/incubator-rya/merges","archive_url":"https://api.github.com/repos/ejwhite922/incubator-rya/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ejwhite922/incubator-rya/downloads","issues_url":"https://api.github.com/repos/ejwhite922/incubator-rya/issues{/number}","pulls_url":"https://api.github.com/repos/ejwhite922/incubator-rya/pulls{/number}","milestones_url":"https://api.github.com/repos/ejwhite922/incubator-rya/milestones{/number}","notifications_url":"https://api.github.com/repos/ejwhite922/incubator-rya/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ejwhite922/incubator-rya/labels{/name}","releases_url":"https://api.github.com/repos/ejwhite922/incubator-rya/releases{/id}","deployments_url":"https://api.github.com/repos/ejwhite922/incubator-rya/deployments","created_at":"2015-12-04T20:51:26Z","updated_at":"2015-12-04T20:51:29Z","pushed_at":"2016-09-20T16:31:01Z","git_url":"git://github.com/ejwhite922/incubator-rya.git","ssh_url":"git@github.com:ejwhite922/incubator-rya.git","clone_url":"https://github.com/ejwhite922/incubator-rya.git","svn_url":"https://github.com/ejwhite922/incubator-rya","homepage":null,"size":3621,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"isper3at:RYA-123_MergeTool","ref":"RYA-123_MergeTool","sha":"e0bc56fcc5fba7e20e7357072541d204a78c32a5","user":{"login":"isper3at","id":12698945,"avatar_url":"https://avatars.githubusercontent.com/u/12698945?v=3","gravatar_id":"","url":"https://api.github.com/users/isper3at","html_url":"https://github.com/isper3at","followers_url":"https://api.github.com/users/isper3at/followers","following_url":"https://api.github.com/users/isper3at/following{/other_user}","gists_url":"https://api.github.com/users/isper3at/gists{/gist_id}","starred_url":"https://api.github.com/users/isper3at/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/isper3at/subscriptions","organizations_url":"https://api.github.com/users/isper3at/orgs","repos_url":"https://api.github.com/users/isper3at/repos","events_url":"https://api.github.com/users/isper3at/events{/privacy}","received_events_url":"https://api.github.com/users/isper3at/received_events","type":"User","site_admin":false},"repo":{"id":54146858,"name":"incubator-rya","full_name":"isper3at/incubator-rya","owner":{"login":"isper3at","id":12698945,"avatar_url":"https://avatars.githubusercontent.com/u/12698945?v=3","gravatar_id":"","url":"https://api.github.com/users/isper3at","html_url":"https://github.com/isper3at","followers_url":"https://api.github.com/users/isper3at/followers","following_url":"https://api.github.com/users/isper3at/following{/other_user}","gists_url":"https://api.github.com/users/isper3at/gists{/gist_id}","starred_url":"https://api.github.com/users/isper3at/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/isper3at/subscriptions","organizations_url":"https://api.github.com/users/isper3at/orgs","repos_url":"https://api.github.com/users/isper3at/repos","events_url":"https://api.github.com/users/isper3at/events{/privacy}","received_events_url":"https://api.github.com/users/isper3at/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/isper3at/incubator-rya","description":"Mirror of Apache Rya (Incubating)","fork":true,"url":"https://api.github.com/repos/isper3at/incubator-rya","forks_url":"https://api.github.com/repos/isper3at/incubator-rya/forks","keys_url":"https://api.github.com/repos/isper3at/incubator-rya/keys{/key_id}","collaborators_url":"https://api.github.com/repos/isper3at/incubator-rya/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/isper3at/incubator-rya/teams","hooks_url":"https://api.github.com/repos/isper3at/incubator-rya/hooks","issue_events_url":"https://api.github.com/repos/isper3at/incubator-rya/issues/events{/number}","events_url":"https://api.github.com/repos/isper3at/incubator-rya/events","assignees_url":"https://api.github.com/repos/isper3at/incubator-rya/assignees{/user}","branches_url":"https://api.github.com/repos/isper3at/incubator-rya/branches{/branch}","tags_url":"https://api.github.com/repos/isper3at/incubator-rya/tags","blobs_url":"https://api.github.com/repos/isper3at/incubator-rya/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/isper3at/incubator-rya/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/isper3at/incubator-rya/git/refs{/sha}","trees_url":"https://api.github.com/repos/isper3at/incubator-rya/git/trees{/sha}","statuses_url":"https://api.github.com/repos/isper3at/incubator-rya/statuses/{sha}","languages_url":"https://api.github.com/repos/isper3at/incubator-rya/languages","stargazers_url":"https://api.github.com/repos/isper3at/incubator-rya/stargazers","contributors_url":"https://api.github.com/repos/isper3at/incubator-rya/contributors","subscribers_url":"https://api.github.com/repos/isper3at/incubator-rya/subscribers","subscription_url":"https://api.github.com/repos/isper3at/incubator-rya/subscription","commits_url":"https://api.github.com/repos/isper3at/incubator-rya/commits{/sha}","git_commits_url":"https://api.github.com/repos/isper3at/incubator-rya/git/commits{/sha}","comments_url":"https://api.github.com/repos/isper3at/incubator-rya/comments{/number}","issue_comment_url":"https://api.github.com/repos/isper3at/incubator-rya/issues/comments{/number}","contents_url":"https://api.github.com/repos/isper3at/incubator-rya/contents/{+path}","compare_url":"https://api.github.com/repos/isper3at/incubator-rya/compare/{base}...{head}","merges_url":"https://api.github.com/repos/isper3at/incubator-rya/merges","archive_url":"https://api.github.com/repos/isper3at/incubator-rya/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/isper3at/incubator-rya/downloads","issues_url":"https://api.github.com/repos/isper3at/incubator-rya/issues{/number}","pulls_url":"https://api.github.com/repos/isper3at/incubator-rya/pulls{/number}","milestones_url":"https://api.github.com/repos/isper3at/incubator-rya/milestones{/number}","notifications_url":"https://api.github.com/repos/isper3at/incubator-rya/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/isper3at/incubator-rya/labels{/name}","releases_url":"https://api.github.com/repos/isper3at/incubator-rya/releases{/id}","deployments_url":"https://api.github.com/repos/isper3at/incubator-rya/deployments","created_at":"2016-03-17T19:47:58Z","updated_at":"2016-03-17T19:48:00Z","pushed_at":"2016-09-20T16:35:27Z","git_url":"git://github.com/isper3at/incubator-rya.git","ssh_url":"git@github.com:isper3at/incubator-rya.git","clone_url":"https://github.com/isper3at/incubator-rya.git","svn_url":"https://github.com/isper3at/incubator-rya","homepage":null,"size":3705,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":3,"forks":0,"open_issues":3,"watchers":0,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/isper3at/incubator-rya/pulls/19"},"html":{"href":"https://github.com/isper3at/incubator-rya/pull/19"},"issue":{"href":"https://api.github.com/repos/isper3at/incubator-rya/issues/19"},"comments":{"href":"https://api.github.com/repos/isper3at/incubator-rya/issues/19/comments"},"review_comments":{"href":"https://api.github.com/repos/isper3at/incubator-rya/pulls/19/comments"},"review_comment":{"href":"https://api.github.com/repos/isper3at/incubator-rya/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/isper3at/incubator-rya/pulls/19/commits"},"statuses":{"href":"https://api.github.com/repos/isper3at/incubator-rya/statuses/f1c905419fc3d0d6750bd22062e5da28087d61a8"}}}}
{"actor":{"display_login":"ejwhite922"}}