{"action":"created","issue":{"url":"https://api.github.com/repos/opencb/opencga/issues/961","repository_url":"https://api.github.com/repos/opencb/opencga","labels_url":"https://api.github.com/repos/opencb/opencga/issues/961/labels{/name}","comments_url":"https://api.github.com/repos/opencb/opencga/issues/961/comments","events_url":"https://api.github.com/repos/opencb/opencga/issues/961/events","html_url":"https://github.com/opencb/opencga/issues/961","id":390616298,"node_id":"MDU6SXNzdWUzOTA2MTYyOTg=","number":961,"title":"Investigate moving data between HBase Clusters","user":{"login":"martinpeck","id":1524451,"node_id":"MDQ6VXNlcjE1MjQ0NTE=","avatar_url":"https://avatars3.githubusercontent.com/u/1524451?v=4","gravatar_id":"","url":"https://api.github.com/users/martinpeck","html_url":"https://github.com/martinpeck","followers_url":"https://api.github.com/users/martinpeck/followers","following_url":"https://api.github.com/users/martinpeck/following{/other_user}","gists_url":"https://api.github.com/users/martinpeck/gists{/gist_id}","starred_url":"https://api.github.com/users/martinpeck/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martinpeck/subscriptions","organizations_url":"https://api.github.com/users/martinpeck/orgs","repos_url":"https://api.github.com/users/martinpeck/repos","events_url":"https://api.github.com/users/martinpeck/events{/privacy}","received_events_url":"https://api.github.com/users/martinpeck/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-12-13T10:35:51Z","updated_at":"2018-12-14T08:48:07Z","closed_at":null,"author_association":"NONE","body":"Many clinical genomic projects have _researchers_ associated to study all the genome data collected. In this scenario we can identify two main big use cases:\r\n- clinical: data from patients are being loaded continuously, annotated and interpreted on a daily basis.\r\n- research: stable data snapshot are needed for researchers to have time to analyse the data. This is _read-only_ data.\r\n\r\nWe need to investigate how we make the different big data snapshots available to researchers in an optimal way.\r\n\r\n**Option 1 - Replication**\r\nOpenCGA implements export/import functionality. Data could be exported every 6 months and imported in a second installation in read-only mode. By doing this there are two copies of the data in different installations with different users and goals, \r\n\r\n**Option 2 - Only Spark**\r\nOpenCGA can export data in Apache Parquet format to be analysed with Spark. OpenCB Oskar (https://github.com/opencb/oskar) implements a genomic analysis spark library and notebooks for researchers to analyse data. This option is easier to setup and there is less data replicated, but researcher will need to query the OpenCGA installation for some queries, there could be some problems between users.\r\n"},"comment":{"url":"https://api.github.com/repos/opencb/opencga/issues/comments/447257113","html_url":"https://github.com/opencb/opencga/issues/961#issuecomment-447257113","issue_url":"https://api.github.com/repos/opencb/opencga/issues/961","id":447257113,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NzI1NzExMw==","user":{"login":"PAOLT","id":26501424,"node_id":"MDQ6VXNlcjI2NTAxNDI0","avatar_url":"https://avatars2.githubusercontent.com/u/26501424?v=4","gravatar_id":"","url":"https://api.github.com/users/PAOLT","html_url":"https://github.com/PAOLT","followers_url":"https://api.github.com/users/PAOLT/followers","following_url":"https://api.github.com/users/PAOLT/following{/other_user}","gists_url":"https://api.github.com/users/PAOLT/gists{/gist_id}","starred_url":"https://api.github.com/users/PAOLT/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PAOLT/subscriptions","organizations_url":"https://api.github.com/users/PAOLT/orgs","repos_url":"https://api.github.com/users/PAOLT/repos","events_url":"https://api.github.com/users/PAOLT/events{/privacy}","received_events_url":"https://api.github.com/users/PAOLT/received_events","type":"User","site_admin":false},"created_at":"2018-12-14T08:48:07Z","updated_at":"2018-12-14T08:48:07Z","author_association":"NONE","body":"Some useful links:\r\n\r\n**Setting up Backup and Replication for HBase and Phoenix on HDInsight**\r\nhttps://github.com/ZoinerTejada/hdinsight-docs/blob/master/hdinsight-hbase-backup-replication.md\r\n\r\n**Working with the HBase Import and Export Utility**\r\nhttps://blogs.msdn.microsoft.com/data_otaku/2016/12/21/working-with-the-hbase-import-and-export-utility/\r\n\r\n**Copy Activity performance and tuning guide**\r\nhttps://docs.microsoft.com/en-us/azure/data-factory/copy-activity-performance"}}
{"actor":{"display_login":"PAOLT"}}