{"action":"created","issue":{"url":"https://api.github.com/repos/src-d/datasets/issues/53","repository_url":"https://api.github.com/repos/src-d/datasets","labels_url":"https://api.github.com/repos/src-d/datasets/issues/53/labels{/name}","comments_url":"https://api.github.com/repos/src-d/datasets/issues/53/comments","events_url":"https://api.github.com/repos/src-d/datasets/issues/53/events","html_url":"https://github.com/src-d/datasets/issues/53","id":321558563,"node_id":"MDU6SXNzdWUzMjE1NTg1NjM=","number":53,"title":"Question on full PGA integrity verification","user":{"login":"bzz","id":5582506,"node_id":"MDQ6VXNlcjU1ODI1MDY=","avatar_url":"https://avatars3.githubusercontent.com/u/5582506?v=4","gravatar_id":"","url":"https://api.github.com/users/bzz","html_url":"https://github.com/bzz","followers_url":"https://api.github.com/users/bzz/followers","following_url":"https://api.github.com/users/bzz/following{/other_user}","gists_url":"https://api.github.com/users/bzz/gists{/gist_id}","starred_url":"https://api.github.com/users/bzz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bzz/subscriptions","organizations_url":"https://api.github.com/users/bzz/orgs","repos_url":"https://api.github.com/users/bzz/repos","events_url":"https://api.github.com/users/bzz/events{/privacy}","received_events_url":"https://api.github.com/users/bzz/received_events","type":"User","site_admin":false},"labels":[{"id":816336840,"node_id":"MDU6TGFiZWw4MTYzMzY4NDA=","url":"https://api.github.com/repos/src-d/datasets/labels/question","name":"question","color":"d876e3","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/src-d/datasets/milestones/1","html_url":"https://github.com/src-d/datasets/milestone/1","labels_url":"https://api.github.com/repos/src-d/datasets/milestones/1/labels","id":3333786,"node_id":"MDk6TWlsZXN0b25lMzMzMzc4Ng==","number":1,"title":"pga-release","description":"","creator":{"login":"campoy","id":2237452,"node_id":"MDQ6VXNlcjIyMzc0NTI=","avatar_url":"https://avatars3.githubusercontent.com/u/2237452?v=4","gravatar_id":"","url":"https://api.github.com/users/campoy","html_url":"https://github.com/campoy","followers_url":"https://api.github.com/users/campoy/followers","following_url":"https://api.github.com/users/campoy/following{/other_user}","gists_url":"https://api.github.com/users/campoy/gists{/gist_id}","starred_url":"https://api.github.com/users/campoy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/campoy/subscriptions","organizations_url":"https://api.github.com/users/campoy/orgs","repos_url":"https://api.github.com/users/campoy/repos","events_url":"https://api.github.com/users/campoy/events{/privacy}","received_events_url":"https://api.github.com/users/campoy/received_events","type":"User","site_admin":false},"open_issues":6,"closed_issues":7,"state":"open","created_at":"2018-05-09T17:10:15Z","updated_at":"2018-05-22T11:25:44Z","due_on":"2018-05-15T07:00:00Z","closed_at":null},"comments":24,"created_at":"2018-05-09T12:54:31Z","updated_at":"2018-06-18T13:30:04Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"I have downloaded full PGA to HDFS using `pga get -v ...` and have full logs of the process.\r\n\r\nIt took a day and has finished eventually and can see that it's \r\n\r\n```\r\n$ hdfs dfs -du -s -h hdfs://hdfs-namenode/pga\r\n2.4 T  hdfs://hdfs-namenode/pga\r\n```\r\n\r\nQuesting: how do I make sure that nothing is missing?\r\n\r\nWhat would be the simplest way to verify consistency and completeness of the results of `pga get` ? Options from the top of my head include\r\n - some grep over logs \r\n - or re-run tool again multiple times, expecting it is safe, and does not take the same amount of time?\r\n -   `hdfs dfs -ls -R hdfs://hdfs-namenode/pga/ |grep \"\\.siva$\" | wc -l` = 239807\r\nand  `pga list | wc -l` = 181481 but it's rooted repository VS actual repository :/\r\n\r\nWould appreciate any recommendations.\r\n\r\n@campoy I guess this might be something that is worth documenting eventually, as other users might have same question eventually. Would be happy to submit a PR."},"comment":{"url":"https://api.github.com/repos/src-d/datasets/issues/comments/398054240","html_url":"https://github.com/src-d/datasets/issues/53#issuecomment-398054240","issue_url":"https://api.github.com/repos/src-d/datasets/issues/53","id":398054240,"node_id":"MDEyOklzc3VlQ29tbWVudDM5ODA1NDI0MA==","user":{"login":"bzz","id":5582506,"node_id":"MDQ6VXNlcjU1ODI1MDY=","avatar_url":"https://avatars3.githubusercontent.com/u/5582506?v=4","gravatar_id":"","url":"https://api.github.com/users/bzz","html_url":"https://github.com/bzz","followers_url":"https://api.github.com/users/bzz/followers","following_url":"https://api.github.com/users/bzz/following{/other_user}","gists_url":"https://api.github.com/users/bzz/gists{/gist_id}","starred_url":"https://api.github.com/users/bzz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bzz/subscriptions","organizations_url":"https://api.github.com/users/bzz/orgs","repos_url":"https://api.github.com/users/bzz/repos","events_url":"https://api.github.com/users/bzz/events{/privacy}","received_events_url":"https://api.github.com/users/bzz/received_events","type":"User","site_admin":false},"created_at":"2018-06-18T13:30:04Z","updated_at":"2018-06-18T13:30:04Z","author_association":"CONTRIBUTOR","body":"After another run of full PGA download:\r\n - total file size is bigger, 2.5Tb\r\n - 74 .siva files are missing from the second run (network failure?)\r\n\r\nMore details in https://github.com/src-d/datasets/pull/69#issuecomment-398053657"}}
{"actor":{"display_login":"bzz"}}