{"action":"created","issue":{"url":"https://api.github.com/repos/src-d/gemini/issues/42","repository_url":"https://api.github.com/repos/src-d/gemini","labels_url":"https://api.github.com/repos/src-d/gemini/issues/42/labels{/name}","comments_url":"https://api.github.com/repos/src-d/gemini/issues/42/comments","events_url":"https://api.github.com/repos/src-d/gemini/issues/42/events","html_url":"https://github.com/src-d/gemini/issues/42","id":290938163,"node_id":"MDU6SXNzdWUyOTA5MzgxNjM=","number":42,"title":"Run Gemini file-level duplicate detection on PGA","user":{"login":"bzz","id":5582506,"node_id":"MDQ6VXNlcjU1ODI1MDY=","avatar_url":"https://avatars3.githubusercontent.com/u/5582506?v=4","gravatar_id":"","url":"https://api.github.com/users/bzz","html_url":"https://github.com/bzz","followers_url":"https://api.github.com/users/bzz/followers","following_url":"https://api.github.com/users/bzz/following{/other_user}","gists_url":"https://api.github.com/users/bzz/gists{/gist_id}","starred_url":"https://api.github.com/users/bzz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bzz/subscriptions","organizations_url":"https://api.github.com/users/bzz/orgs","repos_url":"https://api.github.com/users/bzz/repos","events_url":"https://api.github.com/users/bzz/events{/privacy}","received_events_url":"https://api.github.com/users/bzz/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"bzz","id":5582506,"node_id":"MDQ6VXNlcjU1ODI1MDY=","avatar_url":"https://avatars3.githubusercontent.com/u/5582506?v=4","gravatar_id":"","url":"https://api.github.com/users/bzz","html_url":"https://github.com/bzz","followers_url":"https://api.github.com/users/bzz/followers","following_url":"https://api.github.com/users/bzz/following{/other_user}","gists_url":"https://api.github.com/users/bzz/gists{/gist_id}","starred_url":"https://api.github.com/users/bzz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bzz/subscriptions","organizations_url":"https://api.github.com/users/bzz/orgs","repos_url":"https://api.github.com/users/bzz/repos","events_url":"https://api.github.com/users/bzz/events{/privacy}","received_events_url":"https://api.github.com/users/bzz/received_events","type":"User","site_admin":false},"assignees":[{"login":"bzz","id":5582506,"node_id":"MDQ6VXNlcjU1ODI1MDY=","avatar_url":"https://avatars3.githubusercontent.com/u/5582506?v=4","gravatar_id":"","url":"https://api.github.com/users/bzz","html_url":"https://github.com/bzz","followers_url":"https://api.github.com/users/bzz/followers","following_url":"https://api.github.com/users/bzz/following{/other_user}","gists_url":"https://api.github.com/users/bzz/gists{/gist_id}","starred_url":"https://api.github.com/users/bzz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bzz/subscriptions","organizations_url":"https://api.github.com/users/bzz/orgs","repos_url":"https://api.github.com/users/bzz/repos","events_url":"https://api.github.com/users/bzz/events{/privacy}","received_events_url":"https://api.github.com/users/bzz/received_events","type":"User","site_admin":false}],"milestone":null,"comments":26,"created_at":"2018-01-23T18:03:43Z","updated_at":"2018-06-28T16:03:02Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Document in README the resources, needed to successfully process 1k, 2k, 10k, 100k and whole PGA of the .siva files.\r\n\r\nSo good start would be \r\n - document the known configuration of the cluster we use internally\r\n - running Gemini hash, documenting how long does it take to finish, \r\n - what CPU-load, Mem, IO-thoughtput workload it creates on that cluster (i.e from [Sysdig Dashbord](https://app.sysdigcloud.com/#/dashboards/51248/l:3600), to get access [file an issue](https://github.com/src-d/issues-infrastructure/issues/99))\r\n - see which resource is a bottleneck\r\n - try to optimize, in order to utilize that resource better (i.e in case of throughput - have more executor JVMs running on the same machine)\r\n - see if we are hit and can help with some Engine issues\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/src-d/gemini/issues/comments/401086443","html_url":"https://github.com/src-d/gemini/issues/42#issuecomment-401086443","issue_url":"https://api.github.com/repos/src-d/gemini/issues/42","id":401086443,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMTA4NjQ0Mw==","user":{"login":"bzz","id":5582506,"node_id":"MDQ6VXNlcjU1ODI1MDY=","avatar_url":"https://avatars3.githubusercontent.com/u/5582506?v=4","gravatar_id":"","url":"https://api.github.com/users/bzz","html_url":"https://github.com/bzz","followers_url":"https://api.github.com/users/bzz/followers","following_url":"https://api.github.com/users/bzz/following{/other_user}","gists_url":"https://api.github.com/users/bzz/gists{/gist_id}","starred_url":"https://api.github.com/users/bzz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bzz/subscriptions","organizations_url":"https://api.github.com/users/bzz/orgs","repos_url":"https://api.github.com/users/bzz/repos","events_url":"https://api.github.com/users/bzz/events{/privacy}","received_events_url":"https://api.github.com/users/bzz/received_events","type":"User","site_admin":false},"created_at":"2018-06-28T16:03:02Z","updated_at":"2018-06-28T16:03:02Z","author_association":"CONTRIBUTOR","body":"> Question: how are we sampling the repos for each of these tests?\r\n\r\nGood question. We always just used only a single shard of PGA dataset - all the repos, who's siva file names start with prefix `/ff/`.\r\n\r\nOverall, on Apache Spark performance depends on data distribution A LOT, so attaching .siva file size distribution histogram in 10mb buckets\r\n\r\n```\r\nhdfs dfs -du hdfs://hdfs-namenode/pga/siva/latest/ff/ | grep \"\\.siva$\" | awk -v \"size=100048576\" -f hist.awk\r\n0 100048576 912\r\n100048576 200097152 17\r\n200097152 300145728 2\r\n300145728 400194304 1\r\n400194304 500242880 1\r\n500242880 600291456\r\n600291456 700340032\r\n700340032 800388608\r\n800388608 900437184\r\n900437184 1000485760 1\r\n1000485760 1100534336 1\r\n```\r\n"}}
{"actor":{"display_login":"bzz"}}