{"action":"created","issue":{"url":"https://api.github.com/repos/SmarterApp/RDW_Ingest/issues/466","repository_url":"https://api.github.com/repos/SmarterApp/RDW_Ingest","labels_url":"https://api.github.com/repos/SmarterApp/RDW_Ingest/issues/466/labels{/name}","comments_url":"https://api.github.com/repos/SmarterApp/RDW_Ingest/issues/466/comments","events_url":"https://api.github.com/repos/SmarterApp/RDW_Ingest/issues/466/events","html_url":"https://github.com/SmarterApp/RDW_Ingest/pull/466","id":305314858,"number":466,"title":"DWR-1216 OLAP Migrate: modify to support IAB","user":{"login":"agorina","id":16457484,"avatar_url":"https://avatars2.githubusercontent.com/u/16457484?v=4","gravatar_id":"","url":"https://api.github.com/users/agorina","html_url":"https://github.com/agorina","followers_url":"https://api.github.com/users/agorina/followers","following_url":"https://api.github.com/users/agorina/following{/other_user}","gists_url":"https://api.github.com/users/agorina/gists{/gist_id}","starred_url":"https://api.github.com/users/agorina/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/agorina/subscriptions","organizations_url":"https://api.github.com/users/agorina/orgs","repos_url":"https://api.github.com/users/agorina/repos","events_url":"https://api.github.com/users/agorina/events{/privacy}","received_events_url":"https://api.github.com/users/agorina/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-03-14T20:22:38Z","updated_at":"2018-03-15T02:02:35Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/SmarterApp/RDW_Ingest/pulls/466","html_url":"https://github.com/SmarterApp/RDW_Ingest/pull/466","diff_url":"https://github.com/SmarterApp/RDW_Ingest/pull/466.diff","patch_url":"https://github.com/SmarterApp/RDW_Ingest/pull/466.patch"},"body":"Some stats from AWS DEV:\r\n\r\nWarehouse:\r\n`select count(*) from exam;` -- 36,950,293\r\n`select count(*) FROM (SELECT DISTINCT asmt_id, school_year, student_id FROM exam ) tmp;`\r\n-- 13,455,425\r\n\r\nWhen I was testing the migrate, from Redshift, after these changes:\r\n`select count(*) from fact_student_exam;` -- 1,676,116\r\n`select count(*) from fact_student_iab_exam;` -- 11,779,309\r\n\r\nAnd the execution time with the `migrate.olap-batch.size` set to 5 mil.:\r\n<img width=\"400\" alt=\"olap-batch-5mil\" src=\"https://user-images.githubusercontent.com/16457484/37429037-5f0fe352-278b-11e8-82f7-0296f3e60535.png\">\r\n\r\nOverall run time 46 min.\r\n\r\nThis leads me to a question: any objections if I change the default and all the deployment/config script to `migrate.olap-batch.size=5000000`? (@malaffoon)\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/SmarterApp/RDW_Ingest/issues/comments/373235500","html_url":"https://github.com/SmarterApp/RDW_Ingest/pull/466#issuecomment-373235500","issue_url":"https://api.github.com/repos/SmarterApp/RDW_Ingest/issues/466","id":373235500,"user":{"login":"agorina","id":16457484,"avatar_url":"https://avatars2.githubusercontent.com/u/16457484?v=4","gravatar_id":"","url":"https://api.github.com/users/agorina","html_url":"https://github.com/agorina","followers_url":"https://api.github.com/users/agorina/followers","following_url":"https://api.github.com/users/agorina/following{/other_user}","gists_url":"https://api.github.com/users/agorina/gists{/gist_id}","starred_url":"https://api.github.com/users/agorina/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/agorina/subscriptions","organizations_url":"https://api.github.com/users/agorina/orgs","repos_url":"https://api.github.com/users/agorina/repos","events_url":"https://api.github.com/users/agorina/events{/privacy}","received_events_url":"https://api.github.com/users/agorina/received_events","type":"User","site_admin":false},"created_at":"2018-03-15T02:02:35Z","updated_at":"2018-03-15T02:02:35Z","author_association":"CONTRIBUTOR","body":"About 5 mil.: As you could tell I was using whatever instances we have and they handled it just fine. \r\nThe largest extract from Aurora was 'exam' file and it was about 0.5GB. After 6 GB it starts breaking it into multiple files, and we do not support that.  I admit that 6-8 min per batch seems long enough to me. I am not sure I would want to go higher, but if needed we could test."}}
{"actor":{"display_login":"agorina"}}