{"action":"created","issue":{"url":"https://api.github.com/repos/databricks/spark-redshift/issues/236","repository_url":"https://api.github.com/repos/databricks/spark-redshift","labels_url":"https://api.github.com/repos/databricks/spark-redshift/issues/236/labels{/name}","comments_url":"https://api.github.com/repos/databricks/spark-redshift/issues/236/comments","events_url":"https://api.github.com/repos/databricks/spark-redshift/issues/236/events","html_url":"https://github.com/databricks/spark-redshift/issues/236","id":165346092,"number":236,"title":"How to specify parallel thread for redshift.","user":{"login":"manisha803","id":20174562,"avatar_url":"https://avatars.githubusercontent.com/u/20174562?v=3","gravatar_id":"","url":"https://api.github.com/users/manisha803","html_url":"https://github.com/manisha803","followers_url":"https://api.github.com/users/manisha803/followers","following_url":"https://api.github.com/users/manisha803/following{/other_user}","gists_url":"https://api.github.com/users/manisha803/gists{/gist_id}","starred_url":"https://api.github.com/users/manisha803/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/manisha803/subscriptions","organizations_url":"https://api.github.com/users/manisha803/orgs","repos_url":"https://api.github.com/users/manisha803/repos","events_url":"https://api.github.com/users/manisha803/events{/privacy}","received_events_url":"https://api.github.com/users/manisha803/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-07-13T15:11:25Z","updated_at":"2016-08-04T20:32:44Z","closed_at":"2016-07-13T17:18:23Z","body":"I am loading data into Redshift EMR. It is running damm slow. How to load table in redshift with Parallel thread."},"comment":{"url":"https://api.github.com/repos/databricks/spark-redshift/issues/comments/237674585","html_url":"https://github.com/databricks/spark-redshift/issues/236#issuecomment-237674585","issue_url":"https://api.github.com/repos/databricks/spark-redshift/issues/236","id":237674585,"user":{"login":"joeharris76","id":57046,"avatar_url":"https://avatars.githubusercontent.com/u/57046?v=3","gravatar_id":"","url":"https://api.github.com/users/joeharris76","html_url":"https://github.com/joeharris76","followers_url":"https://api.github.com/users/joeharris76/followers","following_url":"https://api.github.com/users/joeharris76/following{/other_user}","gists_url":"https://api.github.com/users/joeharris76/gists{/gist_id}","starred_url":"https://api.github.com/users/joeharris76/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joeharris76/subscriptions","organizations_url":"https://api.github.com/users/joeharris76/orgs","repos_url":"https://api.github.com/users/joeharris76/repos","events_url":"https://api.github.com/users/joeharris76/events{/privacy}","received_events_url":"https://api.github.com/users/joeharris76/received_events","type":"User","site_admin":false},"created_at":"2016-08-04T20:32:44Z","updated_at":"2016-08-04T20:32:44Z","body":"@manisha803 I was having trouble with something similar. Turns out I needed to parallelise my collection of things to be loaded _before_ calling the write to Redshift. \r\n\r\n```\r\n    // OLD >> loadList.collect.foreach { loadThis =>\r\n    // Revision to run the extracts in parallel\r\n    val loadPar = loadList.collect.par \r\n    loadPar.foreach { loadThis =>\r\n      try{\r\n```\r\nThat `.par` creates a parallelised collection and Spark performs the `.foreach` in parallel for all members of the collection (to the extent that it can given available cluster resources). HTH\r\n"}}
{"actor":{"display_login":"joeharris76"}}