{"action":"created","issue":{"url":"https://api.github.com/repos/snowflakedb/snowflake-connector-python/issues/134","repository_url":"https://api.github.com/repos/snowflakedb/snowflake-connector-python","labels_url":"https://api.github.com/repos/snowflakedb/snowflake-connector-python/issues/134/labels{/name}","comments_url":"https://api.github.com/repos/snowflakedb/snowflake-connector-python/issues/134/comments","events_url":"https://api.github.com/repos/snowflakedb/snowflake-connector-python/issues/134/events","html_url":"https://github.com/snowflakedb/snowflake-connector-python/issues/134","id":385107167,"node_id":"MDU6SXNzdWUzODUxMDcxNjc=","number":134,"title":"Download concurrency control","user":{"login":"artdgn","id":29574203,"node_id":"MDQ6VXNlcjI5NTc0MjAz","avatar_url":"https://avatars0.githubusercontent.com/u/29574203?v=4","gravatar_id":"","url":"https://api.github.com/users/artdgn","html_url":"https://github.com/artdgn","followers_url":"https://api.github.com/users/artdgn/followers","following_url":"https://api.github.com/users/artdgn/following{/other_user}","gists_url":"https://api.github.com/users/artdgn/gists{/gist_id}","starred_url":"https://api.github.com/users/artdgn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/artdgn/subscriptions","organizations_url":"https://api.github.com/users/artdgn/orgs","repos_url":"https://api.github.com/users/artdgn/repos","events_url":"https://api.github.com/users/artdgn/events{/privacy}","received_events_url":"https://api.github.com/users/artdgn/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-11-28T05:24:27Z","updated_at":"2018-11-28T06:33:22Z","closed_at":null,"author_association":"NONE","body":"It seems that there is no easy way to set the download concurrency settings for the connector. \r\n\r\nThis results in very slow downloads for large result sets (as the default values are quite small).\r\n\r\nFrom what I understand the smaller between `DEFAULT_CLIENT_RESULT_PREFETCH_SLOTS` and `DEFAULT_CLIENT_RESULT_PREFETCH_THREADS` controls the concurrency. \r\n\r\nAt runtime those parameters are also received from the query response, but there doesn't seem to be a way to set them. Executing `ALTER SESSION SET CLIENT_PREFETCH_THREADS = 10` has no effect, and `ALTER SESSION SET CLIENT_RESULT_PREFETCH_SLOTS = 10` results in `invalid parameter 'CLIENT_RESULT_PREFETCH_SLOTS'` error.\r\n\r\nWhat this causes is a very slow download, because if some chunk is slow to download or the connection is dropped, it delays the download of all other chunks.\r\n\r\nMonkey patching the relevant parts of the code to increase those values at runtime results in much faster download speeds.\r\n\r\nThe questions are:\r\n1) What is the supported way to increase concurrency? \r\n2) If there is none yet, will you be willing to accept a PR to the connector that enables a simpler override for those values?"},"comment":{"url":"https://api.github.com/repos/snowflakedb/snowflake-connector-python/issues/comments/442335855","html_url":"https://github.com/snowflakedb/snowflake-connector-python/issues/134#issuecomment-442335855","issue_url":"https://api.github.com/repos/snowflakedb/snowflake-connector-python/issues/134","id":442335855,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MjMzNTg1NQ==","user":{"login":"artdgn","id":29574203,"node_id":"MDQ6VXNlcjI5NTc0MjAz","avatar_url":"https://avatars0.githubusercontent.com/u/29574203?v=4","gravatar_id":"","url":"https://api.github.com/users/artdgn","html_url":"https://github.com/artdgn","followers_url":"https://api.github.com/users/artdgn/followers","following_url":"https://api.github.com/users/artdgn/following{/other_user}","gists_url":"https://api.github.com/users/artdgn/gists{/gist_id}","starred_url":"https://api.github.com/users/artdgn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/artdgn/subscriptions","organizations_url":"https://api.github.com/users/artdgn/orgs","repos_url":"https://api.github.com/users/artdgn/repos","events_url":"https://api.github.com/users/artdgn/events{/privacy}","received_events_url":"https://api.github.com/users/artdgn/received_events","type":"User","site_admin":false},"created_at":"2018-11-28T06:33:22Z","updated_at":"2018-11-28T06:33:22Z","author_association":"NONE","body":"In case anyone is interested in the monkey-patch until a fix a proposed / created:\r\n```python\r\nimport snowflake.connector\r\nimport snowflake.connector.chunk_downloader\r\n\r\nclass PatchedSnowflakeChunkDownloader(\r\n    snowflake.connector.chunk_downloader.SnowflakeChunkDownloader):\r\n\r\n    DEFAULT_CLIENT_RESULT_PREFETCH_SLOTS = 1000  # will be set to number of chunks\r\n    DEFAULT_CLIENT_RESULT_PREFETCH_THREADS = 1000  # will be set to number of chunks\r\n\r\n    def _pre_init(self, *args, **kwargs):\r\n        kwargs.pop('prefetch_slots', None)\r\n        kwargs.pop('prefetch_threads', None)\r\n        snowflake.connector.chunk_downloader.logger.info('Setting download concurrency to maximum')\r\n        super()._pre_init(*args,\r\n                          prefetch_slots=self.DEFAULT_CLIENT_RESULT_PREFETCH_SLOTS,\r\n                          prefetch_threads=self.DEFAULT_CLIENT_RESULT_PREFETCH_THREADS,\r\n                          **kwargs)\r\n\r\nsnowflake.connector.connection.\\\r\n    DEFAULT_CONFIGURATION['chunk_downloader_class'] = PatchedSnowflakeChunkDownloader\r\n```"}}
{"actor":{"display_login":"artdgn"}}