{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/DataflowPythonSDK/issues/9","repository_url":"https://api.github.com/repos/GoogleCloudPlatform/DataflowPythonSDK","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/DataflowPythonSDK/issues/9/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/DataflowPythonSDK/issues/9/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/DataflowPythonSDK/issues/9/events","html_url":"https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/9","id":148203908,"number":9,"title":"Connecting to Cloud SQL","user":{"login":"sthomp","id":1254471,"avatar_url":"https://avatars.githubusercontent.com/u/1254471?v=3","gravatar_id":"","url":"https://api.github.com/users/sthomp","html_url":"https://github.com/sthomp","followers_url":"https://api.github.com/users/sthomp/followers","following_url":"https://api.github.com/users/sthomp/following{/other_user}","gists_url":"https://api.github.com/users/sthomp/gists{/gist_id}","starred_url":"https://api.github.com/users/sthomp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sthomp/subscriptions","organizations_url":"https://api.github.com/users/sthomp/orgs","repos_url":"https://api.github.com/users/sthomp/repos","events_url":"https://api.github.com/users/sthomp/events{/privacy}","received_events_url":"https://api.github.com/users/sthomp/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-04-13T22:44:40Z","updated_at":"2016-04-18T16:01:48Z","closed_at":null,"body":"I wanted to leave a note here to update [setup.py](https://github.com/GoogleCloudPlatform/DataflowPythonSDK/blob/master/google/cloud/dataflow/examples/complete/juliaset/setup.py) (or possibly provide more examples of setup.py?) with instructions on connecting to Cloud SQL. I hope this will save someone a few hours (days? :))\r\n\r\nConnecting to Cloud SQL requires use of the [Cloud SQL Proxy](https://cloud.google.com/sql/docs/sql-proxy) which needs to be installed via setup.py. I thought this would be as straight forward as:\r\n\r\n```\r\nCUSTOM_COMMANDS = [\r\n    ['echo', 'Custom command worked!'],\r\n    ['wget', 'https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64'],\r\n    ['mv', 'cloud_sql_proxy.linux.amd64', 'cloud_sql_proxy'],\r\n    ['chmod', '+x', 'cloud_sql_proxy'],\r\n    ['cloud_sql_proxy', '-dir=/cloudsql -instances=my-project:us-central1:mysql-instance=tcp:3307', '&']\r\n]\r\n```\r\n\r\nBut it turns out the ampersand `&` required to run the proxy in the background will cause distutils to freeze and thus cause the entire job to freeze. In this state, the job needs to be manually cancelled or it will just sit idle indefinitely. This is due to the underlying `subprocess.Popen(..., shell=False)`.\r\n\r\nTo fix this I added another parameter to `RunCustomCommand` so that shell=True could be passed in. And then called `RunCustomCommand` separately as follows:\r\n\r\n```\r\nself.RunCustomCommand('./cloud_sql_proxy -dir=/cloudsql -instances=my-project:us-central1:mysql-instance=tcp:3307 &', True)\r\n``` \r\n\r\nUpdate:\r\n\r\nSeems I jumped the gun on coming up with a solution to this problem. The core of the issue was more so related to the line `p.communicate()` and `p.returncode` in setup.py which will block and wait for the sql proxy to return. \r\n\r\nI've managed to get this all working in a local container by having a separate subprocess command in python: \r\n\r\n`subprocess.Popen(./cloud_sql_proxy -dir=/cloudsql -instances=my-project:us-central1:mysql-instance=tcp:3307, shell=True, preexec_fn=os.setpgrp)`\r\n\r\nHowever running this on dataflow causes easy_install to stall and puts the rest of the dataflow job in a hung state"},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/DataflowPythonSDK/issues/comments/211446210","html_url":"https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/9#issuecomment-211446210","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/DataflowPythonSDK/issues/9","id":211446210,"user":{"login":"silviulica","id":4011298,"avatar_url":"https://avatars.githubusercontent.com/u/4011298?v=3","gravatar_id":"","url":"https://api.github.com/users/silviulica","html_url":"https://github.com/silviulica","followers_url":"https://api.github.com/users/silviulica/followers","following_url":"https://api.github.com/users/silviulica/following{/other_user}","gists_url":"https://api.github.com/users/silviulica/gists{/gist_id}","starred_url":"https://api.github.com/users/silviulica/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/silviulica/subscriptions","organizations_url":"https://api.github.com/users/silviulica/orgs","repos_url":"https://api.github.com/users/silviulica/repos","events_url":"https://api.github.com/users/silviulica/events{/privacy}","received_events_url":"https://api.github.com/users/silviulica/received_events","type":"User","site_admin":false},"created_at":"2016-04-18T16:01:47Z","updated_at":"2016-04-18T16:01:47Z","body":"Robert is absolutely right. I am trying to find a solution here given the current reality :-)"}}