{"action":"created","issue":{"url":"https://api.github.com/repos/grpc/grpc/issues/13873","repository_url":"https://api.github.com/repos/grpc/grpc","labels_url":"https://api.github.com/repos/grpc/grpc/issues/13873/labels{/name}","comments_url":"https://api.github.com/repos/grpc/grpc/issues/13873/comments","events_url":"https://api.github.com/repos/grpc/grpc/issues/13873/events","html_url":"https://github.com/grpc/grpc/issues/13873","id":284265142,"number":13873,"title":"gRPC Python 1.8.2 Library incompatibility with fork()","user":{"login":"zunger-humu","id":31084695,"avatar_url":"https://avatars0.githubusercontent.com/u/31084695?v=4","gravatar_id":"","url":"https://api.github.com/users/zunger-humu","html_url":"https://github.com/zunger-humu","followers_url":"https://api.github.com/users/zunger-humu/followers","following_url":"https://api.github.com/users/zunger-humu/following{/other_user}","gists_url":"https://api.github.com/users/zunger-humu/gists{/gist_id}","starred_url":"https://api.github.com/users/zunger-humu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zunger-humu/subscriptions","organizations_url":"https://api.github.com/users/zunger-humu/orgs","repos_url":"https://api.github.com/users/zunger-humu/repos","events_url":"https://api.github.com/users/zunger-humu/events{/privacy}","received_events_url":"https://api.github.com/users/zunger-humu/received_events","type":"User","site_admin":false},"labels":[{"id":168238186,"url":"https://api.github.com/repos/grpc/grpc/labels/lang/Python","name":"lang/Python","color":"fad8c7","default":false}],"state":"open","locked":false,"assignee":{"login":"kpayson64","id":18316330,"avatar_url":"https://avatars2.githubusercontent.com/u/18316330?v=4","gravatar_id":"","url":"https://api.github.com/users/kpayson64","html_url":"https://github.com/kpayson64","followers_url":"https://api.github.com/users/kpayson64/followers","following_url":"https://api.github.com/users/kpayson64/following{/other_user}","gists_url":"https://api.github.com/users/kpayson64/gists{/gist_id}","starred_url":"https://api.github.com/users/kpayson64/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kpayson64/subscriptions","organizations_url":"https://api.github.com/users/kpayson64/orgs","repos_url":"https://api.github.com/users/kpayson64/repos","events_url":"https://api.github.com/users/kpayson64/events{/privacy}","received_events_url":"https://api.github.com/users/kpayson64/received_events","type":"User","site_admin":false},"assignees":[{"login":"kpayson64","id":18316330,"avatar_url":"https://avatars2.githubusercontent.com/u/18316330?v=4","gravatar_id":"","url":"https://api.github.com/users/kpayson64","html_url":"https://github.com/kpayson64","followers_url":"https://api.github.com/users/kpayson64/followers","following_url":"https://api.github.com/users/kpayson64/following{/other_user}","gists_url":"https://api.github.com/users/kpayson64/gists{/gist_id}","starred_url":"https://api.github.com/users/kpayson64/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kpayson64/subscriptions","organizations_url":"https://api.github.com/users/kpayson64/orgs","repos_url":"https://api.github.com/users/kpayson64/repos","events_url":"https://api.github.com/users/kpayson64/events{/privacy}","received_events_url":"https://api.github.com/users/kpayson64/received_events","type":"User","site_admin":false}],"milestone":null,"comments":16,"created_at":"2017-12-22T22:49:49Z","updated_at":"2018-02-05T23:23:40Z","closed_at":null,"author_association":"NONE","body":"This bug is essentially the same as the previously-closed [bug 12455](https://github.com/grpc/grpc/issues/12455). That bug was closed because it was fixed in 1.7.0... except we now have at least two ways to repro it in 1.8.2. :(\r\n\r\nBasic summary of the bug: If a gRPC channel is (1) used at least once and (2) still in scope when you call fork(), and the subprocess tries to open a gRPC channel as well, then all RPC's on that channel will hang. Apparently, some global state is leaking across fork() boundaries in a bad way.\r\n\r\n### What version of gRPC and what language are you using?\r\n \r\ngRPC 1.8.2, Python 3.6\r\n \r\n### What operating system (Linux, Windows, …) and version?\r\n \r\nRepro'ed on OS X 10.13.2\r\n \r\n### What runtime / compiler are you using (e.g. python version or version of gcc)\r\n\r\nPython 3.6.2\r\n \r\n### What did you do?\r\n\r\nHere is a minimal repro using the GCP datastore client:\r\n\r\n```\r\nimport multiprocessing\r\n\r\nfrom google.cloud import datastore\r\n\r\n\r\ndef causeTrouble(where: str):\r\n    client = datastore.Client(project='dev-storage-humu', namespace='aquarium')\r\n    client.get(client.key('c', 'aquarium'))\r\n    # The call to get() hangs forever; this line is never reached.\r\n    print('OK')\r\n\r\n\r\nif __name__ == '__main__':\r\n    # Create a datastore client and do an RPC on it.\r\n    client = datastore.Client(project='dev-storage-humu', namespace='aquarium')\r\n    client.get(client.key('c', 'aquarium'))\r\n\r\n    # Kick off a child process while the first client is still in scope.\r\n    process = multiprocessing.Process(target=causeTrouble,\r\n                                      args=['child process'])\r\n    process.start()\r\n```\r\n\r\n- If you change this so that it doesn't call client.get() from main, just creating the client and then forking, it works fine.\r\n- If you change this so that instead of creating the client and calling client.get() directly, it calls causeTrouble() before forking, it also works fine. (NB that Python has eager GC, so the difference between the two is that the client is no longer in scope if you do it this other way)\r\n- In another server, I worked around this by calling subprocess.run() instead of multiprocessing.Process(); since subprocess() does a fork() + exec(), it has no state surviving across the process boundary. However, the call to exec() strictly limits the communication with the parent process, basically making all of the multiprocessing library unusable.\r\n\r\nIf you kill it while it's hanging, you get this stack trace:\r\n\r\n```\r\nTraceback (most recent call last):\r\n  File \"/Users/zunger/.pyenv/versions/3.6.2/lib/python3.6/multiprocessing/process.py\", line 249, in _bootstrap\r\n    self.run()\r\n  File \"/Users/zunger/.pyenv/versions/3.6.2/lib/python3.6/multiprocessing/process.py\", line 93, in run\r\n    self._target(*self._args, **self._kwargs)\r\n  File \"minimal_repro.py\", line 8, in causeTrouble\r\n    client.get(client.key('c', 'aquarium'))\r\n  File \"/Users/zunger/src/humu/servers/squeegee/server/.build/venv/lib/python3.6/site-packages/google/cloud/datastore/client.py\", line 309, in get\r\n    deferred=deferred, transaction=transaction)\r\n  File \"/Users/zunger/src/humu/servers/squeegee/server/.build/venv/lib/python3.6/site-packages/google/cloud/datastore/client.py\", line 356, in get_multi\r\n    transaction_id=transaction and transaction.id,\r\n  File \"/Users/zunger/src/humu/servers/squeegee/server/.build/venv/lib/python3.6/site-packages/google/cloud/datastore/client.py\", line 138, in _extended_lookup\r\n    project, read_options, key_pbs)\r\n  File \"/Users/zunger/src/humu/servers/squeegee/server/.build/venv/lib/python3.6/site-packages/google/cloud/datastore/_gax.py\", line 115, in lookup\r\n    return super(GAPICDatastoreAPI, self).lookup(*args, **kwargs)\r\n  File \"/Users/zunger/src/humu/servers/squeegee/server/.build/venv/lib/python3.6/site-packages/google/cloud/gapic/datastore/v1/datastore_client.py\", line 204, in lookup\r\n    return self._lookup(request, options)\r\n  File \"/Users/zunger/src/humu/servers/squeegee/server/.build/venv/lib/python3.6/site-packages/google/gax/api_callable.py\", line 452, in inner\r\n    return api_caller(api_call, this_settings, request)\r\n  File \"/Users/zunger/src/humu/servers/squeegee/server/.build/venv/lib/python3.6/site-packages/google/gax/api_callable.py\", line 438, in base_caller\r\n    return api_call(*args)\r\n  File \"/Users/zunger/src/humu/servers/squeegee/server/.build/venv/lib/python3.6/site-packages/google/gax/api_callable.py\", line 376, in inner\r\n    return a_func(*args, **kwargs)\r\n  File \"/Users/zunger/src/humu/servers/squeegee/server/.build/venv/lib/python3.6/site-packages/google/gax/retry.py\", line 121, in inner\r\n    return to_call(*args)\r\n  File \"/Users/zunger/src/humu/servers/squeegee/server/.build/venv/lib/python3.6/site-packages/google/gax/retry.py\", line 68, in inner\r\n    return a_func(*updated_args, **kwargs)\r\n  File \"/Users/zunger/src/humu/servers/squeegee/server/.build/venv/lib/python3.6/site-packages/grpc/_channel.py\", line 484, in __call__\r\n    credentials)\r\n  File \"/Users/zunger/src/humu/servers/squeegee/server/.build/venv/lib/python3.6/site-packages/grpc/_channel.py\", line 478, in _blocking\r\n    _handle_event(completion_queue.poll(), state,\r\n  File \"src/python/grpcio/grpc/_cython/_cygrpc/completion_queue.pyx.pxi\", line 100, in grpc._cython.cygrpc.CompletionQueue.poll\r\n```\r\n\r\nOn b/12455, @katbusch reported that simply _importing_ certain libraries prior to fork (e.g. `from google.cloud import bigquery`) was sufficient to trigger this bug. Presumably they do enough initialization on import to trigger this.\r\n\r\nThat makes workarounds where we pre-fork a pile of worker processes and pass them jobs as needed more difficult, although possible as a short-term measure.\r\n \r\n### Anything else we should know about your project / environment?\r\n\r\nThe use case which necessitates this is that we have a server (which uses GCP features extensively, e.g. for storage) that needs to fork off subprocesses in which to run long-running operations. (It's basically an analysis pipeline manager) Since Python is heavily reliant on fork() for parallelization (thanks to the GIL there's no real 'parallelism' in its threading) this is the main approach possible."},"comment":{"url":"https://api.github.com/repos/grpc/grpc/issues/comments/363255730","html_url":"https://github.com/grpc/grpc/issues/13873#issuecomment-363255730","issue_url":"https://api.github.com/repos/grpc/grpc/issues/13873","id":363255730,"user":{"login":"remram44","id":426784,"avatar_url":"https://avatars3.githubusercontent.com/u/426784?v=4","gravatar_id":"","url":"https://api.github.com/users/remram44","html_url":"https://github.com/remram44","followers_url":"https://api.github.com/users/remram44/followers","following_url":"https://api.github.com/users/remram44/following{/other_user}","gists_url":"https://api.github.com/users/remram44/gists{/gist_id}","starred_url":"https://api.github.com/users/remram44/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/remram44/subscriptions","organizations_url":"https://api.github.com/users/remram44/orgs","repos_url":"https://api.github.com/users/remram44/repos","events_url":"https://api.github.com/users/remram44/events{/privacy}","received_events_url":"https://api.github.com/users/remram44/received_events","type":"User","site_admin":false},"created_at":"2018-02-05T23:23:40Z","updated_at":"2018-02-05T23:23:40Z","author_association":"NONE","body":"Oh my god what is this nonsense\r\n\r\nWhere is this documented?\r\n\r\nDoes the environment variable work with other versions or only `1.8.1`?"}}
{"actor":{"display_login":"remram44"}}