{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/cloud-bigtable-client/issues/630","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/cloud-bigtable-client/issues/630/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/cloud-bigtable-client/issues/630/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/cloud-bigtable-client/issues/630/events","html_url":"https://github.com/GoogleCloudPlatform/cloud-bigtable-client/issues/630","id":127512143,"number":630,"title":"DataFlow CloudBigtableIO leaks threads","user":{"login":"danielnorberg","id":393289,"avatar_url":"https://avatars.githubusercontent.com/u/393289?v=3","gravatar_id":"","url":"https://api.github.com/users/danielnorberg","html_url":"https://github.com/danielnorberg","followers_url":"https://api.github.com/users/danielnorberg/followers","following_url":"https://api.github.com/users/danielnorberg/following{/other_user}","gists_url":"https://api.github.com/users/danielnorberg/gists{/gist_id}","starred_url":"https://api.github.com/users/danielnorberg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielnorberg/subscriptions","organizations_url":"https://api.github.com/users/danielnorberg/orgs","repos_url":"https://api.github.com/users/danielnorberg/repos","events_url":"https://api.github.com/users/danielnorberg/events{/privacy}","received_events_url":"https://api.github.com/users/danielnorberg/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2016-01-19T18:48:34Z","updated_at":"2016-01-25T20:17:15Z","closed_at":null,"body":"We have a DataFlow job that's structurally very similar to the [WordCount](https://cloud.google.com/dataflow/examples/wordcount-example#UnboundedData) example, reading from pubsub, producing per X minute windowed counts that are written to CBT.\r\n\r\nWe're using `0.2.3-SNAPSHOT` of `bigtable-hbase-dataflow` and `bigtable-hbase-1.1` in order to work around https://github.com/GoogleCloudPlatform/cloud-bigtable-client/issues/613.\r\n\r\nWhen deploying our job late last week we discovered that it was failing with the DF workers unable to create more threads.\r\n\r\n```\r\njava.lang.OutOfMemoryError: unable to create new native thread\r\n\tat java.lang.Thread.start0(Native Method)\r\n\tat java.lang.Thread.start(Thread.java:714)\r\n\tat java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)\r\n\tat java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)\r\n\tat java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)\r\n\tat com.google.cloud.bigtable.hbase.BigtableBufferedMutator.initializeAsyncMutators(BigtableBufferedMutator.java:206)\r\n\tat com.google.cloud.bigtable.hbase.BigtableBufferedMutator.<init>(BigtableBufferedMutator.java:201)\r\n\tat com.google.cloud.bigtable.hbase.BigtableBufferedMutator.<init>(BigtableBufferedMutator.java:174)\r\n\tat org.apache.hadoop.hbase.client.AbstractBigtableConnection$2.<init>(AbstractBigtableConnection.java:193)\r\n\tat org.apache.hadoop.hbase.client.AbstractBigtableConnection.getBufferedMutator(AbstractBigtableConnection.java:192)\r\n\tat com.google.cloud.bigtable.dataflow.CloudBigtableIO$CloudBigtableSingleTableWriteFn.getBufferedMutator(CloudBigtableIO.java:615)\r\n\tat com.google.cloud.bigtable.dataflow.CloudBigtableIO$CloudBigtableSingleTableWriteFn.processElement(CloudBigtableIO.java:640)\r\n```\r\n\r\nWe SSH'd to one of the DF worker hosts and saw that the worker JVM process had some 32k+ threads. We attempted to redeploy our job a few times, each time hitting the same issue. \r\n\r\nThe same job deployed the week before had functioned normally so we figured that maybe this issue was caused by some recent change to the CBT client and redeployed our job with an earlier CBT client commit (6cefa8195724eefc2a58428597e68ccdb9513482) instead, which works fine. \r\n\r\nLooking at the recent commit log I wonder if maybe 1433a527a192694b13862776500576e73126219b could be related to this.\r\n\r\nWe have not attempted to repro outside DataFlow."},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/cloud-bigtable-client/issues/comments/174648432","html_url":"https://github.com/GoogleCloudPlatform/cloud-bigtable-client/issues/630#issuecomment-174648432","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/cloud-bigtable-client/issues/630","id":174648432,"user":{"login":"sduskis","id":99305,"avatar_url":"https://avatars.githubusercontent.com/u/99305?v=3","gravatar_id":"","url":"https://api.github.com/users/sduskis","html_url":"https://github.com/sduskis","followers_url":"https://api.github.com/users/sduskis/followers","following_url":"https://api.github.com/users/sduskis/following{/other_user}","gists_url":"https://api.github.com/users/sduskis/gists{/gist_id}","starred_url":"https://api.github.com/users/sduskis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sduskis/subscriptions","organizations_url":"https://api.github.com/users/sduskis/orgs","repos_url":"https://api.github.com/users/sduskis/repos","events_url":"https://api.github.com/users/sduskis/events{/privacy}","received_events_url":"https://api.github.com/users/sduskis/received_events","type":"User","site_admin":false},"created_at":"2016-01-25T20:17:15Z","updated_at":"2016-01-25T20:17:15Z","body":"Thanks for your patience with this.  I just built a -SNAPSHOT that should fix the issue related to this problem:\r\n\r\n```\r\n   java.lang.Thread.State: BLOCKED (on object monitor)\r\n    at java.util.zip.ZipFile.getEntry(ZipFile.java:308)\r\n    - waiting to lock <0x0000000083309e70> (a java.util.jar.JarFile)\r\n    at java.util.jar.JarFile.getEntry(JarFile.java:240)\r\n    at java.util.jar.JarFile.getJarEntry(JarFile.java:223)\r\n    at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1005)\r\n   ...\r\n```\r\n\r\nI'm not sure why we're creating so many connections.  I'd like to figure out why that's happening.\r\n\r\nIn the meantime, I'll put in a fix to share threading resources across connections, since the thread pools auto-expand anyway."}}