title: GCP Dataflow - SSLHandshakeException
body: <p>I'm encountering the following when running large (>1000 cpus) and medium sized (100-1000 cpus) dataflow jobs:</p>
  
  <blockquote>
    <p>exception:  "javax.net.ssl.SSLHandshakeException: Remote host closed
    connection during handshake</p>
  </blockquote>
  
  <p>The error is not fatal, however, once it begins it recurs roughly every 30 seconds. Jobs that display this error never finish (I've waited over 4x the expected run time) and produce very limited results (less than 4% of the expected output). The limited output, when produced, is received early in the job, and afterwards no more is produced.</p>
  
  <p>I use both BigQueryIO and JdbcIO Apache beam sources and sinks.</p>
  
  <p>It is important to note that my jobs worked correctly in early June, but have begun showing this error since the beginning of July.</p>
  
  <p>I have an open case in Google's Enterprise Support, but let's just say results are not forthcoming. The only point of interest that has been yielded from Google is that the error might occur, "in case the workers scale up and heavily access Cloud Storage".</p>
  
  <p>However, there is no solution attached to that statement.
  This is an example of the complete error as recorded in the logs:</p>
  
  <pre><code>   exception:  "javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
      at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002)
      at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
      at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
      at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
      at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
      at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
      at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
      at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:93)
      at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
      at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
      at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
      at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
      at com.google.cloud.hadoop.util.ResilientOperation$AbstractGoogleClientRequestExecutor.call(ResilientOperation.java:166)
      at com.google.cloud.hadoop.util.ResilientOperation.retry(ResilientOperation.java:66)
      at com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.getMetadata(GoogleCloudStorageReadChannel.java:573)
      at com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.openStreamAndSetMetadata(GoogleCloudStorageReadChannel.java:645)
      at com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.performLazySeek(GoogleCloudStorageReadChannel.java:560)
      at com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.read(GoogleCloudStorageReadChannel.java:289)
      at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
      at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
      at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
      at java.io.InputStream.read(InputStream.java:101)
      at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:81)
      at org.apache.beam.sdk.util.VarInt.decodeLong(VarInt.java:79)
      at org.apache.beam.sdk.util.VarInt.decodeInt(VarInt.java:63)
      at org.apache.beam.runners.dataflow.internal.IsmFormat$KeyPrefixCoder.decode(IsmFormat.java:709)
      at com.google.cloud.dataflow.worker.runners.worker.IsmReader.readKey(IsmReader.java:1001)
      at com.google.cloud.dataflow.worker.runners.worker.IsmReader.access$2000(IsmReader.java:79)
      at com.google.cloud.dataflow.worker.runners.worker.IsmReader$WithinShardIsmReaderIterator.advance(IsmReader.java:953)
      at com.google.cloud.dataflow.worker.runners.worker.IsmReader$WithinShardIsmReaderIterator.start(IsmReader.java:943)
      at com.google.cloud.dataflow.worker.runners.worker.IsmReader$IsmCacheLoader.call(IsmReader.java:581)
      at com.google.cloud.dataflow.worker.runners.worker.IsmReader$IsmCacheLoader.call(IsmReader.java:570)
      at com.google.cloud.dataflow.worker.runners.worker.IsmReader$IsmCacheLoader.call(IsmReader.java:555)
      at com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4904)
      at com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3628)
      at com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2336)
      at com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2295)
      at com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2208)
      at com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache.get(LocalCache.java:4053)
      at com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4899)
      at com.google.cloud.dataflow.worker.runners.worker.IsmReader.fetch(IsmReader.java:606)
      at com.google.cloud.dataflow.worker.runners.worker.IsmReader.getBlock(IsmReader.java:771)
      at com.google.cloud.dataflow.worker.runners.worker.IsmReader.access$1000(IsmReader.java:79)
      at com.google.cloud.dataflow.worker.runners.worker.IsmReader$IsmPrefixReaderIterator.get(IsmReader.java:642)
      at com.google.cloud.dataflow.worker.runners.worker.IsmSideInputReader$ListOverReaderIterators.getUsingLong(IsmSideInputReader.java:679)
      at com.google.cloud.dataflow.worker.runners.worker.IsmSideInputReader$ListOverReaderIterators.access$1300(IsmSideInputReader.java:625)
      at com.google.cloud.dataflow.worker.runners.worker.IsmSideInputReader$ListOverReaderIterators$ListIteratorOverReaderIterators.next(IsmSideInputReader.java:720)
      at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1042)
      at com.application.strategy.simulator.MainStrategySimulator$1.processElement(MainStrategySimulator.java:224)
      at com.application.strategy.simulator.MainStrategySimulator$1$auxiliary$4N23tth9.invokeProcessElement(Unknown Source)
      at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:199)
      at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:157)
      at com.google.cloud.dataflow.worker.runners.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:244)
      at com.google.cloud.dataflow.worker.runners.worker.ForwardingParDoFn.processElement(ForwardingParDoFn.java:42)
      at com.google.cloud.dataflow.worker.runners.worker.DataflowWorkerLoggingParDoFn.processElement(DataflowWorkerLoggingParDoFn.java:47)
      at com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:48)
      at com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:52)
      at com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:198)
      at com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
      at com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:72)
      at com.google.cloud.dataflow.worker.runners.worker.DataflowWorker.executeWork(DataflowWorker.java:336)
      at com.google.cloud.dataflow.worker.runners.worker.DataflowWorker.doWork(DataflowWorker.java:295)
      at com.google.cloud.dataflow.worker.runners.worker.DataflowWorker.getAndPerformWork(DataflowWorker.java:242)
      at com.google.cloud.dataflow.worker.runners.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:123)
      at com.google.cloud.dataflow.worker.runners.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:103)
      at com.google.cloud.dataflow.worker.runners.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:90)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
  Caused by: java.io.EOFException: SSL peer shut down incorrectly
      at sun.security.ssl.InputRecord.read(InputRecord.java:505)
      at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
      ... 69 more
  </code></pre>