title: Spark JDBC connection to SQL Server times out often
body: <p>I'm running Spark v2.2.1 via sparklyr v0.6.2 and pulling data from SQL Server via jdbc. I seem to be experiencing some network issue because <em>many times</em> (not every time) my executor doing a write to SQL Server fails with error:</p>
  
  <pre><code>Prelogin error: host &lt;my server&gt; port 1433 Error reading prelogin response: Connection timed out (Read failed) ClientConnectionId:...
  </code></pre>
  
  <p>I am running my sparklyr session with the following configurations:</p>
  
  <pre><code>spark_conf = spark_config()
  spark_conf$spark.executor.cores &lt;- 8
  spark_conf$`sparklyr.shell.driver-memory` &lt;- "8G"
  spark_conf$`sparklyr.shell.executor-memory` &lt;- "12G"
  spark_conf$spark.serializer &lt;- "org.apache.spark.serializer.KryoSerializer"
  spark_conf$spark.network.timeout &lt;- 400
  </code></pre>
  
  <p>But interestingly the network timeout I've set above does not seem to apply based on the executor logs:</p>
  
  <pre><code>18/06/11 17:53:44 INFO BlockManager: Found block rdd_9_16 locally
  18/06/11 17:53:45 WARN SQLServerConnection: ConnectionID:3 ClientConnectionId: d3568a9f-049f-4772-83d4-ed65b907fc8b Prelogin error: host nciensql14.nciwin.local port 1433 Error reading prelogin response: Connection timed out (Read failed) ClientConnectionId:d3568a9f-049f-4772-83d4-ed65b907fc8b
  18/06/11 17:53:45 WARN SQLServerConnection: ConnectionID:2 ClientConnectionId: ecb084e6-99a8-49d1-9215-491324e8d133 Prelogin error: host nciensql14.nciwin.local port 1433 Error reading prelogin response: Connection timed out (Read failed) ClientConnectionId:ecb084e6-99a8-49d1-9215-491324e8d133
  18/06/11 17:53:45 ERROR Executor: Exception in task 10.0 in stage 26.0 (TID 77)
  </code></pre>
  
  <p>Can someone help me understand what a prelogin error is and how to avoid this issue? Here is my write function:</p>
  
  <pre><code>function (df, tbl, db, server = NULL, user, pass, mode = "error", 
      options = list(), ...) 
  {
      sparklyr::spark_write_jdbc(
    df, 
    tbl, 
    options = c(
      list(url = paste0("jdbc:sqlserver://", server, ".nciwin.local;", 
           "databaseName=", db, ";", 
           "user=", user, ";", 
           "password=", pass, ";"), 
         driver = "com.microsoft.sqlserver.jdbc.SQLServerDriver"), 
      options), 
    mode = mode, ...)
  }
  </code></pre>
  
  <p>I've just updated my jdbc driver to version 6.0, but I don't think it made a difference. I hope i installed it correctly. I just dropped it into my <code>Spark/jars</code> folder and then added it into <code>Spark/conf/spark-defaults.conf</code>.</p>
  
  <p><strong>EDIT</strong> 
  I am reading in 23M rows in 24 partitions into Spark. My cluster has 4 nodes with 8 cores each and 18G memory. With my current configurations I have 4 executors with 8 cores each and 12G per executor. My function to read in the data looks as such:</p>
  
  <pre><code>function (sc, tbl, db, server = NULL, user, pass, repartition = 0, options = list(), ...) 
  {
      sparklyr::spark_read_jdbc(
        sc, 
        tbl, 
        options = c(
          list(url = paste0("jdbc:sqlserver://", server, ".nciwin.local;"), 
               user = user, 
               password = pass, 
               databaseName = db, 
               dbtable = tbl, 
               driver = "com.microsoft.sqlserver.jdbc.SQLServerDriver"), 
          options), 
        repartition = repartition, ...)
  }
  </code></pre>
  
  <p>I set <code>repartition</code> to 24 when running. As such, I'm not seeing the connection with the post suggested.</p>
  
  <p><strong>EDIT 2</strong><br>
  I was able to fix my issue by getting rid of repartitioning. Can anyone explain why repartitioning with sparklyr is not effective in this case?</p>