<p>I'm trying to use a container that contains a Java tool to do some DB migrations on a MySQL database in a Kubernetes Job.</p>
<p>When I run the container locally in Docker (using a MySQL container in the same network), the tool runs as expected.
And if I create a Pod using the container and set the command arguments to point to the <code>mysql</code> service running in the same namespace, it does as well.</p>
<p>But if I convert that Pod spec into a Job, the created container can not connect to the MySQL service anymore for some reason.</p>
<p>The container is based on <code>amazoncorretto:8-al2-jdk</code> and just copies the JAR to <code>/opt/</code>.</p>
<p>The MySQL DB is available through the <code>mysql</code> service in the cluster:</p>
<pre class="lang-bash prettyprint-override"><code>$ kubectl describe service mysql -n &lt;namespace&gt;
Name:              mysql
Namespace:         &lt;namespace&gt;
Labels:            app=mysql
Annotations:       &lt;none&gt;
Selector:          app=mysql
Type:              ClusterIP
IP Families:       &lt;none&gt;
IP:                &lt;ip&gt;
IPs:               &lt;ip&gt;
Port:              mysql  3306/TCP
TargetPort:        3306/TCP
Endpoints:         &lt;ip&gt;:3306
Session Affinity:  None
Events:            &lt;none&gt;
</code></pre>
<p>These are the specifications for the Pod:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Pod
metadata:
  name: java-tool-pod
spec:
  containers:
  - name: javatool
    image: &lt;registry&gt;/&lt;image-name&gt;:&lt;version&gt;
    command: [ &quot;/bin/sh&quot; ]
    args: [ &quot;-x&quot;, &quot;-c&quot;, &quot;/usr/bin/java -jar /opt/&lt;tool&gt;.jar \&quot;jdbc:mysql://mysql:3306/&lt;db&gt;\&quot; -u &lt;user&gt; -p&lt;password&gt;&quot; ]
  imagePullSecrets:
  - name: &lt;secret&gt;
</code></pre>
<p>Running the Container as a Pod:</p>
<pre class="lang-bash prettyprint-override"><code>$ kubectl apply -f /tmp/as-pod.yaml -n &lt;namespace&gt;
pod/java-tool-pod created

$ kubectl logs pod/java-tool-pod -n &lt;namespace&gt;
+ /usr/bin/java -jar /opt/&lt;tool&gt;.jar jdbc:mysql://mysql:3306/&lt;db&gt; -u &lt;user&gt; -p&lt;password&gt;
DB Migration Tool
Database Schema, 3.30.0.3300024390, built Wed Jul 14 12:13:52 UTC 2021
Driver class: com.mysql.jdbc.Driver
INFO Flyway 3.2.1 by Boxfuse
INFO Database: jdbc:mysql://mysql:3306/&lt;db&gt; (MySQL 5.7)
INFO Validated 721 migrations (execution time 00:00.253s)
INFO Current version of schema `&lt;db&gt;`: 3.29.0.10859.10
WARN outOfOrder mode is active. Migration of schema `&lt;db&gt;` may not be reproducible.
INFO Schema `&lt;db&gt;` is up to date. No migration necessary.
</code></pre>
<p>These are the specifications for the Job:</p>
<pre class="lang-yaml prettyprint-override"><code>$ cat /tmp/as-job.yaml 
apiVersion: batch/v1
kind: Job
metadata:
  name: javatool-job
spec:
  template:
    spec:
      containers:
      - name: javatool
        image: &lt;registry&gt;/&lt;image-name&gt;:&lt;version&gt;
        command: [ &quot;/bin/sh&quot; ]
        args: [ &quot;-x&quot;, &quot;-c&quot;, &quot;/usr/bin/java -jar /opt/&lt;tool&gt;.jar \&quot;jdbc:mysql://mysql:3306/&lt;db&gt;\&quot; -u &lt;user -p&lt;password&gt;&quot; ]
      imagePullSecrets:
      - name: &lt;secret&gt;
      restartPolicy: Never
</code></pre>
<p>Running the container as a Job:</p>
<pre class="lang-bash prettyprint-override"><code>$ kubectl apply -f /tmp/as-job.yaml -n &lt;namespace&gt;
job.batch/javatool-job created

$ kubectl logs job.batch/javatool-job -n &lt;namespace&gt;
+ /usr/bin/java -jar /opt/&lt;tool&gt;.jar jdbc:mysql://mysql:3306/&lt;db&gt; -u &lt;user&gt; -p&lt;password&gt;
DB Migration Tool
Database Schema, 3.30.0.3300024390, built Wed Jul 14 12:13:52 UTC 2021
Driver class: com.mysql.jdbc.Driver
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
    at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:983)
    at com.mysql.jdbc.MysqlIO.&lt;init&gt;(MysqlIO.java:339)
    at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2252)
    at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2285)
    at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2084)
    at com.mysql.jdbc.ConnectionImpl.&lt;init&gt;(ConnectionImpl.java:795)
    at com.mysql.jdbc.JDBC4Connection.&lt;init&gt;(JDBC4Connection.java:44)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
    at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:400)
    at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:327)
    at java.sql.DriverManager.getConnection(DriverManager.java:664)
    at java.sql.DriverManager.getConnection(DriverManager.java:208)
    at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriverManager(DriverManagerDataSource.java:173)
    at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:164)
    at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver(AbstractDriverBasedDataSource.java:153)
    at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnection(AbstractDriverBasedDataSource.java:119)
    at com.nordija.itv.db.FlywayMigrationSchemaData.isNotFlywaySchemaVersion(FlywayMigrationSchemaData.java:58)
[...]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:607)
    at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:214)
    at com.mysql.jdbc.MysqlIO.&lt;init&gt;(MysqlIO.java:298)
    ... 22 more
INFO Flyway 3.2.1 by Boxfuse
Unable to obtain Jdbc connection from DataSource
[...]

</code></pre>
<p>I haven't seen any significant differences in the containers being created.
The only thing I can think of is some kind of character encoding issue, but I don't see why that should only occur in a Pod that was created for a Job and not in one that was created directly.</p>
<p>Thanks in advance for any help with this issue!</p>
<p><strong>Edit:</strong> I forgot to mention that Istio is active on the Namespace, which turned out to be causing the issues.</p>

wtfc63
<p>The problem was that Istio doesn't play nice with Kubernetes Jobs (I forgot to mention that Istio is active on the Namespace, sorry).</p>
<p>Once I added a short delay (<code>sleep 5</code> before starting the Java tool), the connection could be established.</p>
<p>But then I had another issue: After the container terminated successfully, the Job would not be marked as completed.</p>
<p>And the reason was again Istio.
Jobs are considered complete once <strong>all</strong> Pods are terminated and the Istio Sidecar is a service Pod that doesn't terminate.
After finding <a href="https://medium.com/redbox-techblog/handling-istio-sidecars-in-kubernetes-jobs-c392661c4af7" rel="nofollow noreferrer">this article</a>, I ended up integrating their <a href="https://github.com/redboxllc/scuttle" rel="nofollow noreferrer"><code>scuttle</code></a> tool into the container and now the Job can be completed successfully.</p>
