<p>I have a K8S CronJob that uploads files to a remote SFTP server. To ensure that the Job does not run to completion when an upload error occurs I added error handling logic. The logic however is giving an error as below:</p>
<pre><code>Starting-BACKUP
2023-09-27T07:39:01.384+0000    reading password from standard input
Enter password for mongo user:
/configmap/backup.sh: line 18: warning: here-document at line 9 delimited by end-of-file (wanted `!')
/configmap/backup.sh: line 19: syntax error: unexpected end of file
Database backup was successful
</code></pre>
<p>The error I am trying to handle occurs when the SFTP server is full on capacity. In such cases, the job should ideally not run to completion and the associated container should fail with exit code 1. When an upload is unsuccessful because of out-of-space issues the Job 's log output looks as below:</p>
<pre><code>Starting-BACKUP
2023-09-26T13:25:01.406+0000    reading password from standard input
Enter password for mongo user:
Database backup was successful
Warning: Permanently added '[ftp.mycompany.com]:2020,[10.XX.XXX.XXX]:xxxx' (RSA) to the list of known hosts.     
Connected to ftp.mycompany.com.
Changing to: /oasys
sftp&gt;   put labs-instance_20230926132501.gzip
Uploading labs-instance_20230926132501.gzip to /oasys/labs-instance_20230926132501.gzip
remote open(&quot;/oasys/labs-instance_20230926132501.gzip&quot;): Failure
sftp&gt;   bye
BackUp file labs-instance_20230926132501.gzip was successfully transfered
BackUp file labs-instance_20230926132501.gzip was deleted
</code></pre>
<p>Currently the job runs to completion despite the actual upload being unsuccessful:</p>
<pre><code>NAME                       COMPLETIONS   DURATION   AGE
labs-backupjob-28262274    1/1           15s        7m42s
</code></pre>
<p>The pod created by the job also has a final status of <code>Completed</code>:</p>
<pre><code>NAME                                   READY   STATUS      RESTARTS         AGE
labs-backupjob-28262286--1-s6g72       0/1     Completed   0                38s
</code></pre>
<p>The CronJob and bash script are given below :</p>
<pre><code> apiVersion: v1
    kind: ConfigMap
    metadata:
      name: backup-config
data:
  backup.sh: |-
    #!/bin/bash 
    set -e  # Exit on error

    echo &quot;Starting-BACKUP&quot;
    TIMESTAMP=$(date +&quot;%Y%m%d%H%M%S&quot;)
    BACKUP_FILE_NAME=&quot;labs-instance_$TIMESTAMP.gzip&quot;
    echo $MONGODB_ROOT_PASSWORD | mongodump --uri=&quot;mongodb://mongodb-prom:27017&quot; --username=&quot;root&quot; --authenticationDatabase=&quot;admin&quot; --gzip --archive=&quot;$BACKUP_FILE_NAME&quot; --quiet 
    echo &quot;Database backup was successful&quot;
    if ! sshpass -e sftp -o StrictHostKeyChecking=no -P XXXX oasys@mycompany.com:/oasys &lt;&lt; !
    put $BACKUP_FILE_NAME
    bye
    !; then
      echo &quot;Error uploading file to SFTP server&quot;
      exit 1 
    fi
    echo &quot;BackUp file $BACKUP_FILE_NAME was successfully transfered&quot;
    rm $BACKUP_FILE_NAME
    echo &quot;BackUp file $BACKUP_FILE_NAME was deleted&quot;    

---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: labs-backupjob
spec:
  schedule: &quot;*/3 * * * *&quot;
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: mongodb-backup
            image: registry.mycompany.com/oasys/devops/docker-images/mongosh:6.0
            imagePullPolicy: Always            
            command: [&quot;/configmap/backup.sh&quot;]
            env:
            - name: MONGODB_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mongodb-labs
                  key: mongodb-root-password
            - name: SSHPASS
              valueFrom:
                secretKeyRef:
                  name: sftp-password
                  key: sftp-backup-password
            volumeMounts:
                - name: backup-volume
                  mountPath: /configmap
          volumes:
            - name: backup-volume
              configMap:
                name: backup-config    
                defaultMode: 0777
          restartPolicy: Never
          imagePullSecrets:
            - name: regcred
      backoffLimit: 0
</code></pre>
<p>What am I missing?</p>

Golide
<p>Here-docs can be painful, especially if you need to maintain indentation. You've used it as the test condition, which is causing issues. <code>!; then</code> confuses the parser, at least on my test systems.</p>
<pre><code>$: cat tst
#!/bin/bash

if cat &lt;&lt; !
foo 1
!
then echo ok on 1; else echo no on 1; fi

cat &lt;&lt; !
foo 2
!
if [[ 0 == $? ]]; then echo ok on 2; else echo no on 2; fi

if cat &lt;&lt; !
foo 3
!; then echo ok on 3; else echo no on 3; fi

echo never gets here

$: ./tst
foo 1
ok on 1
foo 2
ok on 2
./tst: line 18: warning: here-document at line 13 delimited by end-of-file (wanted `!')
./tst: line 19: syntax error: unexpected end of file
</code></pre>
<p>I strongly suggest using <code>scp</code> rather than any <code>ftp</code>, even <code>sftp</code>. I do this a lot, like <a href="https://stackoverflow.com/questions/66474845/bash-ftp-upload-events-to-log/66476731#66476731">here</a> and <a href="https://stackoverflow.com/questions/58956793/how-to-connect-to-a-ftp-server-via-bash-script/58977040#58977040">over here</a> and <a href="https://stackoverflow.com/questions/52945672/provide-commands-automatically-to-ftp-in-bash-script/52953316#52953316">also here</a>. Virtually every flavor of CLI <code>ftp</code> <em>expects to be interactive</em>. It assumes you are looking at the screen and will handle whatever goes wrong, so it generally doesn't return an error to the OS if one of your commands fails.</p>
<p>If you can set up public-key trust it's <strong>SO</strong> much easier...</p>
<pre><code>    if ! scp &quot;$BACKUP_FILE_NAME&quot; oasys@mycompany.com:/oasys/
    then
      echo &quot;Error uploading file&quot;
      exit 1
    fi
</code></pre>
<p>and that works well in YAML, which can't use leading tabs, while here-docs can strip leading tab indents with <code>&lt;&lt;-</code> but can't handle leading spaces. <code>scp</code> reports any fail with an error code, and can be easily checked. Just make sure you get the right target path if it's different.</p>
<p>If public key trust isn't an option, see <a href="https://stackoverflow.com/questions/50096/how-to-pass-password-to-scp">this post</a> for alternatives, such as using <code>sshpass</code> (though that leaves the password on the CLI, so possibly in the history, the system logs, any job logs, etc...) or even <code>curl</code> (which lets you supply it on stdin).</p>
<p>Likewise, there are alternatives to here-docs, and ways to reuse and dynamically format your input list. Maybe some of these will be useful to someone -</p>
<pre><code>$: cat tst
#!/bin/bash

if cat; then echo &quot;&gt; literal ok&quot;; else echo no; fi &lt;&lt;&lt;&quot;maybe you prefer literals -
put ${BACKUP_FILE_NAME:-some_file}
bye&quot;;

# if reusing commands and/or concerned about whitespace, I like an array
cmds=(
  &quot;put ${BACKUP_FILE_NAME:-some_file}&quot;
  bye
)

if cat &lt;&lt;&lt;&quot;$( printf &quot;  %s\n&quot; &quot;${cmds[@]}&quot; )&quot;; then echo &quot;&gt; Indented command substitution ok&quot;; else echo no; fi

( IFS=$'\n'; if cat &lt;&lt;&lt;&quot;${cmds[*]^}&quot;; then echo &quot;&gt; Capitalized IFS hack ok&quot;; else echo no; fi; )

printf &quot;\t%s\n&quot; &quot;${cmds[@]}&quot; | if cat; then echo &quot;&gt; tabbed pipe ok&quot;; else echo no; fi

if cat; then echo &quot;&gt; bracketed Process substitution ok&quot;; else echo no; fi &lt; &lt;( printf &quot;[%s]\n&quot; &quot;${cmds[@]}&quot; )

if cat &lt;&lt; !
$( printf &quot;%s\n&quot; &quot;${cmds[@]^^}&quot; )
!
then echo &quot;&gt; here-doc up-cased command substitution ok&quot;; else echo no; fi

( IFS=$'\n'
if cat &lt;&lt; !
${cmds[*]}
!
then echo &quot;&gt; here-doc with IFS hack ok&quot;; else echo no; fi; )
</code></pre>
<p>which, when run, generates:</p>
<pre><code>$: ./tst
maybe you prefer literals -
put some_file
bye
&gt; literal ok
  put some_file
  bye
&gt; Indented command substitution ok
Put some_file
Bye
&gt; Capitalized IFS hack ok
        put some_file
        bye
&gt; tabbed pipe ok
[put some_file]
[bye]
&gt; bracketed Process substitution ok
PUT SOME_FILE
BYE
&gt; here-doc up-cased command substitution ok
put some_file
bye
&gt; here-doc with IFS hack ok
</code></pre>
