title: JobManager reprocesses already finished jobs, resulting in DuplicateJobSubmissionException and cluster corruption
body: <p>Flink JobManager appears to be resubmitting finished jobs and not ever reporting them as DONE to the client. In the JobManager log below, notice that GET requests to the BlobServer fail, then shortly thereafter duplicate job submissions start, and following that the FsJobArchivist failure to archive an already completed/archived job. This is repeatable behavior (happens several hours into a successfully running session cluster that has already handled a few hundred batch pipelines).  After this happens, the cluster will not recover to a point where it successfully processes jobs.  JobManager keeps executing pipelines (the same ones it has already processed), but since it doesn't think they finish, it just loops on them forever. It eventually grinds to a halt.  What could cause this behavior, and is there a configuration option that you could recommend here to alleviate the problem? There is 30% available RAM and 70% available disk space on the machine already.</p>
  <p>Background: running Flink 1.13.1 on EMR in a session mode,</p>
  <pre><code>2022-01-06 11:56:34,803 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: START_CONTAINER for Container container_1641415504751_0002_01_001995
  2022-01-06 11:57:35,818 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: START_CONTAINER for Container container_1641415504751_0002_01_001997
  2022-01-06 12:03:15,847 ERROR org.apache.flink.runtime.blob.BlobServerConnection          [] - GET operation failed for BLOB 90d7abcd5d82aa21555365e79e35d59e/p-c6b61dc9b8804b342e6b489bcae4947d106774ff-c9a6e4a2c8ca876fbf8bb00b0e48c026 from /192.168.60.208.
  java.nio.file.NoSuchFileException: /mnt1/yarn/usercache/hadoop/appcache/application_1641415504751_0002/blobStore-799fd2fa-fc91-44e8-ab93-39821ade8bbc/incoming/temp-00592798
     at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
  --
     at org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:221) [flink-dist_2.12-1.13.1.jar:1.13.1]
     at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:113) [flink-dist_2.12-1.13.1.jar:1.13.1]
  2022-01-06 12:03:15,850 ERROR org.apache.flink.runtime.blob.BlobServerConnection          [] - GET operation failed for BLOB 90d7abcd5d82aa21555365e79e35d59e/p-a60334225d79cfbe5eaba79595e57e9ad0bbde52-71724c141038a12c835d92df94b84482 from /192.168.58.233.
  java.nio.file.NoSuchFileException: /mnt1/yarn/usercache/hadoop/appcache/application_1641415504751_0002/blobStore-799fd2fa-fc91-44e8-ab93-39821ade8bbc/incoming/temp-00592800
     at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
  --
     at org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:221) [flink-dist_2.12-1.13.1.jar:1.13.1]
     at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:113) [flink-dist_2.12-1.13.1.jar:1.13.1]
  2022-01-06 12:03:15,854 ERROR org.apache.flink.runtime.blob.BlobServerConnection          [] - GET operation failed for BLOB 90d7abcd5d82aa21555365e79e35d59e/p-a60334225d79cfbe5eaba79595e57e9ad0bbde52-71724c141038a12c835d92df94b84482 from /192.168.58.233.
  java.nio.file.NoSuchFileException: /mnt1/yarn/usercache/hadoop/appcache/application_1641415504751_0002/blobStore-799fd2fa-fc91-44e8-ab93-39821ade8bbc/incoming/temp-00592804
     at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
  --
  --
     at org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:221) [flink-dist_2.12-1.13.1.jar:1.13.1]
     at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:113) [flink-dist_2.12-1.13.1.jar:1.13.1]
  2022-01-06 12:03:16,138 ERROR org.apache.flink.runtime.blob.BlobServerConnection          [] - GET operation failed for BLOB 90d7abcd5d82aa21555365e79e35d59e/p-850a861ef1d55301c78531cf311e1c9a7f8e7fc2-24509c272d93f2036e1a1dfe5870d869 from /192.168.60.208.
  java.nio.file.NoSuchFileException: /mnt1/yarn/usercache/hadoop/appcache/application_1641415504751_0002/blobStore-799fd2fa-fc91-44e8-ab93-39821ade8bbc/incoming/temp-00592842
     at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
  --
     at org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:221) [flink-dist_2.12-1.13.1.jar:1.13.1]
     at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:113) [flink-dist_2.12-1.13.1.jar:1.13.1]
  2022-01-06 12:03:16,140 ERROR org.apache.flink.runtime.blob.BlobServerConnection          [] - GET operation failed for BLOB 90d7abcd5d82aa21555365e79e35d59e/p-850a861ef1d55301c78531cf311e1c9a7f8e7fc2-24509c272d93f2036e1a1dfe5870d869 from /192.168.60.208.
  java.nio.file.NoSuchFileException: /mnt1/yarn/usercache/hadoop/appcache/application_1641415504751_0002/blobStore-799fd2fa-fc91-44e8-ab93-39821ade8bbc/incoming/temp-00592843
     at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
  --
  2022-01-06 12:04:21,174 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: START_CONTAINER for Container container_1641415504751_0002_01_002012
  2022-01-06 12:05:12,188 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: STOP_CONTAINER for Container container_1641415504751_0002_01_001997
  2022-01-06 12:05:39,252 ERROR org.apache.flink.runtime.rest.handler.job.JobSubmitHandler  [] - Unhandled exception.
  org.apache.flink.runtime.client.DuplicateJobSubmissionException: Job has already been submitted.
     at org.apache.flink.runtime.dispatcher.Dispatcher.submitJob(Dispatcher.java:305) ~[flink-dist_2.12-1.13.1.jar:1.13.1]
  --
  2022-01-06 12:05:42,198 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: START_CONTAINER for Container container_1641415504751_0002_01_002018
  2022-01-06 12:06:36,089 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: STOP_CONTAINER for Container container_1641415504751_0002_01_002009
  2022-01-06 12:06:44,636 ERROR org.apache.flink.runtime.rest.handler.job.JobSubmitHandler  [] - Unhandled exception.
  org.apache.flink.runtime.client.DuplicateJobSubmissionException: Job has already been submitted.
     at org.apache.flink.runtime.dispatcher.Dispatcher.submitJob(Dispatcher.java:305) ~[flink-dist_2.12-1.13.1.jar:1.13.1]
  --
     at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [flink-dist_2.12-1.13.1.jar:1.13.1]
     at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [flink-dist_2.12-1.13.1.jar:1.13.1]
  2022-01-06 12:06:45,184 ERROR org.apache.flink.runtime.rest.handler.job.JobSubmitHandler  [] - Unhandled exception.
  org.apache.flink.runtime.client.DuplicateJobSubmissionException: Job has already been submitted.
     at org.apache.flink.runtime.dispatcher.Dispatcher.submitJob(Dispatcher.java:305) ~[flink-dist_2.12-1.13.1.jar:1.13.1]
  --
  2022-01-06 12:07:50,240 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: START_CONTAINER for Container container_1641415504751_0002_01_002029
  2022-01-06 12:07:50,746 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: START_CONTAINER for Container container_1641415504751_0002_01_002034
  2022-01-06 12:08:15,911 ERROR org.apache.flink.runtime.blob.BlobServerConnection          [] - GET operation failed for BLOB 90d7abcd5d82aa21555365e79e35d59e/p-e7ede76cfed85f51c91b9f8e674f414d144c3a30-1526589304e89b8bee5f65ad9a9b2295 from /192.168.58.233.
  java.nio.file.NoSuchFileException: /mnt1/yarn/usercache/hadoop/appcache/application_1641415504751_0002/blobStore-799fd2fa-fc91-44e8-ab93-39821ade8bbc/incoming/temp-00601324
     at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
  --
     at org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:221) [flink-dist_2.12-1.13.1.jar:1.13.1]
     at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:113) [flink-dist_2.12-1.13.1.jar:1.13.1]
  2022-01-06 12:08:15,913 ERROR org.apache.flink.runtime.blob.BlobServerConnection          [] - GET operation failed for BLOB 90d7abcd5d82aa21555365e79e35d59e/p-e7ede76cfed85f51c91b9f8e674f414d144c3a30-1526589304e89b8bee5f65ad9a9b2295 from /192.168.58.233.
  java.nio.file.NoSuchFileException: /mnt1/yarn/usercache/hadoop/appcache/application_1641415504751_0002/blobStore-799fd2fa-fc91-44e8-ab93-39821ade8bbc/incoming/temp-00601325
     at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
  --
     at org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:221) [flink-dist_2.12-1.13.1.jar:1.13.1]
     at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:113) [flink-dist_2.12-1.13.1.jar:1.13.1]
  2022-01-06 12:08:15,914 ERROR org.apache.flink.runtime.blob.BlobServerConnection          [] - GET operation failed for BLOB 90d7abcd5d82aa21555365e79e35d59e/p-e7ede76cfed85f51c91b9f8e674f414d144c3a30-1526589304e89b8bee5f65ad9a9b2295 from /192.168.58.233.
  java.nio.file.NoSuchFileException: /mnt1/yarn/usercache/hadoop/appcache/application_1641415504751_0002/blobStore-799fd2fa-fc91-44e8-ab93-39821ade8bbc/incoming/temp-00601326
     at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
  --
     at org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:221) [flink-dist_2.12-1.13.1.jar:1.13.1]
     at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:113) [flink-dist_2.12-1.13.1.jar:1.13.1]
  2022-01-06 12:08:15,950 ERROR org.apache.flink.runtime.blob.BlobServerConnection          [] - GET operation failed for BLOB 90d7abcd5d82aa21555365e79e35d59e/p-850a861ef1d55301c78531cf311e1c9a7f8e7fc2-24509c272d93f2036e1a1dfe5870d869 from /192.168.58.233.
  java.nio.file.NoSuchFileException: /mnt1/yarn/usercache/hadoop/appcache/application_1641415504751_0002/blobStore-799fd2fa-fc91-44e8-ab93-39821ade8bbc/incoming/temp-00601339
     at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
  --
  2022-01-06 12:26:57,078 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: STOP_CONTAINER for Container container_1641415504751_0002_01_002068
  2022-01-06 12:27:01,699 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: STOP_CONTAINER for Container container_1641415504751_0002_01_002067
  --
  2022-01-06 12:57:57,314 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: START_CONTAINER for Container container_1641415504751_0002_01_002142
  2022-01-06 12:58:03,318 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: START_CONTAINER for Container container_1641415504751_0002_01_002144
  2022-01-06 12:58:36,833 ERROR org.apache.flink.runtime.history.FsJobArchivist             [] - Failed to archive job.
  java.nio.file.FileAlreadyExistsException: File already exists: file:/tmp/completed-jobs/dd2db85b6b6dc87d113edde723ad6a7f
     at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:257) ~[flink-dist_2.12-1.13.1.jar:1.13.1]
  --
     at java.lang.Thread.run(Thread.java:829) [?:?]
  2022-01-06 12:58:37,149 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: STOP_CONTAINER for Container container_1641415504751_0002_01_002138
  2022-01-06 12:58:37,288 ERROR org.apache.flink.runtime.blob.BlobServerConnection          [] - PUT operation failed
  java.nio.file.NoSuchFileException: /mnt1/yarn/usercache/hadoop/appcache/application_1641415504751_0002/blobStore-799fd2fa-fc91-44e8-ab93-39821ade8bbc/incoming/temp-00630967 -&gt; /mnt1/yarn/usercache/hadoop/appcache/application_1641415504751_0002/blobStore-799fd2fa-fc91-44e8-ab93-39821ade8bbc/job_dd2db85b6b6dc87d113edde723ad6a7f/blob_p-934c04d3cfef185a8008e7bf34331b79730a9d43-e5c7e19cbada968f3c75dcc89e077bfd
     at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
  --
     at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:110) [flink-dist_2.12-1.13.1.jar:1.13.1]
  2022-01-06 12:58:40,448 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: STOP_CONTAINER for Container container_1641415504751_0002_01_002070
  2022-01-06 12:58:40,659 ERROR org.apache.flink.runtime.blob.BlobServerConnection          [] - PUT operation failed
  java.nio.file.NoSuchFileException: /mnt1/yarn/usercache/hadoop/appcache/application_1641415504751_0002/blobStore-799fd2fa-fc91-44e8-ab93-39821ade8bbc/incoming/temp-00631105 -&gt; /mnt1/yarn/usercache/hadoop/appcache/application_1641415504751_0002/blobStore-799fd2fa-fc91-44e8-ab93-39821ade8bbc/job_cd37293d8cb598c7f9e832d8b0c62795/blob_p-710fd6abff4b26b40dc0917050dc4c67efcf60b6-54a6a74d3c02fbaba1064733424d7cbf
     at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
  --
     at org.apache.flink.runtime.blob.BlobServerConnection.put(BlobServerConnection.java:350) [flink-dist_2.12-1.13.1.jar:1.13.1]
     at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:110) [flink-dist_2.12-1.13.1.jar:1.13.1]
  2022-01-06 12:58:51,505 ERROR org.apache.flink.runtime.history.FsJobArchivist             [] - Failed to archive job.
  java.nio.file.FileAlreadyExistsException: File already exists: file:/tmp/completed-jobs/3387161148ad3473e9d01bc22900414e
     at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:257) ~[flink-dist_2.12-1.13.1.jar:1.13.1]
  --
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
     at java.lang.Thread.run(Thread.java:829) [?:?]
  2022-01-06 12:58:52,207 ERROR org.apache.flink.runtime.blob.BlobServerConnection          [] - PUT operation failed
  java.nio.file.NoSuchFileException: /mnt1/yarn/usercache/hadoop/appcache/application_1641415504751_0002/blobStore-799fd2fa-fc91-44e8-ab93-39821ade8bbc/incoming/temp-00631616 -&gt; /mnt1/yarn/usercache/hadoop/appcache/application_1641415504751_0002/blobStore-799fd2fa-fc91-44e8-ab93-39821ade8bbc/job_3387161148ad3473e9d01bc22900414e/blob_p-0858a1e1b677cbafd3b100d5154f491a7051401c-e6e96fbe7a5f8a75b587bbef932bbadf
     at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
  --
     at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:110) [flink-dist_2.12-1.13.1.jar:1.13.1]
  2022-01-06 12:58:52,550 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: STOP_CONTAINER for Container container_1641415504751_0002_01_002142
  2022-01-06 12:59:02,735 ERROR org.apache.flink.runtime.history.FsJobArchivist             [] - Failed to archive job.
  java.nio.file.FileAlreadyExistsException: File already exists: file:/tmp/completed-jobs/557dd38c28122bb4db34e030ddf5b149
     at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:257) ~[flink-dist_2.12-1.13.1.jar:1.13.1]
  --
  2022-01-06 12:59:09,268 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: STOP_CONTAINER for Container container_1641415504751_0002_01_002130
  2022-01-06 12:59:09,334 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: START_CONTAINER for Container container_1641415504751_0002_01_002145
  2022-01-06 12:59:09,648 ERROR org.apache.flink.runtime.blob.BlobServerConnection          [] - PUT operation failed
  java.nio.file.NoSuchFileException: /mnt1/yarn/usercache/hadoop/appcache/application_1641415504751_0002/blobStore-799fd2fa-fc91-44e8-ab93-39821ade8bbc/incoming/temp-00632310 -&gt; /mnt1/yarn/usercache/hadoop/appcache/application_1641415504751_0002/blobStore-799fd2fa-fc91-44e8-ab93-39821ade8bbc/job_8e7e273cbd873357481abfdd7c92f72f/blob_p-1536e1a8fd552dc05f12b5f0827fbb2ee3d5a89b-cfcddc7c2b91155d4d2d45ef4f324a0c
     at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
  --
  2022-01-06 12:59:18,838 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: STOP_CONTAINER for Container container_1641415504751_0002_01_002021
  2022-01-06 12:59:18,838 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: STOP_CONTAINER for Container container_1641415504751_0002_01_002080
  2022-01-06 12:59:39,648 ERROR org.apache.flink.runtime.history.FsJobArchivist             [] - Failed to archive job.
  java.nio.file.FileAlreadyExistsException: File already exists: file:/tmp/completed-jobs/d5347b9d775fee9f62e268a7d5417131
     at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:257) ~[flink-dist_2.12-1.13.1.jar:1.13.1]
  --
     at java.lang.Thread.run(Thread.java:829) [?:?]
  2022-01-06 12:59:39,958 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: STOP_CONTAINER for Container container_1641415504751_0002_01_002144
  2022-01-06 12:59:40,079 ERROR org.apache.flink.runtime.blob.BlobServerConnection          [] - PUT operation failed
  java.nio.file.NoSuchFileException: /mnt1/yarn/usercache/hadoop/appcache/application_1641415504751_0002/blobStore-799fd2fa-fc91-44e8-ab93-39821ade8bbc/incoming/temp-00633627 -&gt; /mnt1/yarn/usercache/hadoop/appcache/application_1641415504751_0002/blobStore-799fd2fa-fc91-44e8-ab93-39821ade8bbc/job_d5347b9d775fee9f62e268a7d5417131/blob_p-99f802e0cb3e953ba3d6e698795c4aeb98d37c48-2930b3c7d19f0787f3a6493b36f97e55
     at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
  --
     at org.apache.flink.runtime.blob.BlobServerConnection.put(BlobServerConnection.java:350) [flink-dist_2.12-1.13.1.jar:1.13.1]
     at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:110) [flink-dist_2.12-1.13.1.jar:1.13.1]
  2022-01-06 12:59:44,683 ERROR org.apache.flink.runtime.history.FsJobArchivist             [] - Failed to archive job.
  java.nio.file.FileAlreadyExistsException: File already exists: file:/tmp/completed-jobs/f307b2fff5429f8cf49ed3be01b456a4
     at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:257) ~[flink-dist_2.12-1.13.1.jar:1.13.1]
  --
     at java.lang.Thread.run(Thread.java:829) [?:?]
  2022-01-06 12:59:44,979 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: STOP_CONTAINER for Container container_1641415504751_0002_01_002141
  2022-01-06 13:00:13,278 ERROR org.apache.flink.runtime.history.FsJobArchivist             [] - Failed to archive job.
  java.nio.file.FileAlreadyExistsException: File already exists: file:/tmp/completed-jobs/e82e85782801ac7da5f8cd6069895e2e
     at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:257) ~[flink-dist_2.12-1.13.1.jar:1.13.1]
  --
  2022-01-06 13:00:15,447 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: START_CONTAINER for Container container_1641415504751_0002_01_002147
  2022-01-06 13:00:20,653 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: STOP_CONTAINER for Container container_1641415504751_0002_01_002147
  2022-01-06 13:00:40,434 ERROR org.apache.flink.runtime.history.FsJobArchivist             [] - Failed to archive job.
  java.nio.file.FileAlreadyExistsException: File already exists: file:/tmp/completed-jobs/439e426e0f13d3bbfb01f6782658051f
     at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:257) ~[flink-dist_2.12-1.13.1.jar:1.13.1]
  --
  2022-01-06 13:04:04,795 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: START_CONTAINER for Container container_1641415504751_0002_01_002164
  2022-01-06 13:04:09,793 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: STOP_CONTAINER for Container container_1641415504751_0002_01_002164
  2022-01-06 13:04:17,651 ERROR org.apache.flink.runtime.history.FsJobArchivist             [] - Failed to archive job.
  java.nio.file.FileAlreadyExistsException: File already exists: file:/tmp/completed-jobs/439e426e0f13d3bbfb01f6782658051f
     at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:257) ~[flink-dist_2.12-1.13.1.jar:1.13.1]
  --
  2022-01-06 13:04:30,808 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: START_CONTAINER for Container container_1641415504751_0002_01_002165
  2022-01-06 13:04:31,812 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: START_CONTAINER for Container container_1641415504751_0002_01_002167
  2022-01-06 13:04:49,092 ERROR org.apache.flink.runtime.history.FsJobArchivist             [] - Failed to archive job.
  java.nio.file.FileAlreadyExistsException: File already exists: file:/tmp/completed-jobs/dd2db85b6b6dc87d113edde723ad6a7f
     at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:257) ~[flink-dist_2.12-1.13.1.jar:1.13.1]
  --
  2022-01-06 14:59:50,036 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Processing Event EventType: START_CONTAINER for Container container_1641415504751_0002_01_002720
  </code></pre>