<p>The following code will throw an <code>ApiException 410 resource too old</code> on the second <code>watch.stream()</code>:</p>
<pre><code># python3 -m venv venv
# source venv/bin/activate
# pip install 'kubernetes==23.3.0'

from kubernetes import client,config,watch

config.load_kube_config(context='my-eks-context')

v1 = client.CoreV1Api()
watcher = watch.Watch()


namespace = 'kube-system'
last_resource_version=0

# this watch will timeout in 5s to have a fast way to simulate a watch that need to be retried 
for i in  watcher.stream(v1.list_namespaced_pod, namespace, resource_version=last_resource_version, timeout_seconds=5):
   print(i['object'].metadata.resource_version)
   last_resource_version = i['object'].metadata.resource_version


# we retry the watch starting from the last resource version known
# but this will raise a kubernetes.client.exceptions.ApiException: (410)
# Reason: Expired: too old resource version: 379140622 (380367990)
for i in  watcher.stream(v1.list_namespaced_pod, namespace, resource_version=last_resource_version, timeout_seconds=5):
    print('second loop', i['object'].metadata.resource_version)
    last_resource_version = i['object'].metadata.resource_version

</code></pre>
<p>The <a href="https://kubernetes.io/docs/reference/using-api/api-concepts/#efficient-detection-of-changes" rel="nofollow noreferrer">kubernetes documentation</a> states that:</p>
<blockquote>
<p>If a client watch is disconnected then that client can start a new watch from the last returned resourceVersion</p>
</blockquote>
<p>which is what I intended in the above code, which always gives the following exception:</p>
<pre><code>Traceback (most recent call last):
  File &quot;main.py&quot;, line 24, in &lt;module&gt;

  File &quot;/Users/rubelagu/git/python-kubernetes-client/venv/lib/python3.8/site-packages/kubernetes/watch/watch.py&quot;, line 182, in stream
    raise client.rest.ApiException(
kubernetes.client.exceptions.ApiException: (410)
Reason: Expired: too old resource version: 379164133 (380432814)
</code></pre>
<p>What am I doing wrong?</p>

<p>It seems that in the initial response to the watch (from an EKS cluster 1.21) the events can be returned in any order.</p>
<p>I did two subsequent watches two seconds apart and they contain the same 30 events in completely different ordering.</p>
<p>So it's not guaranteed that the last resource version that you see is actually the actual last and it's not guaranteed that you can resume from that <code>resourceVersion</code>/<code>resource_version</code>. Also you are not allowed to sort/collate those events by <code>resourceVersion</code>, since the <a href="https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions" rel="nofollow noreferrer">kubernetes documentation for Resource Version Semantics</a> explicity says:</p>
<blockquote>
<p>Resource versions must be treated as <strong>opaque</strong> [...] You <strong>must not</strong> assume resource versions are <strong>numeric or collatable</strong>.</p>
</blockquote>
<p>You must account for that by catching the <code>resource too old exception</code> and retrying without specifying a resource version, see below for an example:</p>
<pre><code>from kubernetes import client,config,watch
from kubernetes.client.exceptions import ApiException

config.load_kube_config(context='eks-prod')

v1 = client.CoreV1Api()
# v1 = config.new_client_from_config(context=&quot;eks-prod&quot;).CoreV1Api()
watcher = watch.Watch()


namespace = 'my-namespace'


def list_pods(resource_version=None):
   print('start watch from resource version: ', str(resource_version))
   try: 
      for i in  watcher.stream(v1.list_namespaced_pod, namespace, resource_version=resource_version, timeout_seconds=2):
         print(i['object'].metadata.resource_version)
         last_resource_version = i['object'].metadata.resource_version
   except ApiException as e: 
      if e.status == 410: # Resource too old
         return list_pods(resource_version=None)
      else:
         raise

   return last_resource_version

last_resource_version = list_pods()

print('last_resource_version', last_resource_version)

list_pods(last_resource_version)

</code></pre>
