<p>I want to get all events that occurred in Kubernetes cluster in some python dictionary using maybe some API to extract data from the events that occurred in the past. I found on internet that it is possible by storing all data of Kube-watch on Prometheus and later accessing it. I am unable to figure out how to set it up and see all past pod events in python. Any alternative solutions to access past events are also appreciated. Thanks!</p>

Dragnoid99
<p>I'll describe a solution that is not complicated and I think meets all your requirements.
There are tools such as <a href="https://github.com/heptiolabs/eventrouter" rel="nofollow noreferrer">Eventrouter</a> that take Kubernetes events and push them to a user specified sink. However, as you mentioned, you only need Pods events, so I suggest a slightly different approach.</p>
<p>In short, you can run the <code>kubectl get events --watch</code> command from within a Pod and collect the output from that command using a log aggregation system like <a href="https://grafana.com/oss/loki/" rel="nofollow noreferrer">Loki</a>.</p>
<p>Below, I will provide a detailed step-by-step explanation.</p>
<h3>1. Running kubectl command from within a Pod</h3>
<p>To display only Pod events, you can use:</p>
<pre><code>$ kubectl get events --watch --field-selector involvedObject.kind=Pod
</code></pre>
<p>We want to run this command from within a Pod. For security reasons, I've created a separate <code>events-collector</code> ServiceAccount with the <code>view</code> Role assigned and our Pod will run under this ServiceAccount.<br />
<strong>NOTE:</strong> I've created a <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/" rel="nofollow noreferrer">Deployment</a> instead of a single Pod.</p>
<pre><code>$ cat all-in-one.yml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: events-collector
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: events-collector-binding
subjects:
  - kind: ServiceAccount
    name: events-collector
    namespace: default
roleRef:
  kind: ClusterRole
  name: view
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: events-collector
  name: events-collector
spec:
  selector:
    matchLabels:
      app: events-collector
  template:
    metadata:
      labels:
        app: events-collector
    spec:
      serviceAccountName: events-collector
      containers:
      - image: bitnami/kubectl
        name: test
        command: [&quot;kubectl&quot;]
        args: [&quot;get&quot;,&quot;events&quot;, &quot;--watch&quot;, &quot;--field-selector&quot;, &quot;involvedObject.kind=Pod&quot;]
</code></pre>
<p>After applying the above manifest, the <code>event-collector</code> was created and collects Pod events as expected:</p>
<pre><code>$ kubectl apply -f all-in-one.yml
serviceaccount/events-collector created
clusterrolebinding.rbac.authorization.k8s.io/events-collector-binding created
deployment.apps/events-collector created

$ kubectl get deploy,pod | grep events-collector
deployment.apps/events-collector           1/1     1            1           14s
pod/events-collector-d98d6c5c-xrltj            1/1     Running   0          14s

$ kubectl logs -f events-collector-d98d6c5c-xrltj
LAST SEEN   TYPE     REASON      OBJECT                                         MESSAGE
77s         Normal   Scheduled   pod/app-1-5d9ccdb595-m9d5n                     Successfully assigned default/app-1-5d9ccdb595-m9d5n to gke-cluster-2-default-pool-8505743b-brmx
76s         Normal   Pulling     pod/app-1-5d9ccdb595-m9d5n                     Pulling image &quot;nginx&quot;
71s         Normal   Pulled      pod/app-1-5d9ccdb595-m9d5n                     Successfully pulled image &quot;nginx&quot; in 4.727842954s
70s         Normal   Created     pod/app-1-5d9ccdb595-m9d5n                     Created container nginx
70s         Normal   Started     pod/app-1-5d9ccdb595-m9d5n                     Started container nginx
73s         Normal   Scheduled   pod/app-2-7747dcb588-h8j4q                     Successfully assigned default/app-2-7747dcb588-h8j4q to gke-cluster-2-default-pool-8505743b-p7qt
72s         Normal   Pulling     pod/app-2-7747dcb588-h8j4q                     Pulling image &quot;nginx&quot;
67s         Normal   Pulled      pod/app-2-7747dcb588-h8j4q                     Successfully pulled image &quot;nginx&quot; in 4.476795932s
66s         Normal   Created     pod/app-2-7747dcb588-h8j4q                     Created container nginx
66s         Normal   Started     pod/app-2-7747dcb588-h8j4q                     Started container nginx
</code></pre>
<h3>2. Installing Loki</h3>
<p>You can install <a href="https://github.com/grafana/loki" rel="nofollow noreferrer">Loki</a> to store logs and process queries. Loki is like Prometheus, but for logs :). The easiest way to install Loki is to use the <a href="https://github.com/grafana/helm-charts/tree/main/charts/loki-stack" rel="nofollow noreferrer">grafana/loki-stack</a> Helm chart:</p>
<pre><code>$ helm repo add grafana https://grafana.github.io/helm-charts
&quot;grafana&quot; has been added to your repositories

$ helm repo update
...
Update Complete. ⎈Happy Helming!⎈

$ helm upgrade --install loki grafana/loki-stack


$ kubectl get pods | grep loki
loki-0                            1/1     Running   0          76s
loki-promtail-hm8kn               1/1     Running   0          76s
loki-promtail-nkv4p               1/1     Running   0          76s
loki-promtail-qfrcr               1/1     Running   0          76s
</code></pre>
<h3>3. Querying Loki with LogCLI</h3>
<p>You can use the <a href="https://grafana.com/docs/loki/latest/getting-started/logcli/" rel="nofollow noreferrer">LogCLI</a> tool to run LogQL queries against a Loki server. Detailed information on installing and using this tool can be found in the <a href="https://grafana.com/docs/loki/latest/getting-started/logcli/#installation" rel="nofollow noreferrer">LogCLI documentation</a>. I'll demonstrate how to install it on Linux:</p>
<pre><code>$ wget https://github.com/grafana/loki/releases/download/v2.2.1/logcli-linux-amd64.zip

$ unzip logcli-linux-amd64.zip
Archive:  logcli-linux-amd64.zip
  inflating: logcli-linux-amd64
  
$ mv logcli-linux-amd64 logcli

$ sudo cp logcli /bin/

$ whereis logcli
logcli: /bin/logcli
</code></pre>
<p>To query the Loki server from outside the Kubernetes cluster, you may need to expose it using the <a href="https://kubernetes.io/docs/concepts/services-networking/ingress/" rel="nofollow noreferrer">Ingress</a> resource:</p>
<pre><code>$ cat ingress.yml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
  name: loki-ingress
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: loki
              servicePort: 3100
            path: /
            
 
$ kubectl apply -f ingress.yml
ingress.networking.k8s.io/loki-ingress created

$ kubectl get ing
NAME           CLASS    HOSTS   ADDRESS         PORTS   AGE
loki-ingress   &lt;none&gt;   *      &lt;PUBLIC_IP&gt;      80      19s
</code></pre>
<p>Finally, I've created a simple python script that we can use to query the Loki server:<br />
<strong>NOTE:</strong> We need to set the <code>LOKI_ADDR</code> environment variable as described in the <a href="https://grafana.com/docs/loki/latest/getting-started/logcli/#example" rel="nofollow noreferrer">documentation</a>. You need to replace the <code>&lt;PUBLIC_IP&gt;</code> with your Ingress IP.</p>
<pre><code>$ cat query_loki.py
#!/usr/bin/env python3

import os

os.environ['LOKI_ADDR'] = &quot;http://&lt;PUBLIC_IP&gt;&quot;

os.system(&quot;logcli query '{app=\&quot;events-collector\&quot;}'&quot;)


$ ./query_loki.py
...
2021-07-02T10:33:01Z {} 2021-07-02T10:33:01.626763464Z stdout F 0s          Normal    Pulling       pod/backend-app-5d99cf4b-c9km4                               Pulling image &quot;nginx&quot;
2021-07-02T10:33:00Z {} 2021-07-02T10:33:00.836755152Z stdout F 0s          Normal    Scheduled     pod/backend-app-5d99cf4b-c9km4                               Successfully assigned default/backend-app-5d99cf4b-c9km4 to gke-cluster-1-default-pool-328bd2b1-288w
2021-07-02T10:33:00Z {} 2021-07-02T10:33:00.649954267Z stdout F 0s          Normal    Started       pod/web-app-6fcf9bb7b8-jbrr9                                 Started container nginx2021-07-02T10:33:00Z {} 2021-07-02T10:33:00.54819851Z stdout F 0s          Normal    Created       pod/web-app-6fcf9bb7b8-jbrr9                                 Created container nginx
2021-07-02T10:32:59Z {} 2021-07-02T10:32:59.414571562Z stdout F 0s          Normal    Pulled        pod/web-app-6fcf9bb7b8-jbrr9                                 Successfully pulled image &quot;nginx&quot; in 4.228468876s
...
</code></pre>
