<p>I'm trying to modify some prometheusrule in my cluster but I'm encountering a timeout error that I don't understand. Here is a sample of the rule I'm trying to modify.</p>
<pre><code># modif.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  annotations:
    meta.helm.sh/release-name: monitoring-platform
    meta.helm.sh/release-namespace: monitoring
  creationTimestamp: &quot;2022-01-04T09:20:58Z&quot;
  generation: 1
  labels:
    app: prometheus-operator
    app.kubernetes.io/managed-by: Helm
    prometheus: kube-op
    release: monitoring-platform
  name: kube-op-apps-rules
  namespace: monitoring
  resourceVersion: &quot;948572193&quot;
  uid: a461d478-9e61-4004-a129-9ed3f5efe8b0
spec:
  groups:
  - name: kubernetes-apps
    rules:
    - alert: KubePodCrashLooping
      annotations:
        message: Pod {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container
          }}) is restarting {{ printf &quot;%.2f&quot; $value }} times / 20 minutes.
        runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepodcrashlooping
      expr: rate(kube_pod_container_status_restarts_total{job=&quot;kube-state-metrics&quot;,
        pod!~&quot;social-reco-prod.*&quot;}[15m]) * 60 * 20 &gt; 0
      for: 1h
      labels:
        severity: critical

</code></pre>
<p>If I try to apply this file I get the following error</p>
<pre><code>09:14:23 bastien@work:/work/$ kubectl -v 6 apply -f modif.yaml
I0103 09:14:25.389081 1969468 loader.go:379] Config loaded from file:  /home/bastien/.kube/config
I0103 09:14:25.436584 1969468 round_trippers.go:445] GET https://1.2.3.4/openapi/v2?timeout=32s 200 OK in 46 milliseconds
I0103 09:14:25.655706 1969468 round_trippers.go:445] GET https://1.2.3.4/apis/external.metrics.k8s.io/v1beta1?timeout=32s 200 OK in 14 milliseconds
I0103 09:14:25.664871 1969468 cached_discovery.go:82] skipped caching discovery info, no resources found
I0103 09:14:25.696947 1969468 round_trippers.go:445] GET https://1.2.3.4/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules/kube-op-apps-rules 404 Not Found in 30 milliseconds
I0103 09:14:25.728817 1969468 round_trippers.go:445] GET https://1.2.3.4/api/v1/namespaces/monitoring 200 OK in 31 milliseconds
I0103 09:14:59.759927 1969468 round_trippers.go:445] POST https://1.2.3.4/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules?fieldManager=kubectl-client-side-apply 504 Gateway Timeout in 34030 milliseconds
</code></pre>
<p>The rule seems to be good since I have a secondary cluster on which the apply command is working fine. Both cluster have the same version :</p>
<pre><code># The malfunctioning one
12:37:34 bastien@work:/work/$ kubectl version
Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;20&quot;, GitVersion:&quot;v1.20.5&quot;, GitCommit:&quot;6b1d87acf3c8253c123756b9e61dac642678305f&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-03-18T01:10:43Z&quot;, GoVersion:&quot;go1.15.8&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;21&quot;, GitVersion:&quot;v1.21.14-gke.4300&quot;, GitCommit:&quot;348bdc1040d273677ca07c0862de867332eeb3a1&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2022-08-17T09:22:54Z&quot;, GoVersion:&quot;go1.16.15b7&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}

# The working one
13:25:07 bastien@work:/work/$ kubectl version
Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;20&quot;, GitVersion:&quot;v1.20.5&quot;, GitCommit:&quot;6b1d87acf3c8253c123756b9e61dac642678305f&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-03-18T01:10:43Z&quot;, GoVersion:&quot;go1.15.8&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;21&quot;, GitVersion:&quot;v1.21.14-gke.4300&quot;, GitCommit:&quot;348bdc1040d273677ca07c0862de867332eeb3a1&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2022-08-17T09:22:54Z&quot;, GoVersion:&quot;go1.16.15b7&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
</code></pre>
<p>Do you have any clue what's wrong? Or at least where could I find some logs or info on what's going on?</p>

b.moyet
<p>I finally stumbled upon someone who had an issue which looked like mine.</p>
<p><a href="https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/8303" rel="nofollow noreferrer">https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/8303</a></p>
<p>In this thread the OP did several tests until someone pointed out a potential solution that consists in allowing GKE master to communicate with kubelet.</p>
<p><a href="https://github.com/prometheus-operator/prometheus-operator/issues/2711#issuecomment-521103022" rel="nofollow noreferrer">https://github.com/prometheus-operator/prometheus-operator/issues/2711#issuecomment-521103022</a></p>
<p>The proposed terraform is the following :</p>
<pre><code>resource &quot;google_compute_firewall&quot; &quot;gke-master-to-kubelet&quot; {
  name    = &quot;k8s-master-to-kubelets&quot;
  network = &quot;XXXXX&quot;
  project = &quot;XXXXX&quot;

  description = &quot;GKE master to kubelets&quot;

  source_ranges = [&quot;${data.terraform_remote_state.network.master_ipv4_cidr_block}&quot;]

  allow {
    protocol = &quot;tcp&quot;
    ports    = [&quot;8443&quot;]
  }

  target_tags = [&quot;gke-main&quot;]
}
</code></pre>
<p>Once I added this firewall rule on my side it completely fix my issue. I still don't know why it suddenly stop working.</p>
