<p>I'm using zalando's postgres operator and been having postgres cluster downtime now. I'm using connection pooler to connection to master and replica, but replica connection pooler not able to connect to replica pod due to replica svc does not have endpoints, which I think the problem to be the svc select postgres pod that has label of <code>spilo-role</code> to be <code>replica</code> but pods does not have such labels, both master and replica which they supposed to be.</p>
<p>The cluster has been running fine for one months and this incident just happened few days ago until now, we are still in soft production stage and there are only few test clients with very low traffic, but soon in real production.</p>
<p>Logs of operator and postgres does not seems to have any errors of what I would have know so I looked into log explorer in google cloud console and found trace from audit logs that patroni actually call to set the pod label but result in <code>504</code> error. The error seems to be from misconfiguration, but it is strange that it has been running fine until now and I am running out of idea of how to debug this, so any guide or help to debug/fix this would be very much appreciated.</p>
<p>Below is the audit logs from google cloud console log explorer that shows that pod has permission to do pod label update but it fails,</p>
<pre class="lang-json prettyprint-override"><code>{
  &quot;protoPayload&quot;: {
    &quot;@type&quot;: &quot;type.googleapis.com/google.cloud.audit.AuditLog&quot;,
    &quot;authenticationInfo&quot;: {
      &quot;principalEmail&quot;: &quot;system:serviceaccount:default:postgres-pod&quot;
    },
    &quot;authorizationInfo&quot;: [
      {
        &quot;granted&quot;: true,
        &quot;permission&quot;: &quot;io.k8s.core.v1.pods.patch&quot;,
        &quot;resource&quot;: &quot;core/v1/namespaces/default/pods/acid-abc-db-1&quot;
      }
    ],
    &quot;methodName&quot;: &quot;io.k8s.core.v1.pods.patch&quot;,
    &quot;request&quot;: {
      &quot;@type&quot;: &quot;k8s.io/Patch&quot;,
      &quot;metadata&quot;: {
        &quot;annotations&quot;: {
          &quot;status&quot;: &quot;{\&quot;conn_url\&quot;:\&quot;postgres://10.52.3.36:5432/postgres\&quot;,\&quot;api_url\&quot;:\&quot;http://10.52.3.36:8008/patroni\&quot;,\&quot;state\&quot;:\&quot;running\&quot;,\&quot;role\&quot;:\&quot;replica\&quot;,\&quot;version\&quot;:\&quot;2.1.3\&quot;,\&quot;xlog_location\&quot;:50331648,\&quot;timeline\&quot;:1}&quot;
        },
        &quot;labels&quot;: {
          &quot;spilo-role&quot;: &quot;replica&quot;
        },
        &quot;name&quot;: &quot;acid-abc-db-1&quot;,
        &quot;namespace&quot;: &quot;default&quot;
      }
    },
    &quot;requestMetadata&quot;: {
      &quot;callerIp&quot;: &quot;10.52.3.36&quot;,
      &quot;callerSuppliedUserAgent&quot;: &quot;Patroni/2.1.3 Python/3.6.9 Linux&quot;
    },
    &quot;resourceName&quot;: &quot;core/v1/namespaces/default/pods/acid-ml-db-1&quot;,
    &quot;response&quot;: {
      &quot;@type&quot;: &quot;core.k8s.io/v1.Status&quot;,
      &quot;apiVersion&quot;: &quot;v1&quot;,
      &quot;code&quot;: 504,
      &quot;details&quot;: {},
      &quot;kind&quot;: &quot;Status&quot;,
      &quot;message&quot;: &quot;Timeout: request did not complete within requested timeout - context canceled&quot;,
      &quot;metadata&quot;: {},
      &quot;reason&quot;: &quot;Timeout&quot;,
      &quot;status&quot;: &quot;Failure&quot;
    },
    &quot;serviceName&quot;: &quot;k8s.io&quot;,
    &quot;status&quot;: {
      &quot;code&quot;: 4,
      &quot;message&quot;: &quot;Gateway Timeout&quot;
    }
  },
  &quot;insertId&quot;: &quot;b6e3cfe7-0125-4652-a77a-f44232198f8c&quot;,
  &quot;resource&quot;: {
    &quot;type&quot;: &quot;k8s_cluster&quot;,
    &quot;labels&quot;: {
      &quot;project_id&quot;: &quot;abc123&quot;,
      &quot;cluster_name&quot;: &quot;abc&quot;,
      &quot;location&quot;: &quot;asia-southeast1&quot;
    }
  },
  &quot;timestamp&quot;: &quot;2022-09-18T09:21:05.017886Z&quot;,
  &quot;labels&quot;: {
    &quot;authorization.k8s.io/decision&quot;: &quot;allow&quot;,
    &quot;authorization.k8s.io/reason&quot;: &quot;RBAC: allowed by ClusterRoleBinding \&quot;postgres-pod\&quot; of ClusterRole \&quot;postgres-pod\&quot; to ServiceAccount \&quot;postgres-pod/default\&quot;&quot;
  },
  &quot;logName&quot;: &quot;projects/ekyc-web-services/logs/cloudaudit.googleapis.com%2Factivity&quot;,
  &quot;operation&quot;: {
    &quot;id&quot;: &quot;b6e3cfe7-0125-4652-a77a-f44232198f8c&quot;,
    &quot;producer&quot;: &quot;k8s.io&quot;,
    &quot;first&quot;: true,
    &quot;last&quot;: true
  },
  &quot;receiveTimestamp&quot;: &quot;2022-09-18T09:21:10.235550735Z&quot;
}
</code></pre>
<p>Usually <code>patronictl list</code> should show state in <code>Running</code> and ip address in <code>Host</code> column, but now they are empty</p>
<pre><code>+ Cluster: acid-abc-db (7144662354080374866) -+-----------+
| Member        | Host | Role    | State | TL | Lag in MB |
+---------------+------+---------+-------+----+-----------+
| acid-abc-db-0 |      | Leader  |       |    |           |
| acid-abc-db-1 |      | Replica |       |    |   unknown |
+---------------+------+---------+-------+----+-----------+
</code></pre>
<p>I also tried to create entirely new cluster with any name, it would also give me the same result.</p>
<p>logs from master pod<code>acid-abc-db-0</code></p>
<pre><code>2022-09-18 10:18:45,881 - bootstrapping - INFO - Figuring out my environment (Google? AWS? Openstack? Local?)
2022-09-18 10:18:45,970 - bootstrapping - INFO - Looks like your running google
2022-09-18 10:18:47,087 - bootstrapping - INFO - Configuring bootstrap
2022-09-18 10:18:47,087 - bootstrapping - INFO - Configuring pgqd
2022-09-18 10:18:47,088 - bootstrapping - INFO - Configuring wal-e
2022-09-18 10:18:47,089 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALE_S3_PREFIX
2022-09-18 10:18:47,090 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALG_S3_PREFIX
2022-09-18 10:18:47,090 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/AWS_ACCESS_KEY_ID
2022-09-18 10:18:47,091 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/AWS_SECRET_ACCESS_KEY
2022-09-18 10:18:47,091 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/AWS_REGION
2022-09-18 10:18:47,091 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALG_S3_SSE
2022-09-18 10:18:47,092 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALG_DOWNLOAD_CONCURRENCY
2022-09-18 10:18:47,092 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALG_UPLOAD_CONCURRENCY
2022-09-18 10:18:47,093 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/USE_WALG_BACKUP
2022-09-18 10:18:47,093 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/USE_WALG_RESTORE
2022-09-18 10:18:47,093 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALE_LOG_DESTINATION
2022-09-18 10:18:47,094 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/PGPORT
2022-09-18 10:18:47,094 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/BACKUP_NUM_TO_RETAIN
2022-09-18 10:18:47,095 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/TMPDIR
2022-09-18 10:18:47,095 - bootstrapping - INFO - Configuring log
2022-09-18 10:18:47,095 - bootstrapping - INFO - Configuring patroni
2022-09-18 10:18:47,104 - bootstrapping - INFO - Writing to file /run/postgres.yabc
2022-09-18 10:18:47,105 - bootstrapping - INFO - Configuring pam-oauth2
2022-09-18 10:18:47,106 - bootstrapping - INFO - Writing to file /etc/pam.d/postgresql
2022-09-18 10:18:47,106 - bootstrapping - INFO - Configuring certificate
2022-09-18 10:18:47,107 - bootstrapping - INFO - Generating ssl self-signed certificate
2022-09-18 10:18:47,226 - bootstrapping - INFO - Configuring standby-cluster
2022-09-18 10:18:47,226 - bootstrapping - INFO - Configuring crontab
2022-09-18 10:18:47,227 - bootstrapping - INFO - Skipping creation of renice cron job due to lack of SYS_NICE capability
2022-09-18 10:18:47,242 - bootstrapping - INFO - Configuring pgbouncer
2022-09-18 10:18:47,242 - bootstrapping - INFO - No PGBOUNCER_CONFIGURATION was specified, skipping
2022-09-18 10:18:48,994 INFO: Selected new K8s API server endpoint https://172.16.0.2:443
2022-09-18 10:18:49,017 INFO: No PostgreSQL configuration items changed, nothing to reload.
2022-09-18 10:18:49,020 INFO: Lock owner: None; I am acid-abc-db-0
2022-09-18 10:18:54,082 INFO: trying to bootstrap a new cluster
The files belonging to this database system will be owned by user &quot;postgres&quot;.
This user must also own the server process.

The database cluster will be initialized with locale &quot;en_US.UTF-8&quot;.
The default text search configuration will be set to &quot;english&quot;.

Data page checksums are enabled.

fixing permissions on existing directory /home/postgres/pgdata/pgroot/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

/usr/lib/postgresql/14/bin/pg_ctl -D /home/postgres/pgdata/pgroot/data -l logfile start

2022-09-18 10:18:56,761 INFO: postmaster pid=92
/var/run/postgresql:5432 - no response
2022-09-18 10:18:56 UTC [92]: [1-1] 6326f090.5c 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-09-18 10:18:56 UTC [92]: [2-1] 6326f090.5c 0     LOG:  pg_stat_kcache.linux_hz is set to 500000
2022-09-18 10:18:56 UTC [92]: [3-1] 6326f090.5c 0     LOG:  redirecting log output to logging collector process
2022-09-18 10:18:56 UTC [92]: [4-1] 6326f090.5c 0     HINT:  Future log output will appear in directory &quot;../pg_log&quot;.
/var/run/postgresql:5432 - accepting connections
/var/run/postgresql:5432 - accepting connections
2022-09-18 10:18:57,834 INFO: establishing a new patroni connection to the postgres cluster
2022-09-18 10:19:02,852 INFO: running post_bootstrap
DO
GRANT ROLE
DO
DO
CREATE EXTENSION
NOTICE:  version &quot;1.1&quot; of extension &quot;pg_auth_mon&quot; is already installed
ALTER EXTENSION
GRANT
CREATE EXTENSION
DO
NOTICE:  version &quot;1.4&quot; of extension &quot;pg_cron&quot; is already installed
ALTER EXTENSION
ALTER POLICY
REVOKE
GRANT
REVOKE
GRANT
ALTER POLICY
REVOKE
GRANT
CREATE FUNCTION
REVOKE
GRANT
REVOKE
GRANT
REVOKE
GRANT
REVOKE
GRANT
REVOKE
GRANT
REVOKE
GRANT
REVOKE
GRANT
REVOKE
GRANT
CREATE EXTENSION
DO
CREATE TABLE
GRANT
ALTER TABLE
ALTER TABLE
ALTER TABLE
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
RESET
SET
NOTICE:  schema &quot;zmon_utils&quot; does not exist, skipping
DROP SCHEMA
DO
NOTICE:  language &quot;plpythonu&quot; does not exist, skipping
DROP LANGUAGE
NOTICE:  function plpython_call_handler() does not exist, skipping
DROP FUNCTION
NOTICE:  function plpython_inline_handler(internal) does not exist, skipping
DROP FUNCTION
NOTICE:  function plpython_validator(oid) does not exist, skipping
DROP FUNCTION
CREATE SCHEMA
GRANT
SET
CREATE TYPE
CREATE FUNCTION
CREATE FUNCTION
GRANT
You are now connected to database &quot;postgres&quot; as user &quot;postgres&quot;.
CREATE SCHEMA
GRANT
SET
CREATE FUNCTION
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
GRANT
RESET
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
NOTICE:  version &quot;3.0&quot; of extension &quot;set_user&quot; is already installed
ALTER EXTENSION
GRANT
GRANT
GRANT
CREATE SCHEMA
GRANT
GRANT
SET
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
RESET
You are now connected to database &quot;template1&quot; as user &quot;postgres&quot;.
CREATE SCHEMA
GRANT
SET
CREATE FUNCTION
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
GRANT
RESET
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
NOTICE:  version &quot;3.0&quot; of extension &quot;set_user&quot; is already installed
ALTER EXTENSION
GRANT
GRANT
GRANT
CREATE SCHEMA
GRANT
GRANT
SET
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
RESET
2022-09-18 10:19:05,009 WARNING: Could not activate Linux watchdog device: &quot;Can't open watchdog device: [Errno 2] No such file or directory: '/dev/watchdog'&quot;
2022-09-18 10:19:10,054 INFO: initialized a new cluster
2022-09-18 10:19:15,087 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:19:25,582 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:19:35,601 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:19:45,588 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:19:47.662 - /scripts/postgres_backup.sh - I was called as: /scripts/postgres_backup.sh /home/postgres/pgdata/pgroot/data
2022-09-18 10:19:48.397 45 LOG Starting pgqd 3.3
2022-09-18 10:19:48.397 45 LOG auto-detecting dbs ...
2022-09-18 10:19:48.941 - /scripts/postgres_backup.sh - producing a new backup
INFO: 2022/09/18 10:19:49.036810 Selecting the latest backup as the base for the current delta backup...
INFO: 2022/09/18 10:19:49.091402 Calling pg_start_backup()
INFO: 2022/09/18 10:19:49.203073 Starting a new tar bundle
INFO: 2022/09/18 10:19:49.203129 Walking ...
INFO: 2022/09/18 10:19:49.203471 Starting part 1 ...
INFO: 2022/09/18 10:19:50.107584 Packing ...
INFO: 2022/09/18 10:19:50.109248 Finished writing part 1.
INFO: 2022/09/18 10:19:50.428312 Starting part 2 ...
INFO: 2022/09/18 10:19:50.428359 /global/pg_control
INFO: 2022/09/18 10:19:50.437376 Finished writing part 2.
INFO: 2022/09/18 10:19:50.439403 Calling pg_stop_backup()
INFO: 2022/09/18 10:19:51.470246 Starting part 3 ...
INFO: 2022/09/18 10:19:51.496912 backup_label
INFO: 2022/09/18 10:19:51.497397 tablespace_map
INFO: 2022/09/18 10:19:51.497645 Finished writing part 3.
INFO: 2022/09/18 10:19:51.632504 Wrote backup with name base_000000010000000000000002
2022-09-18 10:19:55,586 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:20:05,587 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:20:15,579 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:20:18.427 45 LOG {ticks: 0, maint: 0, retry: 0}
2022-09-18 10:20:25,586 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:20:35,578 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:20:45,722 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:20:48.469 45 LOG {ticks: 0, maint: 0, retry: 0}
2022-09-18 10:20:55,583 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:21:05,587 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:21:15,586 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:21:18.470 45 LOG {ticks: 0, maint: 0, retry: 0}
2022-09-18 10:21:25,586 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:21:35,590 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:21:45,587 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:21:48.501 45 LOG {ticks: 0, maint: 0, retry: 0}
2022-09-18 10:21:55,588 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:22:05,589 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:22:15,589 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:22:18.532 45 LOG {ticks: 0, maint: 0, retry: 0}
2022-09-18 10:22:25,585 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:22:35,589 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:22:45,584 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:22:48.580 45 LOG {ticks: 0, maint: 0, retry: 0}
2022-09-18 10:22:55,583 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:23:05,600 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:23:15,586 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:23:18.572 45 LOG {ticks: 0, maint: 0, retry: 0}
2022-09-18 10:23:25,584 INFO: no action. I am (acid-abc-db-0), the leader with the lock
2022-09-18 10:23:35,591 INFO: no action. I am (acid-abc-db-0), the leader with the lock
</code></pre>
<p>operator logs</p>
<pre><code># too long and almost all of the logs are operator
# creating stuff that are mostly debug and info
# except the error of pod label updating
# ... more omits

... level=error msg=&quot;failed to create cluster: pod labels error: still failing after 200 retries&quot; cluster-name=default/acid-abc-db pkg=cluster worker=1
... level=error msg=&quot;could not create cluster: pod labels error: still failing after 200 retries&quot; cluster-name=default/acid-abc-db pkg=controller worker=1

# ... more omits
</code></pre>
<pre class="lang-yaml prettyprint-override"><code># /home/postgres/.config/patroni/patronictl.yaml

bootstrap:
  clone_with_wale:
    command: envdir &quot;/run/etc/wal-e.d/env-clone-acid-abc-db&quot; python3 /scripts/clone_with_wale.py --recovery-target-time=&quot;&quot;
    recovery_conf:
      recovery_target_action: promote
      recovery_target_timeline: latest
      restore_command: envdir &quot;/run/etc/wal-e.d/env-clone-acid-abc-db&quot; timeout &quot;0&quot; /scripts/restore_command.sh &quot;%f&quot; &quot;%p&quot;
  dcs:
    loop_wait: 10
    maximum_lag_on_failover: 33554432
    postgresql:
      parameters:
        archive_mode: 'on'
        archive_timeout: 1800s
        autovacuum_analyze_scale_factor: 0.02
        autovacuum_max_workers: 5
        autovacuum_vacuum_scale_factor: 0.05
        checkpoint_completion_target: '0.9'
        default_statistics_target: '100'
        effective_io_concurrency: '200'
        hot_standby: 'on'
        log_autovacuum_min_duration: 0
        log_checkpoints: 'on'
        log_connections: 'on'
        log_disconnections: 'on'
        log_line_prefix: '%t [%p]: [%l-1] %c %x %d %u %a %h '
        log_lock_waits: 'on'
        log_min_duration_statement: 500
        log_statement: all
        log_temp_files: 0
        max_connections: '512'
        max_parallel_maintenance_workers: '2'
        max_parallel_workers: '32'
        max_parallel_workers_per_gather: '8'
        max_replication_slots: 10
        max_slot_wal_keep_size: 16GB
        max_standby_archive_delay: 0s
        max_standby_streaming_delay: 0s
        max_wal_senders: '16'
        max_wal_size: 4GB
        max_worker_processes: '256'
        min_wal_size: 1GB
        tcp_keepalives_idle: 900
        tcp_keepalives_interval: 100
        track_functions: all
        wal_compression: 'on'
        wal_level: hot_standby
        wal_log_hints: 'on'
      use_pg_rewind: true
      use_slots: true
    retry_timeout: 10
    synchronous_node_count: 1
    ttl: 30
  initdb:
  - auth-host: md5
  - auth-local: trust
  - data-checksums
  - encoding: UTF8
  - locale: en_US.UTF-8
  method: clone_with_wale
  post_init: /scripts/post_init.sh &quot;zalandos&quot;
  users:
    zalandos:
      options:
      - CREATEDB
      - NOLOGIN
      password: ''
kubernetes:
  bypass_api_service: true
  labels:
    application: spilo
  port: tcp://10.56.0.1:443
  port_443_tcp: tcp://10.56.0.1:443
  port_443_tcp_addr: 10.56.0.1
  port_443_tcp_port: '443'
  port_443_tcp_proto: tcp
  ports:
  - name: postgresql
    port: 5432
  role_label: spilo-role
  scope_label: cluster-name
  service_host: 10.56.0.1
  service_port: '443'
  service_port_https: '443'
  use_endpoints: true
postgresql:
  authentication:
    replication:
      password: xxx
      username: standby
    superuser:
      password: xxx
      username: postgres
  basebackup_fast_xlog:
    command: /scripts/basebackup.sh
    retries: 2
  bin_dir: /usr/lib/postgresql/14/bin
  callbacks:
    on_role_change: /scripts/on_role_change.sh zalandos true
  connect_address: 10.52.5.55:5432
  create_replica_method:
  - wal_e
  - basebackup_fast_xlog
  data_dir: /home/postgres/pgdata/pgroot/data
  listen: '*:5432'
  name: acid-abc-db-0
  parameters:
    archive_command: envdir &quot;/run/etc/wal-e.d/env&quot; wal-g wal-push &quot;%p&quot;
    bg_mon.history_buckets: 120
    bg_mon.listen_address: 0.0.0.0
    extwlist.custom_path: /scripts
    extwlist.extensions: btree_gin,btree_gist,citext,extra_window_functions,first_last_agg,hll,hstore,hypopg,intarray,ltree,pgcrypto,pgq,pgq_node,pg_trgm,postgres_fdw,tablefunc,uuid-ossp,timescaledb,pg_partman
    log_destination: csvlog
    log_directory: ../pg_log
    log_file_mode: '0644'
    log_filename: postgresql-%u.log
    log_rotation_age: 1d
    log_truncate_on_rotation: 'on'
    logging_collector: 'on'
    pg_stat_statements.track_utility: 'off'
    shared_buffers: 256MB
    shared_preload_libraries: bg_mon,pg_stat_statements,pgextwlist,pg_auth_mon,set_user,timescaledb,pg_cron,pg_stat_kcache
    ssl: 'on'
    ssl_cert_file: /run/certs/server.crt
    ssl_key_file: /run/certs/server.key
  pg_hba:
  - local   all             all                                   trust
  - hostssl all             +zalandos    127.0.0.1/32       pam
  - host    all             all                127.0.0.1/32       md5
  - hostssl all             +zalandos    ::1/128            pam
  - host    all             all                ::1/128            md5
  - local   replication     standby                    trust
  - hostssl replication     standby all                md5
  - hostnossl all           all                all                reject
  - hostssl all             +zalandos    all                pam
  - hostssl all             all                all                md5
  pgpass: /run/postgresql/pgpass
  recovery_conf:
    restore_command: envdir &quot;/run/etc/wal-e.d/env&quot; timeout &quot;0&quot; /scripts/restore_command.sh &quot;%f&quot; &quot;%p&quot;
  use_unix_socket: true
  use_unix_socket_repl: true
  wal_e:
    command: envdir /run/etc/wal-e.d/env bash /scripts/wale_restore.sh
    no_master: 1
    retries: 2
    threshold_backup_size_percentage: 30
    threshold_megabytes: 102400
restapi:
  connect_address: 10.52.5.55:8008
  listen: :8008
scope: acid-abc-db
</code></pre>
<p>operator configuration</p>
<pre class="lang-yaml prettyprint-override"><code># mostly defaults
# only change common pod secret
# for backups credential
---
apiVersion: acid.zalan.do/v1
configuration:
  aws_or_gcp:
    additional_secret_mount_path: /meta/credentials
    aws_region: ap-southeast-1
    enable_ebs_gp3_migration: false
    enable_ebs_gp3_migration_max_size: 1000
  connection_pooler:
    connection_pooler_default_cpu_limit: &quot;1&quot;
    connection_pooler_default_cpu_request: 500m
    connection_pooler_default_memory_limit: 100Mi
    connection_pooler_default_memory_request: 512Mi
    connection_pooler_image: registry.opensource.zalan.do/acid/pgbouncer:master-22
    connection_pooler_max_db_connections: 512
    connection_pooler_mode: transaction
    connection_pooler_number_of_instances: 2
    connection_pooler_schema: pooler
    connection_pooler_user: pooler
  debug:
    debug_logging: true
    enable_database_access: true
  docker_image: registry.opensource.zalan.do/acid/spilo-14:2.1-p5
  enable_crd_registration: true
  enable_crd_validation: true
  enable_lazy_spilo_upgrade: false
  enable_pgversion_env_var: true
  enable_shm_volume: true
  enable_spilo_wal_path_compat: false
  enable_team_id_clustername_prefix: false
  etcd_host: &quot;&quot;
  kubernetes:
    cluster_domain: cluster.local
    cluster_labels:
      application: spilo
    cluster_name_label: cluster-name
    enable_cross_namespace_secret: false
    enable_init_containers: true
    enable_pod_antiaffinity: true
    enable_pod_disruption_budget: true
    enable_sidecars: true
    master_pod_move_timeout: 20m
    oauth_token_secret_name: postgresql-operator
    pdb_name_format: postgres-{cluster}-pdb
    pod_antiaffinity_topology_key: kubernetes.io/hostname
    pod_environment_secret: postgres-common-secret
    pod_management_policy: ordered_ready
    pod_role_label: spilo-role
    pod_service_account_definition: &quot;&quot;
    pod_service_account_name: postgres-pod
    pod_service_account_role_binding_definition: &quot;&quot;
    pod_terminate_grace_period: 5m
    secret_name_template: '{username}.{cluster}.credentials.{tprkind}.{tprgroup}'
    spilo_allow_privilege_escalation: true
    spilo_privileged: false
    storage_resize_mode: pvc
  kubernetes_use_configmaps: false
  load_balancer:
    db_hosted_zone: db.example.com
    enable_master_load_balancer: false
    enable_master_pooler_load_balancer: false
    enable_replica_load_balancer: false
    enable_replica_pooler_load_balancer: false
    external_traffic_policy: Cluster
    master_dns_name_format: '{cluster}.{team}.{hostedzone}'
    replica_dns_name_format: '{cluster}-repl.{team}.{hostedzone}'
  logging_rest_api:
    api_port: 8080
    cluster_history_entries: 1000
    ring_log_lines: 100
  logical_backup:
    logical_backup_docker_image: registry.opensource.zalan.do/acid/logical-backup:v1.8.1
    logical_backup_job_prefix: logical-backup-
    logical_backup_provider: s3
    logical_backup_s3_bucket: my-bucket-url
    logical_backup_s3_sse: AES256
    logical_backup_schedule: 30 00 * * *
  major_version_upgrade:
    major_version_upgrade_mode: &quot;off&quot;
    minimal_major_version: &quot;9.6&quot;
    target_major_version: &quot;14&quot;
  max_instances: -1
  min_instances: -1
  postgres_pod_resources:
    default_cpu_limit: &quot;1&quot;
    default_cpu_request: 100m
    default_memory_limit: 500Mi
    default_memory_request: 100Mi
    min_cpu_limit: 250m
    min_memory_limit: 250Mi
  repair_period: 5m
  resync_period: 30m
  set_memory_request_to_limit: false
  teams_api:
    enable_admin_role_for_users: true
    enable_postgres_team_crd: true
    enable_postgres_team_crd_superusers: false
    enable_team_member_deprecation: false
    enable_team_superuser: false
    enable_teams_api: false
    pam_configuration: https://info.example.com/oauth2/tokeninfo?access_token= uid
      realm=/employees
    pam_role_name: zalandos
    protected_role_names:
    - admin
    - cron_admin
    role_deletion_suffix: _deleted
    team_admin_role: admin
    team_api_role_configuration:
      log_statement: all
    teams_api_url: https://teams.example.com/api/
  timeouts:
    patroni_api_check_interval: 1s
    patroni_api_check_timeout: 5s
    pod_deletion_wait_timeout: 10m
    pod_label_wait_timeout: 10m
    ready_wait_interval: 4s
    ready_wait_timeout: 30s
    resource_check_interval: 3s
    resource_check_timeout: 10m
  users:
    enable_password_rotation: false
    password_rotation_interval: 90
    password_rotation_user_retention: 180
    replication_username: standby
    super_username: postgres
  workers: 8
kind: OperatorConfiguration
metadata:
  name: postgresql-operator-default-configuration
  namespace: default
</code></pre>
<p>Thanks and appreciate for your time on reading this and thanks in advance for guiding to debug this and helping.</p>
<h3>update, 0</h3>
<p>So I tried manually calling pod patch with <code>curl</code> within one of the pod, and it works as expected</p>
<p><code>curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt   --header &quot;Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)&quot; https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}/api/v1/namespaces/default/pods/acid-abc-db-1 -X PATCH -H 'Content-Type: application/merge-patch+json' -d '{&quot;metadata&quot;: {&quot;labels&quot;: {&quot;spilo-role&quot;: &quot;replica&quot;}}}'</code></p>
<p>and then <code>endpoints</code> of replica became available and connection pooler was able to connect to replica, so why is <code>patroni</code> calling patch to pod result in gateway error, and also <code>patronictl list</code> still did not show in an expected correct result(host is still empty and state not show running)</p>
<h3>update, 1</h3>
<p>so <code>patronictl list</code> depends on kubernetes's <code>annotations</code> for that, so I copy the request from google cloud log explorer that has the pod patch error, and replace the curl command with this</p>
<pre><code>curl \
  --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\
  --header &quot;Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)&quot;\
  https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}/api/v1/namespaces/default/pods/acid-abc-db-0\
  -X PATCH\
  -H 'Content-Type: application/merge-patch+json'\
  -d '{&quot;metadata&quot;: {&quot;name&quot;: &quot;acid-abc-db-1&quot;, &quot;namespace&quot;: &quot;default&quot;, &quot;annotations&quot;: {&quot;status&quot;: &quot;{\&quot;conn_url\&quot;:\&quot;postgres://10.52.3.48:5432/postgres\&quot;,\&quot;api_url\&quot;:\&quot;http://10.52.3.48:8008/patroni\&quot;,\&quot;state\&quot;:\&quot;running\&quot;,\&quot;role\&quot;:\&quot;replica\&quot;,\&quot;version\&quot;:\&quot;2.1.4\&quot;,\&quot;xlog_location\&quot;:77275856896,\&quot;timeline\&quot;:37}&quot;}, &quot;labels&quot;: {&quot;spilo-role&quot;: &quot;replica&quot;}}}'
</code></pre>
<p>now <code>patronictl list</code> showing this</p>
<pre><code>+ Cluster: acid-abc-db (7109013759578136647) -----+----+-----------+
| Member         | Host       | Role    | State   | TL | Lag in MB |
+----------------+------------+---------+---------+----+-----------+
| acid-abc-db-0  |            | Leader  |         |    |           |
| acid-abc-db-1  | 10.52.3.48 | Replica | running | 37 |        16 |
+----------------+------------+---------+---------+----+-----------+
</code></pre>

shadowlegend
<p>so the solution turns out to be quite simple, so I explore <a href="https://patroni.readthedocs.io/en/latest/SETTINGS.html#settings" rel="nofollow noreferrer">patroni doc</a> and realizes that there are options to configure dcs related settings and as I tested with normal <code>curl</code> request which working fine so permission should not be an issue and <code>504</code> error might be related to timeout so I explore the doc to find if I could configure the request timeout on this, thats what led me to trying out some of the option from patroni doc. So I update <code>Postgresql</code> k8s api object like below</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: &quot;acid.zalan.do/v1&quot;
kind: postgresql
metadata:
  name: acid-abc-db
  namespace: default
spec:
  # more omitted
  patroni:
    retry_timeout: 128 # default only 10, change to 128
  * more omitted
</code></pre>
<p>now the cluster is working fine again, thanks for everyone that took your time to read about my problem and sorry for which silly mistake</p>
