{"action":"created","issue":{"url":"https://api.github.com/repos/zalando-incubator/postgres-operator/issues/349","repository_url":"https://api.github.com/repos/zalando-incubator/postgres-operator","labels_url":"https://api.github.com/repos/zalando-incubator/postgres-operator/issues/349/labels{/name}","comments_url":"https://api.github.com/repos/zalando-incubator/postgres-operator/issues/349/comments","events_url":"https://api.github.com/repos/zalando-incubator/postgres-operator/issues/349/events","html_url":"https://github.com/zalando-incubator/postgres-operator/issues/349","id":342895215,"node_id":"MDU6SXNzdWUzNDI4OTUyMTU=","number":349,"title":"Cloning a cluster from wale s3 using a different name errors with no backups found","user":{"login":"valer-cara","id":2016345,"node_id":"MDQ6VXNlcjIwMTYzNDU=","avatar_url":"https://avatars1.githubusercontent.com/u/2016345?v=4","gravatar_id":"","url":"https://api.github.com/users/valer-cara","html_url":"https://github.com/valer-cara","followers_url":"https://api.github.com/users/valer-cara/followers","following_url":"https://api.github.com/users/valer-cara/following{/other_user}","gists_url":"https://api.github.com/users/valer-cara/gists{/gist_id}","starred_url":"https://api.github.com/users/valer-cara/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/valer-cara/subscriptions","organizations_url":"https://api.github.com/users/valer-cara/orgs","repos_url":"https://api.github.com/users/valer-cara/repos","events_url":"https://api.github.com/users/valer-cara/events{/privacy}","received_events_url":"https://api.github.com/users/valer-cara/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-07-19T21:26:11Z","updated_at":"2018-07-23T13:58:12Z","closed_at":"2018-07-23T13:58:12Z","author_association":"NONE","body":"#### TL;DR\r\n- Have an original cluster's (named `web-pg`) backups, and trying to launch a clone (`web-pg-zzzz`). \r\n- Patroni seems to be calling `patroni_wale_restore` with the wrong envdir, namely the one for the normal run (`./etc/wal-e.d/env`), not the env for the cluster to be cloned (present at `./etc/wal-e.d/env-clone-web-pg/`), and finds no backups to bootstrap from\r\n\r\n---------\r\n\r\n##### ps fauxw\r\n```\r\npostgres    32  0.2  0.1 1296036 57692 ?       Sl   16:22   0:34 /usr/bin/python3 /usr/local/bin/patroni /home/postgres/postgres.yml\r\npostgres 15640 21.0  0.0 145096 26088 ?        S    21:02   0:00  \\_ /usr/bin/python3 /usr/local/bin/patroni_wale_restore --envdir=/home/postgres/etc/wal-e.d/env --no_master=1 --retries=2 --threshold_backup_size_percentage=30 --threshold_megabytes=1024 --use_iam=1 --scope=web-zzzz --role=replica --datadir=/home/postgres/pgdata/pgroot/data --connstring=\r\npostgres 15643  9.0  0.0  37408 11636 ?        R    21:02   0:00      \\_ /usr/bin/python3 /usr/local/bin/wal-e --aws-instance-profile backup-list --detail LATEST\r\n```\r\n\r\n##### the 2 env dirs\r\n```\r\nroot@web-zzzz-0:/home/postgres# cat ./etc/wal-e.d/env/WALE_S3_PREFIX\r\ns3://xxxxxx/spilo/web-zzzz/1ce0b55b-8b6e-11e8-a154-122d5c497f7c/wal/\r\n\r\nroot@web-zzzz-0:/home/postgres# cat ./etc/wal-e.d/env-clone-web-pg/WALE_S3_PREFIX\r\ns3://xxxxxx/spilo/web-pg/1fd9f2db-4346-11e8-824b-12d5a13671f4/wal/\r\n```\r\n\r\n##### pod logs\r\n\r\n```\r\nweb-zzzz-0 postgres 2018-07-19 21:20:22,352 INFO: Lock owner: None; I am web-zzzz-0\r\nweb-zzzz-0 postgres 2018-07-19 21:20:22,352 INFO: trying to bootstrap (without leader)\r\nweb-zzzz-0 postgres 2018-07-19 21:20:22,575 DEBUG: calling ['envdir', '/home/postgres/etc/wal-e.d/env', 'wal-e', '--aws-instance-profile', 'backup-list', '--detail', 'LATEST']\r\nweb-zzzz-0 postgres wal_e.main   INFO     MSG: starting WAL-E\r\nweb-zzzz-0 postgres         DETAIL: The subcommand is \"backup-list\".\r\nweb-zzzz-0 postgres         STRUCTURED: time=2018-07-19T21:20:22.776599-00 pid=16613\r\nweb-zzzz-0 postgres wal_e.blobstore.s3.s3_util WARNING  MSG: WALE_S3_ENDPOINT defined, ignoring AWS_REGION\r\nweb-zzzz-0 postgres         HINT: AWS_REGION is only intended for use with AWS S3, and not interface-compatible use cases supported by WALE_S3_ENDPOINT\r\nweb-zzzz-0 postgres         STRUCTURED: time=2018-07-19T21:20:22.812539-00 pid=16613\r\nweb-zzzz-0 postgres 2018-07-19 21:20:22,877 DEBUG: b'name\\tlast_modified\\texpanded_size_bytes\\twal_segment_backup_start\\twal_segment_offset_backup_start\\twal_segment_backup_stop\\twal\r\n_segment_offset_backup_stop\\r\\n'                                                                                                                                                      web-zzzz-0 postgres 2018-07-19 21:20:22,877 WARNING: wal-e did not find any backups\r\nweb-zzzz-0 postgres 2018-07-19 21:20:22,877 DEBUG: exit_code is 2, not retrying\r\nweb-zzzz-0 postgres 2018-07-19 21:20:22,897 ERROR: Error creating replica using method wal_e: patroni_wale_restore exited with code=2\r\nweb-zzzz-0 postgres 2018-07-19 21:20:22,897 ERROR: failed to bootstrap (without leader)\r\nweb-zzzz-0 postgres 2018-07-19 21:20:22,897 INFO: Removing data directory: /home/postgres/pgdata/pgroot/data\r\n\r\n```\r\n\r\n##### AWS backups are available\r\n\r\n```\r\n$ aws s3 ls s3://xxxxxx/spilo/web-pg/1fd9f2db-4346-11e8-824b-12d5a13671f4/wal/\r\n                           PRE basebackups_005/\r\n                           PRE wal_005/\r\n\r\n```\r\n\r\n##### test cluster manifest\r\n \r\n```\r\napiVersion: \"acid.zalan.do/v1\"\r\nkind: postgresql\r\n\r\nmetadata:\r\n  name: web-zzzz\r\n  namespace: app\r\nspec:\r\n  teamId: \"web\"\r\n  dockerImage: \"registry.opensource.zalan.do/acid/spilo-10:1.4-p11\"\r\n  clone:\r\n    uid: \"1fd9f2db-4346-11e8-824b-12d5a13671f4\"\r\n    cluster: \"web-pg\"\r\n    timestamp: \"2018-07-19T10:00:00+00:00\"\r\n\r\n------- SNIP -------\r\n```\r\n\r\nThanks for checking this out!"},"comment":{"url":"https://api.github.com/repos/zalando-incubator/postgres-operator/issues/comments/407067160","html_url":"https://github.com/zalando-incubator/postgres-operator/issues/349#issuecomment-407067160","issue_url":"https://api.github.com/repos/zalando-incubator/postgres-operator/issues/349","id":407067160,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNzA2NzE2MA==","user":{"login":"valer-cara","id":2016345,"node_id":"MDQ6VXNlcjIwMTYzNDU=","avatar_url":"https://avatars1.githubusercontent.com/u/2016345?v=4","gravatar_id":"","url":"https://api.github.com/users/valer-cara","html_url":"https://github.com/valer-cara","followers_url":"https://api.github.com/users/valer-cara/followers","following_url":"https://api.github.com/users/valer-cara/following{/other_user}","gists_url":"https://api.github.com/users/valer-cara/gists{/gist_id}","starred_url":"https://api.github.com/users/valer-cara/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/valer-cara/subscriptions","organizations_url":"https://api.github.com/users/valer-cara/orgs","repos_url":"https://api.github.com/users/valer-cara/repos","events_url":"https://api.github.com/users/valer-cara/events{/privacy}","received_events_url":"https://api.github.com/users/valer-cara/received_events","type":"User","site_admin":false},"created_at":"2018-07-23T13:58:12Z","updated_at":"2018-07-23T13:58:12Z","author_association":"NONE","body":"All confusion aside, recreating the cluster from scratch (delete pg, create pg) had it bootstrapping from the wale archives. I'll close this issue since it seems it was a weird un-reproducible edge case."}}
{"actor":{"display_login":"valer-cara"}}