{"action":"created","issue":{"url":"https://api.github.com/repos/hashicorp/vault/issues/6769","repository_url":"https://api.github.com/repos/hashicorp/vault","labels_url":"https://api.github.com/repos/hashicorp/vault/issues/6769/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/vault/issues/6769/comments","events_url":"https://api.github.com/repos/hashicorp/vault/issues/6769/events","html_url":"https://github.com/hashicorp/vault/issues/6769","id":446835063,"node_id":"MDU6SXNzdWU0NDY4MzUwNjM=","number":6769,"title":"Duplicated pki mount entries, vault unable to start","user":{"login":"tstoermer","id":34302010,"node_id":"MDQ6VXNlcjM0MzAyMDEw","avatar_url":"https://avatars1.githubusercontent.com/u/34302010?v=4","gravatar_id":"","url":"https://api.github.com/users/tstoermer","html_url":"https://github.com/tstoermer","followers_url":"https://api.github.com/users/tstoermer/followers","following_url":"https://api.github.com/users/tstoermer/following{/other_user}","gists_url":"https://api.github.com/users/tstoermer/gists{/gist_id}","starred_url":"https://api.github.com/users/tstoermer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tstoermer/subscriptions","organizations_url":"https://api.github.com/users/tstoermer/orgs","repos_url":"https://api.github.com/users/tstoermer/repos","events_url":"https://api.github.com/users/tstoermer/events{/privacy}","received_events_url":"https://api.github.com/users/tstoermer/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/hashicorp/vault/milestones/52","html_url":"https://github.com/hashicorp/vault/milestone/52","labels_url":"https://api.github.com/repos/hashicorp/vault/milestones/52/labels","id":4269383,"node_id":"MDk6TWlsZXN0b25lNDI2OTM4Mw==","number":52,"title":"1.1.3","description":"","creator":{"login":"briankassouf","id":1916073,"node_id":"MDQ6VXNlcjE5MTYwNzM=","avatar_url":"https://avatars1.githubusercontent.com/u/1916073?v=4","gravatar_id":"","url":"https://api.github.com/users/briankassouf","html_url":"https://github.com/briankassouf","followers_url":"https://api.github.com/users/briankassouf/followers","following_url":"https://api.github.com/users/briankassouf/following{/other_user}","gists_url":"https://api.github.com/users/briankassouf/gists{/gist_id}","starred_url":"https://api.github.com/users/briankassouf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/briankassouf/subscriptions","organizations_url":"https://api.github.com/users/briankassouf/orgs","repos_url":"https://api.github.com/users/briankassouf/repos","events_url":"https://api.github.com/users/briankassouf/events{/privacy}","received_events_url":"https://api.github.com/users/briankassouf/received_events","type":"User","site_admin":false},"open_issues":0,"closed_issues":28,"state":"closed","created_at":"2019-04-29T15:10:37Z","updated_at":"2019-06-11T11:32:36Z","due_on":null,"closed_at":"2019-06-11T11:32:36Z"},"comments":1,"created_at":"2019-05-21T21:31:52Z","updated_at":"2019-09-11T08:38:17Z","closed_at":"2019-06-04T17:33:36Z","author_association":"NONE","body":"**Describe the bug**\r\nThis is the concise bug description after a lot of investigation - see additional context for detailed story.\r\nSpecial cases can cause a duplication of PKI mounts at same path. The PKI secret engine is used by clients with active certs and leases. The secret engine will be deleted with terraform destroy. The destroy takes up to 1 min for cleanup tasks and succeeds. Afterwards, the PKI secret engine is recreated with terraform apply. A vault server restart will now fail with error for duplicated mount:\r\n```\r\n2019-05-16T14:37:07.721Z [ERROR] core: failed to mount entry: path=pki-tsc-max-kube/ error=\"cannot mount under existing mount \"pki-tsc-max-kube/\"\"\r\n```\r\nThis is a dangerous bug, because the problem is not discovered during runtime. Only a vault restart discovers the issue. The delayed occurrence of the error can cause massive data loss, because a backup is required before the duplicated mount has been created.\r\n\r\n**To Reproduce**\r\nSteps to reproduce the behavior:\r\n1. Create a PKI secret engine and role definition with terraform.\r\n2. Run clients creating several active certs/keys with leases (~1,000)\r\n3. Delete the PKI secret engine with terraform destroy. The destroy operation will take up to 1 minute cleaning up all leases etc. Terraform destroy will succeed, but vault server shows error message like:\r\n```\r\n05.04.19, 10:24:15.373  10.52.187.109   vault           ERROR   secrets.system.system_ca91d9d4: unmount failed: path=pki-tsc-max-kube/ error=\"Delete https://127.0.0.1:8500/v1/kv/vault/logical/3c99d546-1b6a-437e-b5fe-b1e77f02bba7/revoked/58-e4-fb-ca-a8-98-59-8a-9d-9a-18-9c-25-27-9f-0c-15-45-45-12: context canceled\"\r\n05.04.19, 10:24:15.372  10.52.187.109   vault           ERROR   core: failed to clear view for path being unmounted: error=\"Delete https://127.0.0.1:8500/v1/kv/vault/logical/3c99d546-1b6a-437e-b5fe-b1e77f02bba7/revoked/58-e4-fb-ca-a8-98-59-8a-9d-9a-18-9c-25-27-9f-0c-15-45-45-12: context canceled\" path=pki-tsc-max-kube/\r\n```\r\n4. Recreate the PKI secret engine with terraform apply. The create will succeed.\r\n5. Restart the vault server and startup will fail with error:\r\n```\r\n2019-05-16T14:37:07.721Z [ERROR] core: failed to mount entry: path=pki-tsc-max-kube/ error=\"cannot mount under existing mount \"pki-tsc-max-kube/\"\"\r\n```\r\n\r\n**Expected behavior**\r\n1. Vault must always fail creation of duplicated secret engine mounts at same path. Even if the secret engine delete operation failed and is leaving inconsistent mount points.\r\n2. Terraform PKI  secret engine destroy must fail if vault delete operation is not successful (this could also be a bug in the terraform vault provider).\r\n\r\n**Environment:**\r\n* Vault Server Version: 1.1.2 (docker container)\r\n* Consul Server Version: 1.4.2 (docker container)\r\n* Terraform v0.11.14 + provider.vault v1.8.0\r\n\r\nVault server configuration file(s):\r\n\r\n```hcl\r\nbackend \"consul\" {\r\n  address = \"######\"\r\n  scheme = \"https\"\r\n  path = \"vault/\"\r\n  token = \"###########\"\r\n  tls_cert_file = \"/#####/####.pem\"\r\n  tls_key_file = \"/#####/####.pem\"\r\n  tls_ca_file = \"/#####/####.pem\"\r\n}\r\n\r\nui = \"true\"\r\n\r\nseal \"awskms\" {\r\n  region = \"#############\"\r\n  kms_key_id = \"###########\"\r\n}\r\n\r\nlistener \"tcp\" {\r\n  address = \"######\"\r\n  cluster_address = \"######\"\r\n  tls_cert_file = \"/#####/####.pem\"\r\n  tls_key_file = \"/#####/####.pem\"\r\n}\r\n\r\ntelemetry {\r\n  prometheus_retention_time = \"24h\",\r\n  disable_hostname = true\r\n}\r\n\r\napi_addr = \"https://#####\"\r\n```\r\n\r\n**Additional context**\r\nThe error occurred during a vault upgrade from version 1.0.2 to 1.1.2. The vault server start failed with error:\r\n```\r\n2019-05-16T14:37:07.721Z [ERROR] core: failed to mount entry: path=pki-tsc-max-kube/ error=\"cannot mount under existing mount \"pki-tsc-max-kube/\"\"\r\n```\r\nFirst attempt was to restore a backup before the update started and using old vault server version. But the error still occurred. In the next steps older backups have been tried until **a 1 month old one** worked. Consequently, the issue must not be related to the vault upgrade.\r\nBased on the error we assumed a PKI mount delete and create operation must have caused the problem. We analyzed vault audit log and found related mount operations. Based on this, we tested the expected last working and first failing backup, that proved our thesis. \r\nNow I reproduced the issue with assumption, heavy cleanup of certs and leases must cause the problem during delete. "},"comment":{"url":"https://api.github.com/repos/hashicorp/vault/issues/comments/530281715","html_url":"https://github.com/hashicorp/vault/issues/6769#issuecomment-530281715","issue_url":"https://api.github.com/repos/hashicorp/vault/issues/6769","id":530281715,"node_id":"MDEyOklzc3VlQ29tbWVudDUzMDI4MTcxNQ==","user":{"login":"DrMagPie","id":23652900,"node_id":"MDQ6VXNlcjIzNjUyOTAw","avatar_url":"https://avatars0.githubusercontent.com/u/23652900?v=4","gravatar_id":"","url":"https://api.github.com/users/DrMagPie","html_url":"https://github.com/DrMagPie","followers_url":"https://api.github.com/users/DrMagPie/followers","following_url":"https://api.github.com/users/DrMagPie/following{/other_user}","gists_url":"https://api.github.com/users/DrMagPie/gists{/gist_id}","starred_url":"https://api.github.com/users/DrMagPie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DrMagPie/subscriptions","organizations_url":"https://api.github.com/users/DrMagPie/orgs","repos_url":"https://api.github.com/users/DrMagPie/repos","events_url":"https://api.github.com/users/DrMagPie/events{/privacy}","received_events_url":"https://api.github.com/users/DrMagPie/received_events","type":"User","site_admin":false},"created_at":"2019-09-11T08:38:17Z","updated_at":"2019-09-11T08:38:17Z","author_association":"NONE","body":"Hello, I have faced same issue but with kv backend in Vault version 1.2.2\r\nhere are log messages\r\n```\r\n2019-09-10T13:05:36.293Z [INFO]  core: successfully mounted backend: type=kv path=test-kv/\r\n2019-09-10T13:05:36.293Z [INFO]  core: successfully mounted backend: type=kv path=secret/\r\n2019-09-10T13:05:36.293Z [INFO]  core: successfully mounted backend: type=system path=sys/\r\n2019-09-10T13:05:36.294Z [INFO]  core: successfully mounted backend: type=identity path=identity/\r\n2019-09-10T13:05:36.294Z [ERROR] core: failed to mount entry: path=test-kv/ error=\"cannot mount under existing mount \"test-kv/\"\"\r\n```\r\n\r\nWhen this issue occurred Vault started to elect leaded and was failing to do so.\r\nOnly thing that helped to restore cluster to operating status was restoring older consul snapshot."}}
{"actor":{"display_login":"DrMagPie"}}