{"action":"created","issue":{"url":"https://api.github.com/repos/hashicorp/consul/issues/4954","repository_url":"https://api.github.com/repos/hashicorp/consul","labels_url":"https://api.github.com/repos/hashicorp/consul/issues/4954/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/consul/issues/4954/comments","events_url":"https://api.github.com/repos/hashicorp/consul/issues/4954/events","html_url":"https://github.com/hashicorp/consul/issues/4954","id":380530390,"node_id":"MDU6SXNzdWUzODA1MzAzOTA=","number":4954,"title":"[ERR] consul: failed to establish leadership: unknown CA provider \"\"","user":{"login":"WyseNynja","id":624221,"node_id":"MDQ6VXNlcjYyNDIyMQ==","avatar_url":"https://avatars2.githubusercontent.com/u/624221?v=4","gravatar_id":"","url":"https://api.github.com/users/WyseNynja","html_url":"https://github.com/WyseNynja","followers_url":"https://api.github.com/users/WyseNynja/followers","following_url":"https://api.github.com/users/WyseNynja/following{/other_user}","gists_url":"https://api.github.com/users/WyseNynja/gists{/gist_id}","starred_url":"https://api.github.com/users/WyseNynja/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/WyseNynja/subscriptions","organizations_url":"https://api.github.com/users/WyseNynja/orgs","repos_url":"https://api.github.com/users/WyseNynja/repos","events_url":"https://api.github.com/users/WyseNynja/events{/privacy}","received_events_url":"https://api.github.com/users/WyseNynja/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-11-14T04:24:52Z","updated_at":"2018-11-14T05:35:31Z","closed_at":null,"author_association":"NONE","body":"#### Overview of the Issue\r\n\r\nI'm trying to use consul connect and now my cluster is partially broken.\r\n\r\n#### Reproduction Steps\r\n\r\nI'm not sure how exactly to reproduce. I've been poking at this cluster too much to have clear steps. I'll try more tomorrow.\r\n\r\nI put this in `/etc/consul.d/config.hcl`:\r\n\r\n```\r\nconnect {\r\n  enabled = true\r\n}\r\n```\r\n\r\nWhen I upgraded consul from 1.2.3 to 1.3.0 and added that config, something went wrong and leader election failed.\r\n\r\nI manually recovered by poking peers.json and now the cluster has a leader:\r\n```\r\nNode      ID                                    Address             State     Voter  RaftProtocol\r\nconsul    375e6536-a4d0-5770-3b7d-98dbe4a65686  192.168.0.100:8300  follower  true   3\r\nconsul-a  65000f53-2857-4daf-19b8-61e5bb4492c0  192.168.0.101:8300  follower  true   3\r\nconsul-b  ef31e65f-4536-14c0-4c8d-fdb1f6725922  192.168.0.102:8300  leader    true   3\r\n```\r\n\r\nHowever, something is still wrong with the cluster. The UI is showing stale data (https://github.com/hashicorp/consul/issues/4923, maybe?) and vault is stuck in standby.\r\n\r\n```\r\n$ vault status\r\nKey                    Value\r\n---                    -----\r\nSeal Type              shamir\r\nInitialized            true\r\nSealed                 false\r\nTotal Shares           1\r\nThreshold              1\r\nVersion                0.11.3\r\nCluster Name           vault-cluster-641f43ea\r\nCluster ID             e9570865-65db-1ea6-738b-c759d68fbfdd\r\nHA Enabled             true\r\nHA Cluster             n/a\r\nHA Mode                standby\r\nActive Node Address    <none>\r\n\r\n$ vault secrets enable pki\r\nError enabling: Error making API request.\r\n\r\nURL: POST http://127.0.0.1:8200/v1/sys/mounts/pki\r\nCode: 500. Errors:\r\n\r\n* local node not active but active cluster node not found\r\n```\r\n\r\nI would like to use vault for consul connect, but was just trying to get it working with the simplest setup first. How do I migrate to vault from this broken connect provider?\r\n\r\nFrom my reading of the docs, the ca stuff was all supposed to be automatic so I don't know how to do it manually. I think I need to run `consul connect ca set-config -config-file ca.json` but don't know what to put for ca.json (also, why no hcl support here?).\r\n\r\n### Consul info for both Client and Server\r\n\r\n<details>\r\n  <summary>Client info</summary>\r\n\r\n```\r\nagent:\r\n\tcheck_monitors = 0\r\n\tcheck_ttls = 1\r\n\tchecks = 1\r\n\tservices = 1\r\nbuild:\r\n\tprerelease = \r\n\trevision = e8757838\r\n\tversion = 1.3.0\r\nconsul:\r\n\tknown_servers = 3\r\n\tserver = false\r\nruntime:\r\n\tarch = amd64\r\n\tcpu_count = 8\r\n\tgoroutines = 52\r\n\tmax_procs = 8\r\n\tos = linux\r\n\tversion = go1.11.1\r\nserf_lan:\r\n\tcoordinate_resets = 0\r\n\tencrypted = false\r\n\tevent_queue = 0\r\n\tevent_time = 168\r\n\tfailed = 0\r\n\thealth_score = 0\r\n\tintent_queue = 0\r\n\tleft = 0\r\n\tmember_time = 1708\r\n\tmembers = 5\r\n\tquery_queue = 0\r\n\tquery_time = 1\r\n```\r\n\r\n</details>\r\n\r\n<details>\r\n  <summary>Server info</summary>\r\n\r\n```\r\nagent:\r\n\tcheck_monitors = 0\r\n\tcheck_ttls = 0\r\n\tchecks = 0\r\n\tservices = 0\r\nbuild:\r\n\tprerelease = \r\n\trevision = e8757838\r\n\tversion = 1.3.0\r\nconsul:\r\n\tbootstrap = false\r\n\tknown_datacenters = 1\r\n\tleader = true\r\n\tleader_addr = 192.168.0.102:8300\r\n\tserver = true\r\nraft:\r\n\tapplied_index = 6311623\r\n\tcommit_index = 6311623\r\n\tfsm_pending = 0\r\n\tlast_contact = 0\r\n\tlast_log_index = 6311623\r\n\tlast_log_term = 897\r\n\tlast_snapshot_index = 6311248\r\n\tlast_snapshot_term = 700\r\n\tlatest_configuration = [{Suffrage:Voter ID:375e6536-a4d0-5770-3b7d-98dbe4a65686 Address:192.168.0.100:8300} {Suffrage:Voter ID:65000f53-2857-4daf-19b8-61e5bb4492c0 Address:192.168.0.101:8300} {Suffrage:Voter ID:ef31e65f-4536-14c0-4c8d-fdb1f6725922 Address:192.168.0.102:8300}]\r\n\tlatest_configuration_index = 1\r\n\tnum_peers = 2\r\n\tprotocol_version = 3\r\n\tprotocol_version_max = 3\r\n\tprotocol_version_min = 0\r\n\tsnapshot_version_max = 1\r\n\tsnapshot_version_min = 0\r\n\tstate = Leader\r\n\tterm = 897\r\nruntime:\r\n\tarch = amd64\r\n\tcpu_count = 8\r\n\tgoroutines = 94\r\n\tmax_procs = 8\r\n\tos = linux\r\n\tversion = go1.11.1\r\nserf_lan:\r\n\tcoordinate_resets = 0\r\n\tencrypted = false\r\n\tevent_queue = 0\r\n\tevent_time = 168\r\n\tfailed = 0\r\n\thealth_score = 0\r\n\tintent_queue = 0\r\n\tleft = 2\r\n\tmember_time = 1708\r\n\tmembers = 7\r\n\tquery_queue = 0\r\n\tquery_time = 1\r\nserf_wan:\r\n\tcoordinate_resets = 0\r\n\tencrypted = false\r\n\tevent_queue = 0\r\n\tevent_time = 1\r\n\tfailed = 0\r\n\thealth_score = 0\r\n\tintent_queue = 0\r\n\tleft = 0\r\n\tmember_time = 228\r\n\tmembers = 3\r\n\tquery_queue = 0\r\n\tquery_time = 1\r\n```\r\n\r\n</details>\r\n\r\n### Operating system and Environment details\r\n\r\nI'm running everything inside docker containers on a single host.\r\n\r\n### Log Fragments\r\n\r\n```\r\nconsul_b_1  | bootstrap_expect > 0: expecting 3 servers\r\nconsul_b_1  |     2018/11/14 04:04:31 [ERR] consul: failed to establish leadership: unknown CA provider \"\"\r\nconsul_b_1  |     2018/11/14 04:05:31 [ERR] consul: failed to establish leadership: unknown CA provider \"\"\r\nconsul_b_1  |     2018/11/14 04:06:31 [ERR] consul: failed to establish leadership: unknown CA provider \"\"\r\nconsul_b_1  |     2018/11/14 04:07:31 [ERR] consul: failed to establish leadership: unknown CA provider \"\"\r\nconsul_b_1  |     2018/11/14 04:08:31 [ERR] consul: failed to establish leadership: unknown CA provider \"\"\r\nconsul_b_1  |     2018/11/14 04:09:31 [ERR] consul: failed to establish leadership: unknown CA provider \"\"\r\nconsul_b_1  |     2018/11/14 04:10:31 [ERR] consul: failed to establish leadership: unknown CA provider \"\"\r\nconsul_b_1  |     2018/11/14 04:11:31 [ERR] consul: failed to establish leadership: unknown CA provider \"\"\r\n```"},"comment":{"url":"https://api.github.com/repos/hashicorp/consul/issues/comments/438543688","html_url":"https://github.com/hashicorp/consul/issues/4954#issuecomment-438543688","issue_url":"https://api.github.com/repos/hashicorp/consul/issues/4954","id":438543688,"node_id":"MDEyOklzc3VlQ29tbWVudDQzODU0MzY4OA==","user":{"login":"kyhavlov","id":4177697,"node_id":"MDQ6VXNlcjQxNzc2OTc=","avatar_url":"https://avatars1.githubusercontent.com/u/4177697?v=4","gravatar_id":"","url":"https://api.github.com/users/kyhavlov","html_url":"https://github.com/kyhavlov","followers_url":"https://api.github.com/users/kyhavlov/followers","following_url":"https://api.github.com/users/kyhavlov/following{/other_user}","gists_url":"https://api.github.com/users/kyhavlov/gists{/gist_id}","starred_url":"https://api.github.com/users/kyhavlov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kyhavlov/subscriptions","organizations_url":"https://api.github.com/users/kyhavlov/orgs","repos_url":"https://api.github.com/users/kyhavlov/repos","events_url":"https://api.github.com/users/kyhavlov/events{/privacy}","received_events_url":"https://api.github.com/users/kyhavlov/received_events","type":"User","site_admin":false},"created_at":"2018-11-14T05:35:31Z","updated_at":"2018-11-14T05:35:31Z","author_association":"CONTRIBUTOR","body":"Hey @WyseNynja, thanks for the bug report. That config you gave for connect (`enabled = true`) should be enough to get things working in both 1.2.3 and 1.3.0, and the empty provider type *should* be an impossible state to get into since the provider defaults to \"consul\" in both of those versions. \r\n\r\nI tried a few things to reproduce the state you show where it's failing to establish leadership but couldn't get the same result:\r\n- Upgrading a lone server from 1.2.3 with Connect disabled to 1.3.0 with just the `connect = enabled` config.\r\n- A rolling upgrade of a set of 3 servers from 1.2.3 with Connect disabled to 1.3.0 with the above config.\r\n- Partially upgrading a set of 3 servers so that 2 were running 1.3.0 and the other was on 1.2.3, then getting one of the 1.3.0 servers to become leader and bootstrap the CA before restarting it to cause the 1.2.3 server to become leader again (in case some state was backwards incompatible here).\r\n\r\nAny steps you have to help reproduce this are appreciated. It may take a fresh cluster to do so if you've got things back to a working state though, as it sounds like the invalid CA config that was preventing the leader election has been fixed."}}
{"actor":{"display_login":"kyhavlov"}}