{"action":"created","issue":{"url":"https://api.github.com/repos/turbinelabs/rotor/issues/16","repository_url":"https://api.github.com/repos/turbinelabs/rotor","labels_url":"https://api.github.com/repos/turbinelabs/rotor/issues/16/labels{/name}","comments_url":"https://api.github.com/repos/turbinelabs/rotor/issues/16/comments","events_url":"https://api.github.com/repos/turbinelabs/rotor/issues/16/events","html_url":"https://github.com/turbinelabs/rotor/issues/16","id":336476621,"node_id":"MDU6SXNzdWUzMzY0NzY2MjE=","number":16,"title":"Not able to get healthy hosts using ROTOR","user":{"login":"vgomprakash","id":14318585,"node_id":"MDQ6VXNlcjE0MzE4NTg1","avatar_url":"https://avatars2.githubusercontent.com/u/14318585?v=4","gravatar_id":"","url":"https://api.github.com/users/vgomprakash","html_url":"https://github.com/vgomprakash","followers_url":"https://api.github.com/users/vgomprakash/followers","following_url":"https://api.github.com/users/vgomprakash/following{/other_user}","gists_url":"https://api.github.com/users/vgomprakash/gists{/gist_id}","starred_url":"https://api.github.com/users/vgomprakash/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vgomprakash/subscriptions","organizations_url":"https://api.github.com/users/vgomprakash/orgs","repos_url":"https://api.github.com/users/vgomprakash/repos","events_url":"https://api.github.com/users/vgomprakash/events{/privacy}","received_events_url":"https://api.github.com/users/vgomprakash/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-06-28T05:22:38Z","updated_at":"2018-06-28T12:55:54Z","closed_at":null,"author_association":"NONE","body":"Hi Team, \r\n\r\n I am trying to integrate with Turbine Labs rotor for Envoy xDS. \r\nI was able to bring up , Rotor, and Envoy(configured to talk to Rotor).\r\n\r\n\r\n\r\nIssue: The hosts though are not getting updated to Envoy . \r\n\r\nHere are my logs from Rotor and Envoy, any help here would be deeply appreciated to figure out why the hosts are not getting updated to Envoy.\r\n\r\n**Envoy is brought as below**\r\ndocker run -it -d --name envoy-rotor --link rotor -v /envoy/envoy-rotor.yaml:/etc/envoy-rotor.yaml -v /var/tmp/:/var/tmp/ -p 8005:8005  -p 8001:8001 envoyproxy/envoy envoy --v2-config-only -c /etc/envoy-rotor.yaml \r\n\r\n[Envoy.txt](https://github.com/turbinelabs/rotor/files/2143874/Envoy.txt)\r\n\r\n**Rotor is brought as below**\r\ndocker run -it --name rotor -d  -e \"ROTOR_CMD=consul\" -e \"ROTOR_CONSUL_DC=xxx\"  -e \"ROTOR_CONSUL_HOSTPORT=consul_ip:8500\" -e \"ROTOR_CONSOLE_LEVEL=trace\" -p 50000:50000  turbinelabs/rotor:0.17.2 \r\n\r\n**Rotor Logs:**\r\n\r\n2018-06-25T21:38:28.860117000Z [info] 2018/06/25 21:38:28 No API key specified, the API stats backend will not be configured.\r\n2018-06-25T21:38:28.864072000Z [info] 2018/06/25 21:38:28 serving xDS on [::]:50000\r\n2018-06-25T21:38:28.864689000Z [info] 2018/06/25 21:38:28 log streaming enabled\r\n018-06-25T21:55:29.202937000Z [info] 2018/06/25 21:55:29 respond type.googleapis.com/envoy.api.v2.ClusterLoadAssignment[envoyservice-1] version \"rh0E+4kL7sP/bq9Mchn6nQ==\" with version \"hppZLEmbKZg9WVOtbF1aQQ==\"\r\n2018-06-25T21:55:29.218037000Z [info] 2018/06/25 21:55:29 Stream 1, type.googleapis.com/envoy.api.v2.ClusterLoadAssignment: ack response version \"hppZLEmbKZg9WVOtbF1aQQ==\"\r\n2018-06-25T21:55:29.218281000Z [info] 2018/06/25 21:55:29 open watch 33 for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment[envoyservice-1] from nodeID \"{\\\"proxy_name\\\":\\\"default-cluster\\\",\\\"zone_name\\\":\\\"default-zone\\\"}\", version \"hppZLEmbKZg9WVOtbF1aQQ==\"\r\n2018-06-25T21:56:29.231249000Z [info] 2018/06/25 21:56:29 respond open watch 33[envoyservice-1] with new version \"HpiopUqw68kQRX56Z1eKIg==\"\r\n2018-06-25T21:56:29.231556000Z [info] 2018/06/25 21:56:29 respond type.googleapis.com/envoy.api.v2.ClusterLoadAssignment[envoyservice-1] version \"hppZLEmbKZg9WVOtbF1aQQ==\" with version \"HpiopUqw68kQRX56Z1eKIg==\"\r\n\r\n**Envoy Logs:\r\n\r\n**\r\n2018-06-25T21:55:20.720849000Z [2018-06-25 21:55:20.720][5][debug][upstream] source/common/network/dns_impl.cc:147] Setting DNS resolution timer for 3749 milliseconds\r\n2018-06-25T21:55:20.721800000Z [2018-06-25 21:55:20.721][5][debug][upstream] source/common/upstream/logical_dns_cluster.cc:77] async DNS resolution complete for rotor\r\n2018-06-25T21:55:24.683702000Z [2018-06-25 21:55:24.682][5][debug][main] source/server/server.cc:118] flushing stats\r\n2018-06-25T21:55:25.722346000Z [2018-06-25 21:55:25.721][5][debug][upstream] source/common/upstream/logical_dns_cluster.cc:69] starting async DNS resolution for rotor\r\n2018-06-25T21:55:25.722733000Z [2018-06-25 21:55:25.721][5][debug][upstream] source/common/network/dns_impl.cc:147] Setting DNS resolution timer for 3124 milliseconds\r\n2018-06-25T21:55:25.723396000Z [2018-06-25 21:55:25.723][5][debug][upstream] source/common/network/dns_impl.cc:147] Setting DNS resolution timer for 2811 milliseconds\r\n2018-06-25T21:55:25.724564000Z [2018-06-25 21:55:25.724][5][debug][upstream] source/common/network/dns_impl.cc:147] Setting DNS resolution timer for 3749 milliseconds\r\n2018-06-25T21:55:25.725642000Z [2018-06-25 21:55:25.725][5][debug][upstream] source/common/network/dns_impl.cc:147] Setting DNS resolution timer for 4061 milliseconds\r\n2018-06-25T21:55:25.726663000Z [2018-06-25 21:55:25.726][5][debug][upstream] source/common/upstream/logical_dns_cluster.cc:77] async DNS resolution complete for rotor\r\n2018-06-25T21:55:29.204203000Z [2018-06-25 21:55:29.202][5][trace][connection] source/common/network/connection_impl.cc:384] [C0] socket event: 3\r\n2018-06-25T21:55:29.204458000Z [2018-06-25 21:55:29.202][5][trace][connection] source/common/network/connection_impl.cc:452] [C0] write ready\r\n2018-06-25T21:55:29.204700000Z [2018-06-25 21:55:29.202][5][trace][connection] source/common/network/connection_impl.cc:422] [C0] read ready\r\n2018-06-25T21:55:29.204946000Z [2018-06-25 21:55:29.202][5][trace][connection] source/common/network/raw_buffer_socket.cc:21] [C0] read returns: 100\r\n2018-06-25T21:55:29.205206000Z [2018-06-25 21:55:29.202][5][trace][connection] source/common/network/raw_buffer_socket.cc:21] [C0] read returns: -1\r\n2018-06-25T21:55:29.205430000Z [2018-06-25 21:55:29.202][5][trace][connection] source/common/network/raw_buffer_socket.cc:29] [C0] read error: 11\r\n2018-06-25T21:55:29.205693000Z [2018-06-25 21:55:29.202][5][trace][http2] source/common/http/http2/codec_impl.cc:277] [C0] dispatching 100 bytes\r\n2018-06-25T21:55:29.205928000Z [2018-06-25 21:55:29.202][5][trace][http2] source/common/http/http2/codec_impl.cc:335] [C0] recv frame type=0\r\n2018-06-25T21:55:29.206166000Z [2018-06-25 21:55:29.202][5][trace][http] source/common/http/async_client_impl.cc:100] async http request response data (length=91 end_stream=false)\r\n2018-06-25T21:55:29.206403000Z [2018-06-25 21:55:29.202][5][debug][upstream] source/common/config/grpc_mux_impl.cc:160] Received gRPC message for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment at version hppZLEmbKZg9WVOtbF1aQQ==\r\n2018-06-25T21:55:29.206645000Z [2018-06-25 21:55:29.202][5][debug][upstream] source/common/upstream/eds.cc:51] Missing ClusterLoadAssignment for envoyservice-1 in onConfigUpdate()\r\n2018-06-25T21:55:29.206870000Z [2018-06-25 21:55:29.202][5][debug][config] bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:60] gRPC config for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment accepted with 0 resources: []\r\n2018-06-25T21:55:29.207093000Z [2018-06-25 21:55:29.202][5][trace][upstream] source/common/config/grpc_mux_impl.cc:80] Sending DiscoveryRequest for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment: version_info: \"hppZLEmbKZg9WVOtbF1aQQ==\"\r\n2018-06-25T21:55:29.207295000Z node {\r\n2018-06-25T21:55:29.207533000Z   id: \"node504\"\r\n2018-06-25T21:55:29.207747000Z   cluster: \"default-cluster\"\r\n2018-06-25T21:55:29.208032000Z   locality {\r\n2018-06-25T21:55:29.208245000Z     zone: \"default-zone\"\r\n2018-06-25T21:55:29.208453000Z   }\r\n2018-06-25T21:55:29.208700000Z   build_version: \"067f8f6523f63d6f0ccd3d44e6fd2db97804af20/1.7.0-dev/Clean/RELEASE\"\r\n2018-06-25T21:55:29.208923000Z }\r\n2018-06-25T21:55:29.209145000Z resource_names: \"envoyservice-1\"\r\n2018-06-25T21:55:29.209371000Z type_url: \"type.googleapis.com/envoy.api.v2.ClusterLoadAssignment\"\r\n2018-06-25T21:55:29.211162000Z response_nonce: \"33\"\r\n2018-06-25T21:55:29.211414000Z\r\n2018-06-25T21:55:29.211659000Z [2018-06-25 21:55:29.202][5][trace][router] source/common/router/router.cc:872] [C0][S9549407309502274984] proxying 217 bytes\r\n2018-06-25T21:55:29.211905000Z [2018-06-25 21:55:29.202][5][trace][http2] source/common/http/http2/codec_impl.cc:292] [C0] dispatched 100 bytes\r\n2018-06-25T21:55:29.212129000Z [2018-06-25 21:55:29.202][5][trace][connection] source/common/network/connection_impl.cc:321] [C0] writing 226 bytes, end_stream false\r\n2018-06-25T21:55:29.212342000Z [2018-06-25 21:55:29.202][5][trace][http2] source/common/http/http2/codec_impl.cc:446] [C0] sent frame type=0\r\n2018-06-25T21:55:29.212595000Z [2018-06-25 21:55:29.202][5][trace][connection] source/common/network/connection_impl.cc:384] [C0] socket event: 2\r\n2018-06-25T21:55:29.212836000Z [2018-06-25 21:55:29.202][5][trace][connection] source/common/network/connection_impl.cc:452] [C0] write ready\r\n2018-06-25T21:55:29.213051000Z [2018-06-25 21:55:29.202][5][trace][connection] source/common/network/raw_buffer_socket.cc:63] [C0] write returns: 226\r\n2018-06-25T21:55:29.213275000Z [2018-06-25 21:55:29.203][5][trace][connection] source/common/network/connection_impl.cc:384] [C0] socket event: 3\r\n2018-06-25T21:55:29.213528000Z [2018-06-25 21:55:29.203][5][trace][connection] source/common/network/connection_impl.cc:452] [C0] write ready\r\n2018-06-25T21:55:29.213742000Z [2018-06-25 21:55:29.203][5][trace][connection] source/common/network/connection_impl.cc:422] [C0] read ready\r\n2018-06-25T21:55:29.213963000Z [2018-06-25 21:55:29.203][5][trace][connection] source/common/network/raw_buffer_socket.cc:21] [C0] read returns: 30\r\n2018-06-25T21:55:29.214194000Z [2018-06-25 21:55:29.203][5][trace][connection] source/common/network/raw_buffer_socket.cc:21] [C0] read returns: -1\r\n2018-06-25T21:55:29.214405000Z [2018-06-25 21:55:29.203][5][trace][connection] source/common/network/raw_buffer_socket.cc:29] [C0] read error: 11\r\n2018-06-25T21:55:29.214651000Z [2018-06-25 21:55:29.203][5][trace][http2] source/common/http/http2/codec_impl.cc:277] [C0] dispatching 30 bytes\r\n\r\n\r\n**Envoy Admin Stats**\r\ncluster.envoyservice-1.bind_errors: 0\r\ncluster.envoyservice-1.lb_healthy_panic: 0\r\ncluster.envoyservice-1.lb_local_cluster_not_ok: 0\r\ncluster.envoyservice-1.lb_recalculate_zone_structures: 0\r\ncluster.envoyservice-1.lb_subsets_active: 0\r\ncluster.envoyservice-1.lb_subsets_created: 0\r\ncluster.envoyservice-1.lb_subsets_fallback: 0\r\ncluster.envoyservice-1.lb_subsets_removed: 0\r\ncluster.envoyservice-1.lb_subsets_selected: 0\r\ncluster.envoyservice-1.lb_zone_cluster_too_small: 0\r\ncluster.envoyservice-1.lb_zone_no_capacity_left: 0\r\ncluster.envoyservice-1.lb_zone_number_differs: 0\r\ncluster.envoyservice-1.lb_zone_routing_all_directly: 0\r\ncluster.envoyservice-1.lb_zone_routing_cross_zone: 0\r\ncluster.envoyservice-1.lb_zone_routing_sampled: 0\r\ncluster.envoyservice-1.max_host_weight: 0\r\ncluster.envoyservice-1.membership_change: 0\r\ncluster.envoyservice-1.membership_healthy: 0\r\ncluster.envoyservice-1.membership_total: 0\r\ncluster.envoyservice-1.original_dst_host_invalid: 0\r\ncluster.envoyservice-1.retry_or_shadow_abandoned: 0\r\ncluster.envoyservice-1.update_attempt: 51\r\ncluster.envoyservice-1.update_empty: 50\r\ncluster.envoyservice-1.update_failure: 0\r\ncluster.envoyservice-1.update_no_rebuild: 0"},"comment":{"url":"https://api.github.com/repos/turbinelabs/rotor/issues/comments/401024667","html_url":"https://github.com/turbinelabs/rotor/issues/16#issuecomment-401024667","issue_url":"https://api.github.com/repos/turbinelabs/rotor/issues/16","id":401024667,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMTAyNDY2Nw==","user":{"login":"9len","id":362808,"node_id":"MDQ6VXNlcjM2MjgwOA==","avatar_url":"https://avatars1.githubusercontent.com/u/362808?v=4","gravatar_id":"","url":"https://api.github.com/users/9len","html_url":"https://github.com/9len","followers_url":"https://api.github.com/users/9len/followers","following_url":"https://api.github.com/users/9len/following{/other_user}","gists_url":"https://api.github.com/users/9len/gists{/gist_id}","starred_url":"https://api.github.com/users/9len/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/9len/subscriptions","organizations_url":"https://api.github.com/users/9len/orgs","repos_url":"https://api.github.com/users/9len/repos","events_url":"https://api.github.com/users/9len/events{/privacy}","received_events_url":"https://api.github.com/users/9len/received_events","type":"User","site_admin":false},"created_at":"2018-06-28T12:55:54Z","updated_at":"2018-06-28T12:55:54Z","author_association":"CONTRIBUTOR","body":"Would also be good to add `-l trace` to the envoy flags. "}}
{"actor":{"display_login":"9len"}}