{"action":"created","issue":{"url":"https://api.github.com/repos/envoyproxy/envoy/issues/2920","repository_url":"https://api.github.com/repos/envoyproxy/envoy","labels_url":"https://api.github.com/repos/envoyproxy/envoy/issues/2920/labels{/name}","comments_url":"https://api.github.com/repos/envoyproxy/envoy/issues/2920/comments","events_url":"https://api.github.com/repos/envoyproxy/envoy/issues/2920/events","html_url":"https://github.com/envoyproxy/envoy/issues/2920","id":309227208,"number":2920,"title":"Feature request: a means to refuse subsequent TCP connections while allowing current connections enough time to drain","user":{"login":"rosenhouse","id":4128521,"avatar_url":"https://avatars0.githubusercontent.com/u/4128521?v=4","gravatar_id":"","url":"https://api.github.com/users/rosenhouse","html_url":"https://github.com/rosenhouse","followers_url":"https://api.github.com/users/rosenhouse/followers","following_url":"https://api.github.com/users/rosenhouse/following{/other_user}","gists_url":"https://api.github.com/users/rosenhouse/gists{/gist_id}","starred_url":"https://api.github.com/users/rosenhouse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rosenhouse/subscriptions","organizations_url":"https://api.github.com/users/rosenhouse/orgs","repos_url":"https://api.github.com/users/rosenhouse/repos","events_url":"https://api.github.com/users/rosenhouse/events{/privacy}","received_events_url":"https://api.github.com/users/rosenhouse/received_events","type":"User","site_admin":false},"labels":[{"id":421403909,"url":"https://api.github.com/repos/envoyproxy/envoy/labels/enhancement","name":"enhancement","color":"84b6eb","default":true},{"id":645516726,"url":"https://api.github.com/repos/envoyproxy/envoy/labels/help%20wanted","name":"help wanted","color":"fbca04","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":21,"created_at":"2018-03-28T05:35:59Z","updated_at":"2018-05-09T20:26:56Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"#### update\r\nWe've edited this issue to be less prescriptive about solution.  it now presents 3 possible solutions that we can see\r\n\r\n### summary\r\n\r\nGiven I've configured Envoy with LDS serving a TCP proxy listener on some port\r\nand there are connections in flight\r\nI would like a way to refuse subsequent TCP connections to that port while allowing current established connections to drain\r\n\r\nWe tried the following approaches, but none of them achieve our goals:\r\n1. the LDS is updated to remove the listener\r\n2. Envoy is signalled with a `SIGTERM`\r\n3. Send a `GET` request to `/healthcheck/fail`\r\n\r\n### steps to reproduce\r\n\r\nwrite a `bootstrap.yaml` like\r\n\r\n```yaml\r\n---\r\nadmin:\r\n  access_log_path: /tmp/admin_access.log\r\n  address:\r\n    socket_address:\r\n      address: 0.0.0.0\r\n      port_value: 9901\r\n\r\nnode:\r\n  id: some-envoy-node\r\n  cluster: some-envoy-cluster\r\n\r\ndynamic_resources:\r\n  lds_config:\r\n    path: /cfg/lds-current.yaml\r\n\r\nstatic_resources:\r\n  clusters:\r\n  - name: example_cluster\r\n    connect_timeout: 0.25s\r\n    type: STATIC\r\n    lb_policy: ROUND_ROBIN\r\n    hosts:\r\n    - socket_address:\r\n        address: 93.184.216.3    # IP address of example.com\r\n        port_value: 80\r\n```\r\n\r\nwrite a lds-current.yaml file like\r\n\r\n```yaml\r\nversion_info: \"0\"\r\nresources:\r\n- \"@type\": type.googleapis.com/envoy.api.v2.Listener\r\n  name: listener_0\r\n  address:\r\n    socket_address:\r\n      address: 0.0.0.0\r\n      port_value: 8080\r\n  filter_chains:\r\n    - filters:\r\n        - name: envoy.tcp_proxy\r\n          config:\r\n            stat_prefix: ingress_tcp\r\n            cluster: example_cluster\r\n```\r\n\r\nlaunch envoy (I'm using v1.6.0)\r\n\r\n```\r\nenvoy -c /cfg/bootstrap.yaml --v2-config-only --drain-time-s 30\r\n```\r\n\r\nconfirm that the TCP proxy is working\r\n\r\n```\r\ncurl -v -H 'Host: example.com' 127.0.0.1:8080\r\n```\r\n\r\n#### Possible approach 1: remove the listener\r\n\r\nupdate the LDS to return an empty set of listeners. this is a two step process. first, write an empty LDS response file lds-empty.yaml\r\n\r\n```yaml\r\nversion_info: \"1\"\r\nresources: []\r\n```\r\n\r\nsecond, move that file on top of the file being watched:\r\n\r\n```\r\nmv lds-empty.yaml lds-current.yaml\r\n```\r\n\r\nin the Envoy stdout logs you'll see a line\r\n\r\n```\r\nsource/server/lds_api.cc:68] lds: remove listener 'listener_0'\r\n```\r\n\r\nattempt to connect to the port where the listener used to be:\r\n\r\n```\r\ncurl -v -H 'Host: example.com' 127.0.0.1:8080\r\n```\r\n\r\n##### expected behavior\r\n\r\nWould like to see all new TCP connections be refused immediately, as if a listener had never been added in the first place. Existing TCP connections should continue to be serviced.\r\n\r\n##### actual behavior\r\n\r\nthe port is still open, even after the LDS update occurs\r\n\r\n```\r\nlsof -i\r\nCOMMAND PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME\r\nenvoy     1 root    9u  IPv4  30166      0t0  TCP *:9901 (LISTEN)\r\nenvoy     1 root   22u  IPv4  30171      0t0  TCP *:8080 (LISTEN)\r\n```\r\n\r\nclients can connect to the port, but the TCP proxying seems to hang (can't tell where)\r\n\r\n```\r\ncurl -H 'Host: example.com' -v 127.0.0.1:8080\r\n* Rebuilt URL to: 127.0.0.1:8080/\r\n*   Trying 127.0.0.1...\r\n* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)\r\n> GET / HTTP/1.1\r\n> Host: example.com\r\n> User-Agent: curl/7.47.0\r\n> Accept: */*\r\n>\r\n^C\r\n```\r\n\r\nthis state remains until --drain-time-s time has elapsed (30 seconds in this example). At that point the port is finally closed, so you see\r\n\r\n```\r\ncurl 127.0.0.1:8080\r\ncurl: (7) Failed to connect to 127.0.0.1 port 8080: Connection refused\r\n```\r\n\r\n#### Possible approach 2: pkill -SIGTERM envoy\r\n\r\nIf instead of removing the listeners we signaled Envoy\r\n\r\n```\r\npkill -SIGTERM envoy\r\n```\r\n\r\nEnvoy exits immediately without allowing current connections to drain\r\n\r\n```\r\n[2018-03-28 17:42:06.995][3563][warning][main] source/server/server.cc:312] caught SIGTERM\r\n[2018-03-28 17:42:06.995][3563][info][main] source/server/server.cc:357] main dispatch loop exited\r\n[2018-03-28 17:42:07.004][3563][info][main] source/server/server.cc:392] exiting\r\n```\r\n\r\n**EDITED** to remove incorrect bit about listeners staying open after SIGTERM.\r\n\r\n#### Possible approach 3: admin healthcheck fail\r\n\r\nWe could instead `GET /healthcheck/fail` to trigger this behavior.  As above, we would expect that new TCP connections should be refused while existing TCP connections are serviced.\r\n\r\n\r\n\r\n### background\r\n\r\nIn Cloud Foundry, we have the following setup currently:\r\n\r\n```\r\n       Router         =====>       Envoy    ---->    App\r\n(shared ingress)      (TLS)                 TCP\r\n```\r\n\r\nEach application instance has a sidecar Envoy which terminates TLS connections from the shared ingress router. Applications may not speak HTTP, so we use basic TCP connectivity checks from the shared Router to the Envoy in order to infer application health and determine if a client connection should be load-balanced to that Envoy. When the upstream Envoy accepts the TCP connection, the Router considers that upstream healthy. When the upstream refuses the TCP connection, the Router considers that upstream unhealthy.\r\n\r\nDuring a graceful shutdown, the scheduler ought to be able to drain the Envoy before terminating the application. This would mean that the Envoy ought to service any in-flight TCP connections without accepting any new ones.\r\n\r\n\r\n### acknowledgements\r\nh/t @jvshahid and @emalm for investigation and edits"},"comment":{"url":"https://api.github.com/repos/envoyproxy/envoy/issues/comments/387865030","html_url":"https://github.com/envoyproxy/envoy/issues/2920#issuecomment-387865030","issue_url":"https://api.github.com/repos/envoyproxy/envoy/issues/2920","id":387865030,"user":{"login":"ggreenway","id":8603643,"avatar_url":"https://avatars2.githubusercontent.com/u/8603643?v=4","gravatar_id":"","url":"https://api.github.com/users/ggreenway","html_url":"https://github.com/ggreenway","followers_url":"https://api.github.com/users/ggreenway/followers","following_url":"https://api.github.com/users/ggreenway/following{/other_user}","gists_url":"https://api.github.com/users/ggreenway/gists{/gist_id}","starred_url":"https://api.github.com/users/ggreenway/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ggreenway/subscriptions","organizations_url":"https://api.github.com/users/ggreenway/orgs","repos_url":"https://api.github.com/users/ggreenway/repos","events_url":"https://api.github.com/users/ggreenway/events{/privacy}","received_events_url":"https://api.github.com/users/ggreenway/received_events","type":"User","site_admin":false},"created_at":"2018-05-09T20:26:56Z","updated_at":"2018-05-09T20:26:56Z","author_association":"CONTRIBUTOR","body":"@mattklein123 front-end health check failing doesn't really work with TcpProxy, does it?"}}
{"actor":{"display_login":"ggreenway"}}