{"action":"created","issue":{"url":"https://api.github.com/repos/runconduit/conduit/issues/986","repository_url":"https://api.github.com/repos/runconduit/conduit","labels_url":"https://api.github.com/repos/runconduit/conduit/issues/986/labels{/name}","comments_url":"https://api.github.com/repos/runconduit/conduit/issues/986/comments","events_url":"https://api.github.com/repos/runconduit/conduit/issues/986/events","html_url":"https://github.com/runconduit/conduit/issues/986","id":325582460,"number":986,"title":"Proxy may not gracefully close all streams","user":{"login":"siggy","id":236915,"avatar_url":"https://avatars2.githubusercontent.com/u/236915?v=4","gravatar_id":"","url":"https://api.github.com/users/siggy","html_url":"https://github.com/siggy","followers_url":"https://api.github.com/users/siggy/followers","following_url":"https://api.github.com/users/siggy/following{/other_user}","gists_url":"https://api.github.com/users/siggy/gists{/gist_id}","starred_url":"https://api.github.com/users/siggy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/siggy/subscriptions","organizations_url":"https://api.github.com/users/siggy/orgs","repos_url":"https://api.github.com/users/siggy/repos","events_url":"https://api.github.com/users/siggy/events{/privacy}","received_events_url":"https://api.github.com/users/siggy/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"olix0r","id":240738,"avatar_url":"https://avatars3.githubusercontent.com/u/240738?v=4","gravatar_id":"","url":"https://api.github.com/users/olix0r","html_url":"https://github.com/olix0r","followers_url":"https://api.github.com/users/olix0r/followers","following_url":"https://api.github.com/users/olix0r/following{/other_user}","gists_url":"https://api.github.com/users/olix0r/gists{/gist_id}","starred_url":"https://api.github.com/users/olix0r/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/olix0r/subscriptions","organizations_url":"https://api.github.com/users/olix0r/orgs","repos_url":"https://api.github.com/users/olix0r/repos","events_url":"https://api.github.com/users/olix0r/events{/privacy}","received_events_url":"https://api.github.com/users/olix0r/received_events","type":"User","site_admin":false},"assignees":[{"login":"olix0r","id":240738,"avatar_url":"https://avatars3.githubusercontent.com/u/240738?v=4","gravatar_id":"","url":"https://api.github.com/users/olix0r","html_url":"https://github.com/olix0r","followers_url":"https://api.github.com/users/olix0r/followers","following_url":"https://api.github.com/users/olix0r/following{/other_user}","gists_url":"https://api.github.com/users/olix0r/gists{/gist_id}","starred_url":"https://api.github.com/users/olix0r/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/olix0r/subscriptions","organizations_url":"https://api.github.com/users/olix0r/orgs","repos_url":"https://api.github.com/users/olix0r/repos","events_url":"https://api.github.com/users/olix0r/events{/privacy}","received_events_url":"https://api.github.com/users/olix0r/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/runconduit/conduit/milestones/7","html_url":"https://github.com/runconduit/conduit/milestone/7","labels_url":"https://api.github.com/repos/runconduit/conduit/milestones/7/labels","id":3134345,"number":7,"title":"0.5.0","description":"Controllable Deadlines/Timeouts","creator":{"login":"olix0r","id":240738,"avatar_url":"https://avatars3.githubusercontent.com/u/240738?v=4","gravatar_id":"","url":"https://api.github.com/users/olix0r","html_url":"https://github.com/olix0r","followers_url":"https://api.github.com/users/olix0r/followers","following_url":"https://api.github.com/users/olix0r/following{/other_user}","gists_url":"https://api.github.com/users/olix0r/gists{/gist_id}","starred_url":"https://api.github.com/users/olix0r/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/olix0r/subscriptions","organizations_url":"https://api.github.com/users/olix0r/orgs","repos_url":"https://api.github.com/users/olix0r/repos","events_url":"https://api.github.com/users/olix0r/events{/privacy}","received_events_url":"https://api.github.com/users/olix0r/received_events","type":"User","site_admin":false},"open_issues":57,"closed_issues":49,"state":"open","created_at":"2018-02-21T19:35:41Z","updated_at":"2018-05-25T21:57:41Z","due_on":null,"closed_at":null},"comments":9,"created_at":"2018-05-23T07:31:37Z","updated_at":"2018-05-28T21:37:24Z","closed_at":null,"author_association":"MEMBER","body":"Using the lifecycle test environment (introduced in https://github.com/runconduit/conduit-examples/pull/41) we're observing a gradual throughput drop from 3000qps with 100% SR to 0qps.\r\n\r\nIt's unclear whether this issue lies in the proxy, test environment, or some interaction with Kubernetes.\r\n\r\n## Setup\r\n\r\nTest environment topology:\r\n```\r\nslow_cooker ->\r\n  HTTP 1.1 ->\r\n    bb point-to-point ->\r\n      gRPC ->\r\n        bb terminus\r\n```\r\nThe `bb terminus` server exits after one minute's worth of requests, causing the gRPC connection to fail. Kubernetes then restarts the container in the existing pod.\r\n\r\n## Repro\r\n\r\nThese results were produced on Docker for Mac with Kubernetes.\r\n\r\nBuild Conduit off of master, currently at https://github.com/runconduit/conduit/tree/c5f0adafc8a831c0cc19b9abdf2fa1016e1942c5 (with #985 applied for docker build fix).\r\n\r\n```bash\r\nbin/docker-build\r\n```\r\n\r\nDeploy conduit and test environment\r\n\r\n```bash\r\nbin/conduit install - | kubectl apply -f -\r\ncurl -s https://raw.githubusercontent.com/runconduit/conduit-examples/f9ad69bfadcc817e3042b9027cba37467d726cf4/lifecycle/lifecycle.yml | bin/conduit inject - | kubectl apply -f -\r\n```\r\n\r\n### Logs\r\n\r\n#### slow-cooker\r\n```bash\r\n$ kubectl -n lifecycle logs -f $(\r\n  kubectl -n lifecycle get po --selector=job-name=slow-cooker -o jsonpath='{.items[*].metadata.name}'\r\n) slow-cooker\r\n# sending 100 GET req/s with concurrency=10 to http://bb-p2p.lifecycle.svc.cluster.local:8080 ...\r\n#                      good/b/f t   good%   min [p50 p95 p99  p999]  max change\r\n2018-05-23T06:34:57Z   2990/0/0 3000  99% 30s   7 [ 27  57  81  149 ]  149 +\r\n2018-05-23T06:35:27Z   2893/0/0 3000  96% 30s   7 [ 26  62  92 1367 ] 1367\r\n2018-05-23T06:35:57Z   2684/0/0 3000  89% 30s   6 [ 26  62 747 1958 ] 1958 +\r\n2018-05-23T06:36:27Z   3000/0/0 3000 100% 30s   7 [ 27  53  80  152 ]  152\r\n2018-05-23T06:36:57Z    816/18/0 3000  27% 30s   4 [ 30 116 10007 10039 ] 10036 +\r\n2018-05-23T06:37:27Z   1475/19/0 3000  49% 30s   6 [ 38 110 10007 10023 ] 10018\r\n2018-05-23T06:37:57Z   2988/0/0 3000  99% 30s   9 [ 32  96 171  278 ]  278 -\r\n2018-05-23T06:38:27Z    626/17/0 3000  20% 30s   4 [ 17  66 10007 10039 ] 10033\r\n2018-05-23T06:38:57Z     91/30/0 3000   3% 30s   5 [ 24 10007 10015 10023 ] 10016\r\n...\r\n2018-05-23T06:56:27Z      8/30/0 3000   0% 30s  13 [10007 10015 10031 10031 ] 10026\r\n2018-05-23T06:56:57Z      7/30/0 3000   0% 30s   5 [10007 10023 10039 10039 ] 10037\r\n...\r\n2018-05-23T07:06:27Z      0/30/0 3000   0% 30s 10000 [10007 10015 10071 10071 ] 10069\r\n2018-05-23T07:06:57Z      0/30/0 3000   0% 30s 10000 [10007 10023 10087 10087 ] 10080\r\n```\r\n\r\n#### proxy log in `bb-p2p` pod\r\n```bash\r\n$ kubectl -n lifecycle logs -f po/bb-p2p-578cf865cb-vgspz conduit-proxy\r\nINFO conduit_proxy using controller at HostAndPort { host: DnsName(Name(\"proxy-api.conduit.svc.cluster.local\")), port: 8086 }\r\nINFO conduit_proxy routing on V4(127.0.0.1:4140)\r\nINFO conduit_proxy proxying on V4(0.0.0.0:4143) to None\r\nINFO conduit_proxy serving Prometheus metrics on V4(0.0.0.0:4191)\r\nINFO conduit_proxy protocol detection disabled for inbound ports {25, 3306}\r\nINFO conduit_proxy protocol detection disabled for outbound ports {25, 3306}\r\nINFO conduit_proxy::transport::connect \"controller-client\", DNS resolved DnsName(Name(\"proxy-api.conduit.svc.cluster.local\")) to 10.98.179.79\r\nERR! conduit_proxy turning operation timed out after 10s into 500\r\nERR! conduit_proxy turning operation timed out after 10s into 500\r\nERR! conduit_proxy turning operation timed out after 10s into 500\r\nERR! conduit_proxy turning operation timed out after 10s into 500\r\n```\r\ntail of `bb-p2p` proxy  with `--proxy-log-level debug,conduit_proxy=debug`:\r\nhttps://gist.github.com/siggy/874ea90f5f4c35ec7260b835bef82625\r\n\r\n#### proxy log in `bb terminus` pod\r\n```bash\r\n$ kubectl -n lifecycle logs -f po/bb-terminus-748dddbb75-6hm4g conduit-proxy\r\nINFO conduit_proxy using controller at HostAndPort { host: DnsName(Name(\"proxy-api.conduit.svc.cluster.local\")), port: 8086 }\r\nINFO conduit_proxy routing on V4(127.0.0.1:4140)\r\nINFO conduit_proxy proxying on V4(0.0.0.0:4143) to None\r\nINFO conduit_proxy serving Prometheus metrics on V4(0.0.0.0:4191)\r\nINFO conduit_proxy protocol detection disabled for inbound ports {25, 3306}\r\nINFO conduit_proxy protocol detection disabled for outbound ports {25, 3306}\r\nINFO conduit_proxy::transport::connect \"controller-client\", DNS resolved DnsName(Name(\"proxy-api.conduit.svc.cluster.local\")) to 10.98.179.79\r\nWARN conduit_proxy::bind connect error to Endpoint { address: V4(10.1.15.228:9090), dst_labels: None }: Error attempting to establish underlying session layer: Connection refused (os error 111)\r\nWARN conduit_proxy::bind connect error to Endpoint { address: V4(10.1.15.228:9090), dst_labels: None }: Error attempting to establish underlying session layer: Connection refused (os error 111)\r\nWARN conduit_proxy::bind connect error to Endpoint { address: V4(10.1.15.228:9090), dst_labels: None }: Error attempting to establish underlying session layer: Connection refused (os error 111)\r\n```\r\ntail of `bb terminus` proxy with `--proxy-log-level debug,conduit_proxy=debug`:\r\nhttps://gist.github.com/siggy/654a54be40175783f3d45c86933292ec\r\n\r\n## Baseline\r\n\r\nConfirmed that without proxy injection, this issue does not arise.\r\n\r\n```bash\r\ncat lifecycle.yml | kubectl apply -f -\r\n```\r\n\r\n```bash\r\n$  kubectl -n lifecycle logs -f $(\r\n>   kubectl -n lifecycle get po --selector=job-name=slow-cooker -o jsonpath='{.items[*].metadata.name}'\r\n> ) slow-cooker\r\n# sending 100 GET req/s with concurrency=10 to http://bb-p2p.lifecycle.svc.cluster.local:8080 ...\r\n#                      good/b/f t   good%   min [p50 p95 p99  p999]  max change\r\n2018-05-23T22:21:14Z   2500/490/0 3000  83% 30s   0 [  7  18  34  102 ]  102 +\r\n2018-05-23T22:21:44Z   2920/0/0 3000  97% 30s   1 [  7  18  44  987 ]  987\r\n2018-05-23T22:22:14Z   2896/0/0 3000  96% 30s   1 [  9  66 133  201 ]  201\r\n2018-05-23T22:22:44Z   2988/0/0 3000  99% 30s   1 [ 11  53 100  256 ]  256\r\n2018-05-23T22:23:14Z   2992/0/0 3000  99% 30s   1 [  8  32  84  199 ]  199\r\n2018-05-23T22:23:44Z   2999/0/0 3000  99% 30s   2 [  8  32  88  226 ]  226\r\n2018-05-23T22:24:14Z   3004/0/0 3000 100% 30s   1 [  7  16  27   42 ]   42\r\n2018-05-23T22:24:44Z   3004/0/0 3000 100% 30s   2 [  7  16  37   63 ]   63\r\n2018-05-23T22:25:14Z   1590/1410/0 3000  53% 30s   0 [  5  14  25   50 ]   50\r\n...\r\n2018-05-23T22:30:14Z   1010/1990/0 3000  33% 30s   0 [  4  11  30   39 ]   39\r\n2018-05-23T22:30:44Z    740/2260/0 3000  24% 30s   0 [  3  11  27   41 ]   41\r\n```\r\nNote that some degradation does occur, most likely due to https://github.com/runconduit/conduit-examples/issues/42."},"comment":{"url":"https://api.github.com/repos/runconduit/conduit/issues/comments/392606938","html_url":"https://github.com/runconduit/conduit/issues/986#issuecomment-392606938","issue_url":"https://api.github.com/repos/runconduit/conduit/issues/986","id":392606938,"user":{"login":"olix0r","id":240738,"avatar_url":"https://avatars3.githubusercontent.com/u/240738?v=4","gravatar_id":"","url":"https://api.github.com/users/olix0r","html_url":"https://github.com/olix0r","followers_url":"https://api.github.com/users/olix0r/followers","following_url":"https://api.github.com/users/olix0r/following{/other_user}","gists_url":"https://api.github.com/users/olix0r/gists{/gist_id}","starred_url":"https://api.github.com/users/olix0r/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/olix0r/subscriptions","organizations_url":"https://api.github.com/users/olix0r/orgs","repos_url":"https://api.github.com/users/olix0r/repos","events_url":"https://api.github.com/users/olix0r/events{/privacy}","received_events_url":"https://api.github.com/users/olix0r/received_events","type":"User","site_admin":false},"created_at":"2018-05-28T21:37:24Z","updated_at":"2018-05-28T21:37:24Z","author_association":"MEMBER","body":"On a working branch, I've enhanced logging to better differentiate client and server logs, which helped narrow down the nature of the issue:\r\n\r\ntl;dr: it appears that the proxy is erroneously queueing a canceled request to be sent to the application\r\n\r\n#### Request 1\r\n\r\n_mid_ sends a request to _srv_, which connects to its application and sends the request.\r\nThe application responds and sends a GOAWAY, closing the connection. _srv_ continues to\r\nsend the response to _mid_.\r\n\r\n```\r\n2018-05-28T19:14:51.052271043Z mid TRCE conduit_proxy::transparency::server server={proxy=inbound; peer=10.1.1.197:56536} transparency detected HTTP/1\r\n2018-05-28T19:14:51.058250699Z mid TRCE h2::proto::connection server={proxy=outbound; peer=10.1.1.196:51912} recv HEADERS; frame=Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: false } }\r\n2018-05-28T19:14:51.058881815Z mid TRCE h2::proto::connection server={proxy=outbound; peer=10.1.1.196:51912} recv DATA; frame=Data { stream_id: StreamId(1), flags: DataFlags { end_stream: true } }\r\n2018-05-28T19:14:51.060108926Z mid TRCE conduit_proxy::control::destination server={proxy=outbound; peer=10.1.1.196:51912} resolve; authority=DnsNameAndPort { host: Name(\"server.bb.svc.cluster.local\"), port: 9090 }\r\n2018-05-28T19:14:51.093466316Z mid TRCE conduit_proxy::control::destination Out { dest: Hostname(DnsNameAndPort { host: Name(\"server.bb.svc.cluster.local\"), port: 9090 }), proto: Http2 }, watch: Ok(Ready(Some(Insert(V4(10.1.1.195:9090), Metadata { metric_labels: Some(DstLabels { formatted: \"dst_conduit_io_control_plane_ns=\\\"conduit\\\",dst_deployment=\\\"server\\\",dst_namespace=\\\"bb\\\",dst_pod=\\\"server-65d49dd9dc-b982b\\\",dst_pod_template_hash=\\\"2180588587\\\",dst_service=\\\"server\\\"\", original: {\"pod\": \"server-65d49dd9dc-b982b\", \"service\": \"server\", \"deployment\": \"server\", \"pod_template_hash\": \"2180588587\", \"namespace\": \"bb\", \"conduit_io_control_plane_ns\": \"conduit\"} }) }))))\r\n2018-05-28T19:14:51.098327394Z srv TRCE conduit_proxy::transparency::server server={proxy=inbound; peer=10.1.1.196:52140} transparency detected HTTP/2\r\n2018-05-28T19:14:51.105399115Z mid TRCE h2::proto::connection client={proxy=outbound}, recv SETTINGS; frame=Settings { flags: SettingsFlags(0), ... }\r\n2018-05-28T19:14:51.107034500Z mid TRCE h2::proto::streams::prioritize client={proxy=outbound; addr=10.1.1.195:9090; proto=Http2}, writing frame=Frame::Headers(Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: false } })\r\n2018-05-28T19:14:51.110369301Z srv TRCE h2::proto::connection server={proxy=inbound; peer=10.1.1.196:52140} recv SETTINGS; frame=Settings { flags: SettingsFlags(0), ... }\r\n2018-05-28T19:14:51.110386780Z srv TRCE h2::proto::connection server={proxy=inbound; peer=10.1.1.196:52140} recv SETTINGS; frame=Settings { flags: SettingsFlags(1), ... }\r\n2018-05-28T19:14:51.111280580Z srv TRCE h2::proto::connection server={proxy=inbound; peer=10.1.1.196:52140} recv HEADERS; frame=Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: false } }\r\n2018-05-28T19:14:51.115283609Z srv TRCE h2::proto::connection server={proxy=inbound; peer=10.1.1.196:52140} recv DATA; frame=Data { stream_id: StreamId(1), flags: DataFlags { end_stream: true } }\r\n2018-05-28T19:14:51.116944451Z srv TRCE h2::proto::connection client={proxy=inbound}, recv SETTINGS; frame=Settings { flags: SettingsFlags(0), ... }\r\n2018-05-28T19:14:51.117206440Z mid TRCE h2::proto::streams::prioritize client={proxy=outbound; addr=10.1.1.195:9090; proto=Http2}, writing frame=Frame::Data(Data { stream_id: StreamId(1), flags: DataFlags { end_stream: true } })\r\n2018-05-28T19:14:51.117234632Z mid TRCE h2::proto::connection client={proxy=outbound}, recv SETTINGS; frame=Settings { flags: SettingsFlags(1), ... }\r\n2018-05-28T19:14:51.117450564Z srv TRCE h2::proto::streams::prioritize client={proxy=inbound; addr=10.1.1.195:9090; proto=Http2}, writing frame=Frame::Headers(Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: false } })\r\n2018-05-28T19:14:51.119549784Z srv TRCE h2::proto::streams::prioritize client={proxy=inbound; addr=10.1.1.195:9090; proto=Http2}, writing frame=Frame::Data(Data { stream_id: StreamId(1), flags: DataFlags { end_stream: true } })\r\n2018-05-28T19:14:51.126360894Z srv TRCE h2::proto::connection client={proxy=inbound}, recv SETTINGS; frame=Settings { flags: SettingsFlags(1), ... }\r\n2018-05-28T19:14:51.129624471Z srv TRCE h2::proto::connection client={proxy=inbound}, recv WINDOW_UPDATE; frame=WindowUpdate { stream_id: StreamId(0), size_increment: 66 }\r\n2018-05-28T19:14:51.129817128Z srv TRCE h2::proto::connection client={proxy=inbound}, recv PING; frame=Ping { ack: false, payload: [2, 4, 16, 16, 9, 14, 7, 7] }\r\n2018-05-28T19:14:51.130244793Z srv TRCE h2::proto::connection client={proxy=inbound}, recv HEADERS; frame=Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: false } }\r\n2018-05-28T19:14:51.130398366Z srv TRCE h2::proto::connection client={proxy=inbound}, recv DATA; frame=Data { stream_id: StreamId(1), flags: DataFlags }\r\n2018-05-28T19:14:51.130982498Z srv TRCE h2::proto::connection client={proxy=inbound}, recv HEADERS; frame=Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: true } }\r\n2018-05-28T19:14:51.134926114Z srv TRCE h2::proto::streams::prioritize (\"inbound\", V4(10.1.1.196:52140)), writing frame=Frame::Headers(Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: false } })\r\n2018-05-28T19:14:51.137176403Z srv TRCE h2::proto::streams::prioritize (\"inbound\", V4(10.1.1.196:52140)), writing frame=Frame::Data(Data { stream_id: StreamId(1), flags: DataFlags })\r\n2018-05-28T19:14:51.138137300Z srv TRCE h2::proto::streams::prioritize (\"inbound\", V4(10.1.1.196:52140)), writing frame=Frame::Headers(Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: true } })\r\n2018-05-28T19:14:51.162318439Z mid TRCE h2::proto::connection client={proxy=outbound}, recv HEADERS; frame=Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: false } }\r\n2018-05-28T19:14:51.162774663Z srv TRCE h2::proto::connection client={proxy=inbound}, recv GOAWAY; frame=GoAway { last_stream_id: StreamId(2147483647), error_code: NO_ERROR }\r\n2018-05-28T19:14:51.162783551Z srv TRCE h2::proto::connection client={proxy=inbound}, recv PING; frame=Ping { ack: false, payload: [1, 6, 1, 8, 0, 3, 3, 9] }\r\n2018-05-28T19:14:51.162798245Z mid TRCE h2::proto::connection client={proxy=outbound}, recv DATA; frame=Data { stream_id: StreamId(1), flags: DataFlags }\r\n2018-05-28T19:14:51.168062151Z mid TRCE h2::proto::connection client={proxy=outbound}, recv HEADERS; frame=Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: true } }\r\n2018-05-28T19:14:51.180433821Z mid TRCE h2::proto::streams::prioritize (\"outbound\", V4(10.1.1.196:51912)), writing frame=Frame::Headers(Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: false } })\r\n2018-05-28T19:14:51.181867031Z mid TRCE h2::proto::streams::prioritize (\"outbound\", V4(10.1.1.196:51912)), writing frame=Frame::Data(Data { stream_id: StreamId(1), flags: DataFlags })\r\n2018-05-28T19:14:51.182854360Z mid TRCE h2::proto::streams::prioritize (\"outbound\", V4(10.1.1.196:51912)), writing frame=Frame::Headers(Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: true } })\r\n2018-05-28T19:14:51.185756724Z mid TRCE h2::proto::connection server={proxy=outbound; peer=10.1.1.196:51912} recv WINDOW_UPDATE; frame=WindowUpdate { stream_id: StreamId(0), size_increment: 78 }\r\n2018-05-28T19:14:51.185821714Z mid TRCE h2::proto::connection server={proxy=outbound; peer=10.1.1.196:51912} recv PING; frame=Ping { ack: false, payload: [2, 4, 16, 16, 9, 14, 7, 7] }\r\n```\r\n\r\n\r\n#### Request 2\r\n\r\n_mid_ sends another request to _srv_.\r\n\r\nAs _mid_ routes the request, it processes service discovery updates reflecting the fact\r\nthat _srv_'s application restarted. Because the node is removed and added immediately,\r\n_mid_ establishes anew connection to _srv_:\r\n\r\n```\r\n2018-05-28T19:14:52.047130374Z mid TRCE h2::proto::connection server={proxy=outbound; peer=10.1.1.196:51912} recv HEADERS; frame=Headers { stream_id: StreamId(3), flags: HeadersFlag { end_stream: false } }\r\n2018-05-28T19:14:52.050584771Z mid TRCE h2::proto::connection server={proxy=outbound; peer=10.1.1.196:51912} recv DATA; frame=Data { stream_id: StreamId(3), flags: DataFlags { end_stream: true } }\r\n2018-05-28T19:14:52.051841616Z mid TRCE conduit_proxy::control::destination Out { dest: Hostname(DnsNameAndPort { host: Name(\"server.bb.svc.cluster.local\"), port: 9090 }), proto: Http2 }, watch: Ok(Ready(Some(Remove(V4(10.1.1.195:9090)))))\r\n2018-05-28T19:14:52.470053615Z mid TRCE conduit_proxy::control::destination Out { dest: Hostname(DnsNameAndPort { host: Name(\"server.bb.svc.cluster.local\"), port: 9090 }), proto: Http2 }, watch: Ok(Ready(Some(Insert(V4(10.1.1.195:9090), Metadata { metric_labels: Some(DstLabels { formatted: \"dst_conduit_io_control_plane_ns=\\\"conduit\\\",dst_deployment=\\\"server\\\",dst_namespace=\\\"bb\\\",dst_pod=\\\"server-65d49dd9dc-b982b\\\",dst_pod_template_hash=\\\"2180588587\\\",dst_service=\\\"server\\\"\", original: {\"pod\": \"server-65d49dd9dc-b982b\", \"service\": \"server\", \"deployment\": \"server\", \"pod_template_hash\": \"2180588587\", \"namespace\": \"bb\", \"conduit_io_control_plane_ns\": \"conduit\"} }) }))))\r\n2018-05-28T19:14:52.481024954Z mid TRCE h2::proto::streams::prioritize client={proxy=outbound; addr=10.1.1.195:9090; proto=Http2}, writing frame=Frame::Headers(Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: false } })\r\n2018-05-28T19:14:52.490801480Z srv TRCE conduit_proxy::transparency::server server={proxy=inbound; peer=10.1.1.196:52162} transparency detected HTTP/2\r\n2018-05-28T19:14:52.494736315Z srv TRCE h2::proto::connection server={proxy=inbound; peer=10.1.1.196:52162} recv SETTINGS; frame=Settings { flags: SettingsFlags(0), ... }\r\n2018-05-28T19:14:52.495417989Z mid TRCE h2::proto::streams::prioritize client={proxy=outbound; addr=10.1.1.195:9090; proto=Http2}, writing frame=Frame::Data(Data { stream_id: StreamId(1), flags: DataFlags { end_stream: true } })\r\n```\r\n\r\nAgain, _srv_ connects to its application and sends the request. The application responds\r\nand sends a GOAWAY, closing the connection. _srv_ continues to send the response to _mid_.\r\n\r\n```\r\n2018-05-28T19:14:52.497711748Z srv TRCE h2::proto::connection server={proxy=inbound; peer=10.1.1.196:52162} recv HEADERS; frame=Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: false } }\r\n2018-05-28T19:14:52.498858277Z mid TRCE h2::proto::connection client={proxy=outbound}, recv SETTINGS; frame=Settings { flags: SettingsFlags(0), ... }\r\n2018-05-28T19:14:52.499958200Z srv TRCE h2::proto::connection server={proxy=inbound; peer=10.1.1.196:52162} recv DATA; frame=Data { stream_id: StreamId(1), flags: DataFlags { end_stream: true } }\r\n2018-05-28T19:14:52.504755095Z srv TRCE h2::proto::connection server={proxy=inbound; peer=10.1.1.196:52162} recv SETTINGS; frame=Settings { flags: SettingsFlags(1), ... }\r\n2018-05-28T19:14:52.510992604Z mid TRCE h2::proto::connection client={proxy=outbound}, recv SETTINGS; frame=Settings { flags: SettingsFlags(1), ... }\r\n2018-05-28T19:14:52.526728635Z srv TRCE h2::proto::streams::prioritize client={proxy=inbound; addr=10.1.1.195:9090; proto=Http2}, writing frame=Frame::Headers(Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: false } })\r\n2018-05-28T19:14:52.537449895Z srv TRCE h2::proto::streams::prioritize client={proxy=inbound; addr=10.1.1.195:9090; proto=Http2}, writing frame=Frame::Data(Data { stream_id: StreamId(1), flags: DataFlags { end_stream: true } })\r\n2018-05-28T19:14:52.552733742Z srv TRCE h2::proto::connection client={proxy=inbound}, recv SETTINGS; frame=Settings { flags: SettingsFlags(0), ... }\r\n2018-05-28T19:14:52.563941837Z srv TRCE h2::proto::connection client={proxy=inbound}, recv SETTINGS; frame=Settings { flags: SettingsFlags(1), ... }\r\n2018-05-28T19:14:52.564048148Z srv TRCE h2::proto::connection client={proxy=inbound}, recv WINDOW_UPDATE; frame=WindowUpdate { stream_id: StreamId(0), size_increment: 66 }\r\n2018-05-28T19:14:52.571657606Z srv TRCE h2::proto::connection client={proxy=inbound}, recv PING; frame=Ping { ack: false, payload: [2, 4, 16, 16, 9, 14, 7, 7] }\r\n2018-05-28T19:14:52.579655443Z srv TRCE h2::proto::connection client={proxy=inbound}, recv HEADERS; frame=Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: false } }\r\n2018-05-28T19:14:52.581484835Z srv TRCE h2::proto::connection client={proxy=inbound}, recv DATA; frame=Data { stream_id: StreamId(1), flags: DataFlags }\r\n2018-05-28T19:14:52.585093113Z srv TRCE h2::proto::connection client={proxy=inbound}, recv HEADERS; frame=Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: true } }\r\n2018-05-28T19:14:52.585180634Z srv TRCE h2::proto::connection client={proxy=inbound}, recv GOAWAY; frame=GoAway { last_stream_id: StreamId(2147483647), error_code: NO_ERROR }\r\n2018-05-28T19:14:52.585189910Z srv TRCE h2::proto::connection client={proxy=inbound}, recv PING; frame=Ping { ack: false, payload: [1, 6, 1, 8, 0, 3, 3, 9] }\r\n2018-05-28T19:14:52.596526701Z srv TRCE h2::proto::streams::prioritize (\"inbound\", V4(10.1.1.196:52162)), writing frame=Frame::Headers(Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: false } })\r\n2018-05-28T19:14:52.607168897Z mid TRCE h2::proto::connection client={proxy=outbound}, recv HEADERS; frame=Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: false } }\r\n2018-05-28T19:14:52.607208425Z mid TRCE h2::proto::connection client={proxy=outbound}, recv DATA; frame=Data { stream_id: StreamId(1), flags: DataFlags }\r\n2018-05-28T19:14:52.607209737Z srv TRCE h2::proto::streams::prioritize (\"inbound\", V4(10.1.1.196:52162)), writing frame=Frame::Data(Data { stream_id: StreamId(1), flags: DataFlags })\r\n2018-05-28T19:14:52.607240388Z mid TRCE h2::proto::connection client={proxy=outbound}, recv HEADERS; frame=Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: true } }\r\n2018-05-28T19:14:52.607260469Z srv TRCE h2::proto::streams::prioritize (\"inbound\", V4(10.1.1.196:52162)), writing frame=Frame::Headers(Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: true } })\r\n2018-05-28T19:14:52.624208705Z mid TRCE h2::proto::streams::prioritize (\"outbound\", V4(10.1.1.196:51912)), writing frame=Frame::Headers(Headers { stream_id: StreamId(3), flags: HeadersFlag { end_stream: false } })\r\n2018-05-28T19:14:52.625716796Z mid TRCE h2::proto::streams::prioritize (\"outbound\", V4(10.1.1.196:51912)), writing frame=Frame::Data(Data { stream_id: StreamId(3), flags: DataFlags })\r\n2018-05-28T19:14:52.626403215Z mid TRCE h2::proto::streams::prioritize (\"outbound\", V4(10.1.1.196:51912)), writing frame=Frame::Headers(Headers { stream_id: StreamId(3), flags: HeadersFlag { end_stream: true } })\r\n2018-05-28T19:14:52.630189717Z mid TRCE h2::proto::connection server={proxy=outbound; peer=10.1.1.196:51912} recv WINDOW_UPDATE; frame=WindowUpdate { stream_id: StreamId(0), size_increment: 78 }\r\n2018-05-28T19:14:52.631078667Z mid TRCE h2::proto::connection server={proxy=outbound; peer=10.1.1.196:51912} recv PING; frame=Ping { ack: false, payload: [2, 4, 16, 16, 9, 14, 7, 7] }\r\n2018-05-28T19:14:53.062622642Z mid TRCE h2::proto::streams::prioritize client={proxy=outbound; addr=10.1.1.195:9090; proto=Http2}, writing frame=Frame::Headers(Headers { stream_id: StreamId(3), flags: HeadersFlag { end_stream: false } })\r\n2018-05-28T19:14:53.074354565Z mid TRCE h2::proto::streams::prioritize client={proxy=outbound; addr=10.1.1.195:9090; proto=Http2}, writing frame=Frame::Data(Data { stream_id: StreamId(3), flags: DataFlags { end_stream: true } })\r\n```\r\n\r\n#### Request 3\r\n\r\n_mid_ sends another request to _srv_. This time, a destination removal event hasn't been\r\nreceived, so the request goes on the same connection as last time. However, Kubernetes\r\nputs the application in `CrashLoopBackoff` so the proxy can't forward the request and it\r\neventually times out:\r\n\r\n```\r\n2018-05-28T19:14:53.044642518Z mid TRCE h2::proto::connection server={proxy=outbound; peer=10.1.1.196:51912} recv HEADERS; frame=Headers { stream_id: StreamId(5), flags: HeadersFlag { end_stream: false } }\r\n2018-05-28T19:14:53.047798177Z mid TRCE h2::proto::connection server={proxy=outbound; peer=10.1.1.196:51912} recv DATA; frame=Data { stream_id: StreamId(5), flags: DataFlags { end_stream: true } }\r\n2018-05-28T19:14:53.062282130Z srv TRCE h2::proto::connection server={proxy=inbound; peer=10.1.1.196:52162} recv HEADERS; frame=Headers { stream_id: StreamId(3), flags: HeadersFlag { end_stream: false } }\r\n2018-05-28T19:14:53.076237679Z srv TRCE h2::proto::connection server={proxy=inbound; peer=10.1.1.196:52162} recv DATA; frame=Data { stream_id: StreamId(3), flags: DataFlags { end_stream: true } }\r\n2018-05-28T19:14:53.071404760Z srv WARN conduit_proxy::bind   In { addr: V4(10.1.1.195:9090), proto: Http2 }, connect error to Endpoint { address: V4(10.1.1.195:9090), dst_labels: None }: Error attempting to establish underlying session layer: Connection refused (os error 111)\r\n2018-05-28T19:15:02.071820317Z srv DBUG conduit_proxy::bind   In { addr: V4(10.1.1.195:9090), proto: Http2 }, connect error to Endpoint { address: V4(10.1.1.195:9090), dst_labels: None }: Error attempting to establish underlying session layer: Connection refused (os error 111)\r\n2018-05-28T19:15:03.033557940Z mid ERR! conduit_proxy serve={proxy=outbound; addr=127.0.0.1:4140}, turning operation timed out after 10s into 500\r\n2018-05-28T19:15:03.043722011Z srv TRCE h2::proto::connection server={proxy=inbound; peer=10.1.1.196:52162} recv RST_STREAM; frame=Reset { stream_id: StreamId(3), error_code: CANCEL }\r\n2018-05-28T19:15:03.047542662Z mid TRCE h2::proto::connection server={proxy=outbound; peer=10.1.1.196:51912} recv RST_STREAM; frame=Reset { stream_id: StreamId(5), error_code: CANCEL }\r\n```\r\n\r\n#### Request 4\r\n\r\n_mid_ sends another request to _srv_. This time, the only endpoint is removed from service\r\ndiscovery as we attempt to route it.\r\n\r\n```\r\n2018-05-28T19:15:03.040195266Z mid TRCE h2::proto::streams::prioritize client={proxy=outbound; addr=10.1.1.195:9090; proto=Http2}, writing frame=Frame::Reset(Reset { stream_id: StreamId(3), error_code: CANCEL })\r\n2018-05-28T19:15:03.043738152Z mid TRCE h2::proto::streams::prioritize (\"outbound\", V4(10.1.1.196:51912)), writing frame=Frame::Headers(Headers { stream_id: StreamId(5), flags: HeadersFlag { end_stream: true } })\r\n2018-05-28T19:15:03.061682240Z mid TRCE h2::proto::connection server={proxy=outbound; peer=10.1.1.196:51912} recv HEADERS; frame=Headers { stream_id: StreamId(7), flags: HeadersFlag { end_stream: false } }\r\n2018-05-28T19:15:03.066603938Z mid TRCE h2::proto::connection server={proxy=outbound; peer=10.1.1.196:51912} recv DATA; frame=Data { stream_id: StreamId(7), flags: DataFlags { end_stream: true } }\r\n2018-05-28T19:15:03.075574493Z mid TRCE conduit_proxy::control::destination Out { dest: Hostname(DnsNameAndPort { host: Name(\"server.bb.svc.cluster.local\"), port: 9090 }), proto: Http2 }, watch: Ok(Ready(Some(Remove(V4(10.1.1.195:9090)))))\r\n```\r\n\r\nBut the endpoint is restored before the bind timeout is reached. As the endpoint is\r\nrestored, a new connection is established from _mid_ to _srv_ and the request is issued\r\n\r\n```\r\n2018-05-28T19:15:07.730676665Z mid TRCE conduit_proxy::control::destination Out { dest: Hostname(DnsNameAndPort { host: Name(\"server.bb.svc.cluster.local\"), port: 9090 }), proto: Http2 }, watch: Ok(Ready(Some(Insert(V4(10.1.1.195:9090), Metadata { metric_labels: Some(DstLabels { formatted: \"dst_conduit_io_control_plane_ns=\\\"conduit\\\",dst_deployment=\\\"server\\\",dst_namespace=\\\"bb\\\",dst_pod=\\\"server-65d49dd9dc-b982b\\\",dst_pod_template_hash=\\\"2180588587\\\",dst_service=\\\"server\\\"\", original: {\"namespace\": \"bb\", \"service\": \"server\", \"pod_template_hash\": \"2180588587\", \"deployment\": \"server\", \"conduit_io_control_plane_ns\": \"conduit\", \"pod\": \"server-65d49dd9dc-b982b\"} }) }))))\r\n2018-05-28T19:15:07.738428186Z srv TRCE conduit_proxy::transparency::server server={proxy=inbound; peer=10.1.1.196:52314} transparency detected HTTP/2\r\n2018-05-28T19:15:07.743387003Z mid TRCE h2::proto::connection client={proxy=outbound}, recv SETTINGS; frame=Settings { flags: SettingsFlags(0), ... }\r\n2018-05-28T19:15:07.744950182Z mid TRCE h2::proto::streams::prioritize client={proxy=outbound; addr=10.1.1.195:9090; proto=Http2}, writing frame=Frame::Headers(Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: false } })\r\n2018-05-28T19:15:07.748623048Z srv TRCE h2::proto::connection server={proxy=inbound; peer=10.1.1.196:52314} recv SETTINGS; frame=Settings { flags: SettingsFlags(0), ... }\r\n2018-05-28T19:15:07.752403469Z mid TRCE h2::proto::streams::prioritize client={proxy=outbound; addr=10.1.1.195:9090; proto=Http2}, writing frame=Frame::Data(Data { stream_id: StreamId(1), flags: DataFlags { end_stream: true } })\r\n2018-05-28T19:15:07.753386678Z srv TRCE h2::proto::connection server={proxy=inbound; peer=10.1.1.196:52314} recv SETTINGS; frame=Settings { flags: SettingsFlags(1), ... }\r\n2018-05-28T19:15:07.756102494Z srv TRCE h2::proto::connection server={proxy=inbound; peer=10.1.1.196:52314} recv HEADERS; frame=Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: false } }\r\n2018-05-28T19:15:07.760171124Z srv TRCE h2::proto::connection server={proxy=inbound; peer=10.1.1.196:52314} recv DATA; frame=Data { stream_id: StreamId(1), flags: DataFlags { end_stream: true } }\r\n2018-05-28T19:15:07.761401808Z mid TRCE h2::proto::connection client={proxy=outbound}, recv SETTINGS; frame=Settings { flags: SettingsFlags(1), ... }\r\n```\r\n\r\nHere's where things get weird: somehow **two requests** are issued to the application. We\r\nonly send request **headers** for the second request, however. The first request is never\r\nfully sent so, even though the application issues a GoAway, it does not terminate since\r\nit's waiting for this request to complete.\r\n\r\nPerhaps _Request 3_, which was canceled, is somehow queued and partially sent?\r\n\r\n```\r\n2018-05-28T19:15:08.098993188Z srv TRCE h2::proto::streams::prioritize client={proxy=inbound; addr=10.1.1.195:9090; proto=Http2}, writing frame=Frame::Headers(Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: false } })\r\n2018-05-28T19:15:08.100790694Z srv TRCE h2::proto::streams::prioritize client={proxy=inbound; addr=10.1.1.195:9090; proto=Http2}, writing frame=Frame::Headers(Headers { stream_id: StreamId(3), flags: HeadersFlag { end_stream: false } })\r\n2018-05-28T19:15:08.119429334Z srv TRCE h2::proto::connection client={proxy=inbound}, recv SETTINGS; frame=Settings { flags: SettingsFlags(0), ... }\r\n2018-05-28T19:15:08.120500366Z srv TRCE h2::proto::connection client={proxy=inbound}, recv SETTINGS; frame=Settings { flags: SettingsFlags(1), ... }\r\n2018-05-28T19:15:08.123763008Z srv TRCE h2::proto::streams::prioritize client={proxy=inbound; addr=10.1.1.195:9090; proto=Http2}, writing frame=Frame::Data(Data { stream_id: StreamId(3), flags: DataFlags { end_stream: true } })\r\n2018-05-28T19:15:08.127286629Z srv TRCE h2::proto::connection client={proxy=inbound}, recv WINDOW_UPDATE; frame=WindowUpdate { stream_id: StreamId(0), size_increment: 66 }\r\n2018-05-28T19:15:08.127964826Z srv TRCE h2::proto::connection client={proxy=inbound}, recv PING; frame=Ping { ack: false, payload: [2, 4, 16, 16, 9, 14, 7, 7] }\r\n2018-05-28T19:15:08.128894074Z srv TRCE h2::proto::connection client={proxy=inbound}, recv HEADERS; frame=Headers { stream_id: StreamId(3), flags: HeadersFlag { end_stream: false } }\r\n2018-05-28T19:15:08.131167852Z srv TRCE h2::proto::connection client={proxy=inbound}, recv DATA; frame=Data { stream_id: StreamId(3), flags: DataFlags }\r\n2018-05-28T19:15:08.133010468Z srv TRCE h2::proto::connection client={proxy=inbound}, recv HEADERS; frame=Headers { stream_id: StreamId(3), flags: HeadersFlag { end_stream: true } }\r\n2018-05-28T19:15:08.134929940Z srv TRCE h2::proto::connection client={proxy=inbound}, recv GOAWAY; frame=GoAway { last_stream_id: StreamId(2147483647), error_code: NO_ERROR }\r\n2018-05-28T19:15:08.135516526Z srv TRCE h2::proto::connection client={proxy=inbound}, recv PING; frame=Ping { ack: false, payload: [1, 6, 1, 8, 0, 3, 3, 9] }\r\n2018-05-28T19:15:08.154145304Z srv TRCE h2::proto::streams::prioritize (\"inbound\", V4(10.1.1.196:52314)), writing frame=Frame::Headers(Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: false } })\r\n2018-05-28T19:15:08.161287802Z srv TRCE h2::proto::streams::prioritize (\"inbound\", V4(10.1.1.196:52314)), writing frame=Frame::Data(Data { stream_id: StreamId(1), flags: DataFlags })\r\n2018-05-28T19:15:08.164750313Z srv TRCE h2::proto::streams::prioritize (\"inbound\", V4(10.1.1.196:52314)), writing frame=Frame::Headers(Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: true } })\r\n2018-05-28T19:15:08.164774155Z srv TRCE h2::proto::connection client={proxy=inbound}, recv GOAWAY; frame=GoAway { last_stream_id: StreamId(3), error_code: NO_ERROR }\r\n2018-05-28T19:15:08.164781915Z mid TRCE h2::proto::connection client={proxy=outbound}, recv HEADERS; frame=Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: false } }\r\n2018-05-28T19:15:08.164797485Z mid TRCE h2::proto::connection client={proxy=outbound}, recv DATA; frame=Data { stream_id: StreamId(1), flags: DataFlags }\r\n2018-05-28T19:15:08.164821074Z mid TRCE h2::proto::connection client={proxy=outbound}, recv HEADERS; frame=Headers { stream_id: StreamId(1), flags: HeadersFlag { end_stream: true } }\r\n2018-05-28T19:15:08.176461219Z mid TRCE h2::proto::streams::prioritize (\"outbound\", V4(10.1.1.196:51912)), writing frame=Frame::Headers(Headers { stream_id: StreamId(7), flags: HeadersFlag { end_stream: false } })\r\n2018-05-28T19:15:08.176547355Z mid TRCE h2::proto::streams::prioritize (\"outbound\", V4(10.1.1.196:51912)), writing frame=Frame::Data(Data { stream_id: StreamId(7), flags: DataFlags })\r\n2018-05-28T19:15:08.176589472Z mid TRCE h2::proto::streams::prioritize (\"outbound\", V4(10.1.1.196:51912)), writing frame=Frame::Headers(Headers { stream_id: StreamId(7), flags: HeadersFlag { end_stream: true } })\r\n2018-05-28T19:15:08.182875364Z mid TRCE h2::proto::connection server={proxy=outbound; peer=10.1.1.196:51912} recv WINDOW_UPDATE; frame=WindowUpdate { stream_id: StreamId(0), size_increment: 78 }\r\n2018-05-28T19:15:08.184368051Z mid TRCE h2::proto::connection server={proxy=outbound; peer=10.1.1.196:51912} recv PING; frame=Ping { ack: false, payload: [2, 4, 16, 16, 9, 14, 7, 7] }\r\n2018-05-28T19:15:08.197935769Z mid TRCE h2::proto::connection server={proxy=outbound; peer=10.1.1.196:51912} recv HEADERS; frame=Headers { stream_id: StreamId(9), flags: HeadersFlag { end_stream: false } }\r\n2018-05-28T19:15:08.203245306Z mid TRCE h2::proto::connection server={proxy=outbound; peer=10.1.1.196:51912} recv DATA; frame=Data { stream_id: StreamId(9), flags: DataFlags { end_stream: true } }\r\n2018-05-28T19:15:08.212146861Z mid TRCE h2::proto::streams::prioritize client={proxy=outbound; addr=10.1.1.195:9090; proto=Http2}, writing frame=Frame::Headers(Headers { stream_id: StreamId(3), flags: HeadersFlag { end_stream: false } })\r\n2018-05-28T19:15:08.217834309Z srv TRCE h2::proto::connection server={proxy=inbound; peer=10.1.1.196:52314} recv HEADERS; frame=Headers { stream_id: StreamId(3), flags: HeadersFlag { end_stream: false } }\r\n2018-05-28T19:15:08.237250796Z mid TRCE h2::proto::streams::prioritize client={proxy=outbound; addr=10.1.1.195:9090; proto=Http2}, writing frame=Frame::Data(Data { stream_id: StreamId(3), flags: DataFlags { end_stream: true } })\r\n2018-05-28T19:15:08.249470148Z srv TRCE h2::proto::connection server={proxy=inbound; peer=10.1.1.196:52314} recv DATA; frame=Data { stream_id: StreamId(3), flags: DataFlags { end_stream: true } }\r\n2018-05-28T19:15:08.254502064Z srv WARN conduit_proxy::bind   In { addr: V4(10.1.1.195:9090), proto: Http2 }, connect error to Endpoint { address: V4(10.1.1.195:9090), dst_labels: None }: Error attempting to establish underlying session layer: Connection refused (os error 111)\r\n```\r\n"}}
{"actor":{"display_login":"olix0r"}}