{"action":"created","issue":{"url":"https://api.github.com/repos/dropwizard/metrics/issues/917","repository_url":"https://api.github.com/repos/dropwizard/metrics","labels_url":"https://api.github.com/repos/dropwizard/metrics/issues/917/labels{/name}","comments_url":"https://api.github.com/repos/dropwizard/metrics/issues/917/comments","events_url":"https://api.github.com/repos/dropwizard/metrics/issues/917/events","html_url":"https://github.com/dropwizard/metrics/issues/917","id":133101243,"number":917,"title":"Graphite reporter does not detect connections in CLOSE_WAIT state","user":{"login":"bwhaley","id":503816,"avatar_url":"https://avatars.githubusercontent.com/u/503816?v=3","gravatar_id":"","url":"https://api.github.com/users/bwhaley","html_url":"https://github.com/bwhaley","followers_url":"https://api.github.com/users/bwhaley/followers","following_url":"https://api.github.com/users/bwhaley/following{/other_user}","gists_url":"https://api.github.com/users/bwhaley/gists{/gist_id}","starred_url":"https://api.github.com/users/bwhaley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bwhaley/subscriptions","organizations_url":"https://api.github.com/users/bwhaley/orgs","repos_url":"https://api.github.com/users/bwhaley/repos","events_url":"https://api.github.com/users/bwhaley/events{/privacy}","received_events_url":"https://api.github.com/users/bwhaley/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-02-11T22:11:53Z","updated_at":"2016-02-11T22:27:30Z","closed_at":null,"body":"I have an environment with two Graphite carbon-relays behind an ELB. The metrics reporter connects to the ELB which proxies to the relays. This mostly works except that the IP address of the ELB changes periodically. When this happens, the ELB sends a FIN and the connection should be closed. At that point the client needs to do another DNS lookup to find the new IP address. This works fine with other clients (for example, collectd) but not with v3.1.2 of the metrics reporter. Connections remain in CLOSE_WAIT. \r\n\r\nTo wit:\r\n\r\n```\r\n$ sudo netstat -pan | grep 2003\r\ntcp        1      0 172.17.53.222:43049         172.17.5.77:2003            CLOSE_WAIT  14884/java\r\ntcp        0      0 172.17.53.222:46365         172.17.5.243:2003           ESTABLISHED 8077/collectd\r\n```\r\n\r\nThis is several hours after the IP address has changed. Note that collectd has picked up the new ELB IP address but the java process hasn't. The related stack trace:\r\n\r\n```\r\n[WARN ] 2016-02-11 20:08:02.982 [metrics-graphite-reporter-1-thread-1] GraphiteReporter - Unable to report to Graphite\r\njava.net.SocketException: Broken pipe\r\n        at java.net.SocketOutputStream.socketWrite0(Native Method) ~[?:1.8.0_51]\r\n        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) ~[?:1.8.0_51]\r\n        at java.net.SocketOutputStream.write(SocketOutputStream.java:153) ~[?:1.8.0_51]\r\n        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) ~[?:1.8.0_51]\r\n        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291) ~[?:1.8.0_51]\r\n        at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295) ~[?:1.8.0_51]\r\n        at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141) ~[?:1.8.0_51]\r\n        at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) ~[?:1.8.0_51]\r\n        at java.io.BufferedWriter.flush(BufferedWriter.java:254) ~[?:1.8.0_51]\r\n        at com.codahale.metrics.graphite.Graphite.flush(Graphite.java:151) ~[metrics-graphite-3.1.2.jar!/:3.1.2]\r\n        at com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:190) [metrics-graphite-3.1.2.jar!/:3.1.2]\r\n        at com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162) [metrics-core-3.1.2.jar!/:3.1.2]\r\n        at com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117) [metrics-core-3.1.2.jar!/:3.1.2]\r\n        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_51]\r\n        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_51]\r\n        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_51]\r\n        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_51]\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_51]\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_51]\r\n        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_51]\r\n```\r\n\r\nI ain't passed the bar but I know a lil' bit, enough that I think the problem lies in `isConnected()` in [`Graphite.java`](https://github.com/dropwizard/metrics/blob/068821a9798ddf81c4fd457b8fe1df09a0a99e22/metrics-graphite/src/main/java/io/dropwizard/metrics/graphite/Graphite.java#L128-L130). \r\n\r\n\r\n```java\r\n    public boolean isConnected() {\r\n    \t\treturn socket != null && socket.isConnected() && !socket.isClosed();\r\n    }\r\n```\r\n\r\n`socket.isConnected()` and `socket.isClosed()` is not sufficient for checking whether the remote end has closed the connection. Instead, it needs a `read()`, which will return -1 if the connection is no longer open.\r\n"},"comment":{"url":"https://api.github.com/repos/dropwizard/metrics/issues/comments/183084645","html_url":"https://github.com/dropwizard/metrics/issues/917#issuecomment-183084645","issue_url":"https://api.github.com/repos/dropwizard/metrics/issues/917","id":183084645,"user":{"login":"bwhaley","id":503816,"avatar_url":"https://avatars.githubusercontent.com/u/503816?v=3","gravatar_id":"","url":"https://api.github.com/users/bwhaley","html_url":"https://github.com/bwhaley","followers_url":"https://api.github.com/users/bwhaley/followers","following_url":"https://api.github.com/users/bwhaley/following{/other_user}","gists_url":"https://api.github.com/users/bwhaley/gists{/gist_id}","starred_url":"https://api.github.com/users/bwhaley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bwhaley/subscriptions","organizations_url":"https://api.github.com/users/bwhaley/orgs","repos_url":"https://api.github.com/users/bwhaley/repos","events_url":"https://api.github.com/users/bwhaley/events{/privacy}","received_events_url":"https://api.github.com/users/bwhaley/received_events","type":"User","site_admin":false},"created_at":"2016-02-11T22:27:30Z","updated_at":"2016-02-11T22:27:30Z","body":"I'm no strange to Java DNS caching issues, that could very well be at fault here. Are you suggesting I should set `networkaddress.cache.ttl` to match the ELB's TTL (which is 60s)?"}}