{"push_id":2406030480,"size":1,"distinct_size":1,"ref":"refs/heads/master","head":"ead756844ce9ada904fcc3666df25692f63e76b8","before":"8204f43f0204c172c4a799f9ccda0592d3b02138","commits":[{"sha":"ead756844ce9ada904fcc3666df25692f63e76b8","author":{"name":"Todd Lipcon","email":"dd5965a1ccaca30b2dd5053f256a6d9b88ca567f@apache.org"},"message":"java: reduce scary exception spew and improve logging\n\nIn real environments there are various types of failures that are\n\"expected\". In particular, it's not unlikely for the client to\noccasionally connect to a server which has gone down or which is not the\nleader for a given tablet anymore. In that case, we should make sure\nthat the logging output is easy to read and doesn't include irrelevant\nstack traces stemming from Netty internals.\n\nThis patch addresses several of those cases. In particular, it gets rid\nof \"unexpected exception\" messages (and associated stacks) in the case\nof \"connection refused\" errors. It also helpfully includes the\nunderlying error when invalidating the tablet cache for a location.\n\nThere's a relatively simple new unit test, and I also ran some\nbefore/after comparisons using the \"loadgen\" sample from the\nkudu-examples repo against a local cluster with one TS:\n\nCase 1: Master down when we start the client:\n============================================================\n\nBefore the patch\n--------------\n[New I/O worker #12] WARN org.apache.kudu.client.ConnectToCluster - Error receiving response from localhost:7051\norg.apache.kudu.client.RecoverableException: connection closed\n\tat org.apache.kudu.client.Connection.channelClosed(Connection.java:254)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88)\n\tat org.apache.kudu.client.Connection.handleUpstream(Connection.java:238)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.apache.kudu.shaded.org.jboss.netty.handler.timeout.ReadTimeoutHandler.channelClosed(ReadTimeoutHandler.java:176)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.apache.kudu.shaded.org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:60)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:493)\n\tat org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.channelClosed(FrameDecoder.java:371)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireChannelClosed(Channels.java:468)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.Channels$6.run(Channels.java:457)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.socket.ChannelRunnableWrapper.run(ChannelRunnableWrapper.java:40)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)\n\tat org.apache.kudu.shaded.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)\n\tat org.apache.kudu.shaded.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n[New I/O boss #17] ERROR org.apache.kudu.client.Connection - [peer master-localhost:7051(localhost:7051)] unexpected exception from downstream on [id: 0xf9759051]\njava.net.ConnectException: Connection refused: localhost/127.0.0.1:7051\n\tat sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)\n\tat sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)\n\tat org.apache.kudu.shaded.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)\n\tat org.apache.kudu.shaded.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n[New I/O worker #12] WARN org.apache.kudu.client.ConnectToCluster - Unable to find the leader master localhost:7051; will retry\n\n... repeats with exponential backoff\n\nAfter the patch\n--------------\n[New I/O boss #17] INFO org.apache.kudu.client.Connection - Failed to connect to peer master-localhost:7051(localhost:7051): Connection refused: localhost/127.0.0.1:7051\n[New I/O worker #1] INFO org.apache.kudu.client.ConnectToCluster - Unable to connect to master localhost:7051: Connection refused: localhost/127.0.0.1:7051\n[New I/O worker #1] WARN org.apache.kudu.client.ConnectToCluster - Unable to find the leader master localhost:7051; will retry\n\n... repeats with exponential backoff\n\nCase 2: Tserver is down\n============================================================\n\nBefore the patch\n----------------\n[New I/O worker #11] INFO org.apache.kudu.client.AsyncKuduClient - Removing server fc9f6ab955cc47a8ab3653d0170305b0 from this tablet's cache 9da164fbcd93404ea8e38e9491eb3fa8\n[New I/O boss #17] ERROR org.apache.kudu.client.Connection - [peer fc9f6ab955cc47a8ab3653d0170305b0(todd-laptop:7050)] unexpected exception from downstream on [id: 0x270e3a82]\njava.net.ConnectException: Connection refused: todd-laptop/127.0.1.1:7050\n\tat sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)\n\tat sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)\n\tat org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)\n\tat org.apache.kudu.shaded.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)\n\tat org.apache.kudu.shaded.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n\n... repeats with exponential backoff\n\nAfter the patch\n---------------\n[New I/O boss #17] INFO org.apache.kudu.client.Connection - Failed to connect to peer fc9f6ab955cc47a8ab3653d0170305b0(todd-laptop:7050): Connection refused: todd-laptop/127.0.1.1:7050\n[New I/O worker #15] INFO org.apache.kudu.client.AsyncKuduClient - Invalidating location fc9f6ab955cc47a8ab3653d0170305b0(todd-laptop:7050) for tablet 478ddbaafb5b494ea0de8e2d0ed45a00: Connection refused: todd-laptop/127.0.1.1:7050\n\n... repeats with exponential backoff\n\nCase 3: A tablet is bad\n============================================================\nIn this case I manually delete a non-replicated tablet that is being written to\nusing 'kudu remote_replica delete'. This is unrecoverable since there is\nno other replica.\n\nBefore the patch\n-----------------\n[New I/O worker #10] INFO org.apache.kudu.client.AsyncKuduClient - Removing server fc9f6ab955cc47a8ab3653d0170305b0 from this tablet's cache a633f4a0c14b4404a8ae1f825b378867\n\n... repeats with exponential backoff\n\nAfter the patch\n----------------\n[New I/O worker #9] INFO org.apache.kudu.client.AsyncKuduClient - Invalidating location 36622c46b25f4da9ada43f8591728053(todd-laptop:7050) for tablet c0b9488597e8447595ad0ee0a3378f95: Tablet not RUNNING: STOPPED\n\n... repeats with exponential backoff\n\nChange-Id: I4b8be871693fecc1ee46a4e238dd2ed8f0730d4b\nReviewed-on: http://gerrit.cloudera.org:8080/9644\nReviewed-by: Grant Henke <granthenke@gmail.com>\nTested-by: Grant Henke <granthenke@gmail.com>","distinct":true,"url":"https://api.github.com/repos/cloudera/kudu/commits/ead756844ce9ada904fcc3666df25692f63e76b8"}]}
{"actor":{"display_login":"cloudera-gerrit"}}