{"push_id":959723130,"size":1,"distinct_size":1,"ref":"refs/heads/cdh5-trunk","head":"d8bdfaecf57999e0cb0fa4a8b26507a3781bf665","before":"29825f0128863a00bd48a240911b6cedf3c09e7a","commits":[{"sha":"d8bdfaecf57999e0cb0fa4a8b26507a3781bf665","author":{"email":"3eca10f30041813f045165784e24b5a950a6cc7e@cloudera.com","name":"Henry Robinson"},"message":"IMPALA-1599: Allow data stream senders to start before receivers\n\nThis patch allows data stream senders to start before their receivers,\nby forcing the sender to block during its first transmission until the\nreceiver is ready.\n\nThis is done by establishing a 'rendezvous' between sender and receiver,\nbased on a Promise whose return type is the receiver itself. All senders\nthat can not find their receiver wait on that Promise. If the receiver\narrives promptly, it will trigger the Promise and the senders can all\ncontinue. If the receiver does not arrive within 60s, the sender will\ntime out.\n\nTODO: When a receiver isn't present, the sender will fail quietly (this\nwas always the case, see comments). We should look at fixing this,\nalthough the false-positives need working around.\n\nThis commit also adds a 'tombstone' cache for recently cancelled\nreceivers.\n\nWhen a data-stream sender can't find its receiver (during\nTransmitData()), it waits for up to 60s by default for the receiver to\narrive. However, this can lead to unnecessary waiting in the case when\nthe receiver has already been closed, and will never show up.\n\nIn most cases, this should not happen, but there are two that are worthy\nof some attention:\n\n  * Cancellation happens while TransmitData() is in-flight, and the\n    receiver is closed, but the sender does not know until the RPC times\n    out (we don't have cancellable RPCs which would be the natural way\n    to implement this).\n  * The receiver is closed forcibly through some other path than the\n    usual EOS-receipt from all its senders. This happens, for example,\n    when an agg. node with a limit is above an exchange - the agg. node\n    will close the exchange once it considers all the rows it needs to\n    be read. Then we can have the situation where the sender is trying\n    to TransmitData() to close its connection to the exchange, but the\n    exchange has already closed.\n\nThe result is that any code that waits for a backend to be completed\nwill take longer due to hitting the timeout path.\n\nTo address this, this patch adds a cache of recently expired streams to\nthe data stream manager. Any TransmitData() call that is targeting a\nstream in that cache will fail fast. Entries are kept in the cache for\n30s, and are evicted collaboratively by other closing senders during\nCloseSender().\n\nChange-Id: If9a90f6980ab2cbd4b59b54f737da73c6af88f1a\nReviewed-on: http://gerrit.cloudera.org:8080/493\nReviewed-by: Henry Robinson <henry@cloudera.com>\nTested-by: Internal Jenkins","distinct":true,"url":"https://api.github.com/repos/cloudera/Impala/commits/d8bdfaecf57999e0cb0fa4a8b26507a3781bf665"}]}