{"push_id":1207333307,"size":1,"distinct_size":1,"ref":"refs/heads/3.x-pxc-426","head":"815a9a05e0e9077000483ea8c6182ecc71675757","before":"f324a3424a6236fea5c9abba9fb243f468d5e2ef","commits":[{"sha":"815a9a05e0e9077000483ea8c6182ecc71675757","author":{"name":"Julius Goryavsky","email":"18f7161cd8c950d81f2c0c29770532d6dbbeb9a7@gmail.com"},"message":"- #PXC-426: Race condition during IST\n\nThe node lost connectivity with other nodes in the cluster for some time due to network problems. After this, it need to run IST to synchronize their state with the cluster. However, the IST request has failed.\n\nIf we already have large network delays and packet losses, then during IST we can encounter situation when seqno on the donor node has moved forward so much that IST is not possible and we need to run \"full\" SST request.\n\nTo avoid unsuccessful IST attempts, the Galera introduces the concept of \"safety gap\" - the window for seqno values in which the heuristic algorithm for selecting the donor prefers to switch to SST.\n\nThe immediate cause of the PXC-426 error is the inability of the PXC to start new SST request during node operation (using methods other than mysqldump). PXC and Galera written in such way that normally we need a full restart before launching the new SST.\n\nTo resolve this contradiction, we need to change the heuristic algorithm that selecting a donor in such way, that it took into account the impossibility of SST. If a joiner node sends a request without SST part, then after applying this patch the donor node must ignore the safety gap window and still tries to execute IST.\n\nIt is much better to have a small probability of failure in the IST, than immediately suffer from failure with 100% guarantee (due to heuristics associated with safety gap window). Moreover, if the request contains the SST-related part (not only IST), then we can leave current logic with the additional heuristics related to safety gap window.\n\nHowever, the algorithm of the donor selection, which implemented in the GCS, contains the potential flaw, and in rare situation it still may choose as a donor such node, where seqno can gone forward before the IST request will be received.\n\nTherefore, since heuristic that selects the donor node does not give us a 100% guarantee that seqno does not moved forward while the new node joining the cluster, and during node operation we have only a requests for the IST without SST part, then we need a way to informing joiner node, that it should prepare to receive SST. This return path also added to the patch code.\n\nEspecially we need this signal to notify the user that server unable to perform the SST instead of IST when xtrabackup or rsynch selected as the state transfer method.\n\nIn these cases, we need to show the user an explicit message that indicates that we need to perform SST after node restart. However, current version of the server continues to work, even when the synchronization of the node is no longer possible and it cannot continue normal operation in a cluster. Or vice versa, it falls for one of the assertions in the Galera code. This corrected in the patch.\n\nTheoretically, a similar return path also may be activated if the MySQL server does not prepared structures for the SST in the view callback (in the wsrep_view_handler_cb function) after restart of the node. However, in the current version of PXC this should not happen after node restart, because the wsrep_sst_prepare function (which called from wsrep_view_handler_cb) always returns a non-zero value as length of the SST request (node always prepared for both the SST and IST). However, the assertion in the wsrep_view_handler_cb function only checks for the presence of the SST structure, but not its length. Theoretically, when mysqldump method is used this structure can be created with zero length of the SST part.\n\nTherefore, I changed the Galera code to allow restart SST after failing to perform IST - to be protected against failures in the future, when implementation of the wsrep_sst_prepare function may change.\n\nIn addition, if the IST request canceled due to an error at the GCS level or by initiative of the server, and if the node remain in the S_JOINING state, then we must return it to the S_CONNECTED state.","distinct":true,"url":"https://api.github.com/repos/natsys/galera/commits/815a9a05e0e9077000483ea8c6182ecc71675757"}]}
{"actor":{"display_login":"sysprg"}}