{"push_id":1352761023,"size":1,"distinct_size":1,"ref":"refs/heads/3.x-pxc-252","head":"8fd4491323cf0bb5debf67b58173a63a49d90148","before":"bd47fc94fe726044c23f291f26998c4656ae916d","commits":[{"sha":"8fd4491323cf0bb5debf67b58173a63a49d90148","author":{"name":"Julius Goryavsky","email":"18f7161cd8c950d81f2c0c29770532d6dbbeb9a7@gmail.com"},"message":"- #PXC-252: [pz@percona.com: Testing Auto Recover]\n\nWe cannot restart the SST automatically whenever SST fails because the server usually does not have enough information to understand why this was happened. For example, the analysis of the log indicates that the connection has been restored and the 22 error was not fatal:\n\n```\n>2014-12-05 17:16:08 2596 [Warning] WSREP: Could not find peer:\n>2014-12-05 17:16:08 2596 [Warning] WSREP: 1.0 (nuc3): State transfer to\n>-1.-1 (left the group) failed: -22 (Invalid argument)\n...\n>2014-12-05 17:16:08 2596 [Note] WSREP: Member 1.0 (nuc3) synced with group\n```\n\nAt the same time log analysis shows that in fact the SST process was completed with another error = 1 (\"Operation not permitted\"). Perhaps it was a mistake due to insufficient access rights or incorrect password, but not a network error.\n\nIf this diagnosis is correct, then restart of the SST did not help us to resolve this problem. If it is wrong, it turns out that at this point we do not have enough information to make the right decision. A simple patch cannot eliminate this (in this case, I think here we need to talk about architecture refining as a separate development task, not just a patch on the product support level).\n\nHowever, we definitely has a serious problem that we should eliminate. Namely - after detecting a failure due to permissions, the server process is completed improperly, leaving the unfinished thread:\n\n```\nError in my_thread_global_end(): 1 threads didn't exit\n>141205 17:16:28 mysqld_safe mysqld from pid file /var/lib/mysql/mysqld.pid\n>ended\n```\n\nWe have already eliminated this error in most scenarios by a previous patches, but I found new possible branches, which similar error arise (for example, it is possible when we repeat the connection attempts after the failure of the SST - although I was not able to reproduce this situation manually or by automatic script since it is rare race condition).\n\nIn addition, I found that the failure during the SST, which is diagnosed on the Galera level, leads to the completion of the server using abort() call, but in this case the child processes that have been launched for the SST continues to run after completion of the server.\n\nThis makes it impossible to re-start the server before all timeouts expired (in these SST-related processes). Otherwise we failed due to the busy sockets or it leads to other fatal errors due to interference between new and old instances of the SST scripts.\n\nIn this patch, I added new checks to safely shutdown the server, for correct processing of a failed attempts to make a new connection and the SST, and for the destruction of all child processes when the server terminated abnormally by initiative of the Galera.\n\nTo do this, I needed to add a new callback to wsrep API, because the Galera intercepts the SIGABRT signal. Therefore we cannot intercept abnormal termination of the server process (after it calls abort() function from the Galera side) without expanding the Galera API by adding to it new callback.","distinct":true,"url":"https://api.github.com/repos/natsys/galera/commits/8fd4491323cf0bb5debf67b58173a63a49d90148"}]}
{"actor":{"display_login":"sysprg"}}