Backup/Restore tools performance comparison
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Grippa, Vinicius
Silva, Jean Da
Backup and restore methods are concepts that everyone knows the importance of. Over the years, open-source tools emerged like MyDumper, Xtrabackup, and Mariabackup. Also, with MySQL 8 new shell, new utils for dump and restore were introduced as well. In this presentation, we are going to compare the newest backup/restore methods with the most used ones. We will see how parallelization can influence the speed of backup and restore process and also how the compression algorithms can influence the performance. In this talk, we will compare mysqldump, mydumper/myloader, mysqlpump, MySQL Shell utils, and Xtrabackup.
 Hi everyone, thanks a lot for being here again at Fosden. It's a great pleasure to be present even online on a second time. And today I'm going to talk, me and my colleague, Jean, we are going to talk about backup and restore tools and do a comparison of performance among them. Doing our tests we found some really interesting results and I think it was a nice stop to revisit now in 2022 because for example my SQL shell introduced new tools to do backup and restore and I think it would be nice to do a comparison with what we had in the past like my SQL dump until nowadays. So first for those who doesn't know me, I'm Vinny, Vinicius, feel free to call me any way you want. I've been working in support team at Percona for almost five years. I work specifically with my SQL and MongoDB. Recently me and my colleagues again, we released a book about my SQL, learning my SQL for those who are starting your career on this MySQL open source word. And Jean, would you like to introduce yourself? Thank you. Hello everyone. My name is Jean, I'm from Brazil as well. I've joined Percona in 2021 in the middle of the pandemic. So I'm joined the team since then and helping Percona with my SQL and MongoDB tasks. And here today I hope you guys like this, our presentation. Let's move on. Thanks Jean. So for us to start, let's begin with what tools do we have available to perform the backup and restore procedure. The tools that we chose are MySQL dump, which is the class key one. It comes with all MySQL binaries. It is relatively well known by everyone, I bet that everyone that is watching this, so used MySQL dump at least once in their life. And the other tools are MyLoader and MyDumper. It's part of the same process. First you use MyDumper and then MyLoader. MySQL pump that is present on the MySQL binaries, on community version, Percona as well. Extra backup, which is a physical backup. It's open source tool provided by Percona. And MySQL shell. MySQL shell is present since MySQL 5.7 and 8 and introduced the new tools to execute these tasks. One clarification, I mentioned physical backups. So quickly remembering, reminding this, there are two types of backups. Physical backups and logical backups. Physical backups, they literally copy the files that belong to the database. It's like how extra backup works, copying the files. If you shut down MySQL and do a Cp, scp or something like this, it's a code backup as well. It works as a physical backup. Although as we can see, you can't copy the files while MySQL is running, because then you will lose that and tag it. For this we have extra backup. Then logical backup, you don't take copy of the files. Basically the tool will connect to the database and perform SQL operations to extract the data and save it into one or multiple files. So we won't have any specific files from the database like the redo logs. A few things that we need to consider when taking a backup. These are based on our experience. I wrote a blog post about backup performance, which I got a few comments. And I think it's worth to remember this. So many things around network encryption, file encryption. If we are going to use a single thread or a multi-threaded tool, if we are going to compress this backup, because disk is expensive, especially for people that are on the cloud, which each gigabyte has a cost. Backup time and restore time, those are important for not only the DBAs, but for managers to measure how long the operation can support to be offline because of disaster recovery situation. The total time, of course, we will go over the next sections. We will see that a lot of the tools perform the backup relatively fast, but the restore suffers a bit. So yeah, we'll talk about this. Of course, we talk about total time to backup and restore, point-in-time recovery, which is a very important feature for most of the companies because they can't afford losing more than 30 minutes of data, one hour, whatever, and performing full backups is an expensive operation. So usually people decide to do one full during each week and then incremental backups. And a few others that I couldn't think of, but I'm sure you guys might have any other items. So let's begin with the numbers that people are most interested in. And let's start with the backup benchmark, which is what I like to call the first side of the history because it's where everything is born, where we literally start. So what I used on this benchmark, I used the IWS instance, M5DN 8X large with 128GB of memory, two Iowan disks with 5000 IOPS fixed IOPS because I didn't want those burstable disks interfering into the results. So I provisioned a fixed storage with IOPS, sent to 7.9, and as you can see, the versions of the tools. MySQL 8026, same for MySQL Shell. This at the time of this recording, my damper, both GZIP and ZSTD, they are on 11.5, 011.5 version, and extra backup 8026. We can notice extra backup is not working anymore, like if you have extra backup 8026 and MySQL 8027, extra backup won't work. That was a decision that Percona made recently. And how my data is composed. So we have 96GB in disk of data distributed into 90 tables with different sizes, many tables going from 3GB to a few megabytes only. And I created this using the TPCC script. It's available here and providing it, it's open source. And the idea is, of course, to have more data than my currently buffer pool, because that's what we observe for most customers. The buffer pool contains only the hot data, while the database is much larger than what we have configured. So let's start with the results. This is the backup size. As I mentioned here, the database is 96GB. So extra backup with no compression. It's a bit easier to understand why, because it's a copy of the files. It says this, except binary logs. So if you have 100GB of binary logs, this is not going to be copied. And if we follow, we can see extra backup using compression. My SQL pump with no compression and LZ4, my damper, my damper without compression. My SQL shell uses the STD by default. So it's already compressed. And my SQL dump not compressed, not compressed as is. This I tried to make it a bit different. This is the ratio, which means that the extra backup is equivalent to the database size. And for example, my SQL shell, as we can see, it's 90% of the total size of the database. So you can theorize if you have a database with 100GB, my SQL shell usually uses 90%. And you are going to have around 9 to 10GB of backup. So then you can explode to one, explode to one, that abides to whatever and gets some similar metrics. Now going to the runtime, this is time to execute. Less is better. So smaller the bars, more faster the backup run. As we can see, my SQL dump, the only single thread to running is huge. We can see the other ones running almost at the same time. My SQL shell, we can see my SQL shell and my damper with the STD are the tools that are running, are the fastest tools to perform the backup. And because the my SQL dump took so long, the graph is not so clear. So I took it out so you guys can have a better idea when we are talking about parallel tools. And well, the numbers are the same, this didn't change, but I think this shows with more clarity, like for example that extra backup took more time. And each bar is for 16, 32 and 64 threads. Remember that the machine has 32 cores. So these are the results. I got some questions and comments on the blog post that I wrote. Extra backup currently does not support the STD. You need to do a pipe on the backup and then compress the files manually. However, we opened a feature request. So hopefully we will soon see support to the STD in extra backup too. In terms of performance, during the test, I didn't observe any impact in performance in using socket or TCP IP protocol, both running to the same speed with one or two seconds of difference, which for me, this is the margin of error. And another question that I got, for example, my SQL shell uses SSL by default to connect to the database. Why am I done, per or not? So I got a question like how meaningful is the impact of SSL? And I run the tests, as we can see, SSL took four seconds more than no SSL, but I don't consider this as a big difference. I would say, especially when we see the total time, this is not a meaningful impact to the overall process. And now my colleague, Gian, will talk about the restore benchmark. Okay. Thank you, Vinny, and hello, everyone, again. So let's keep the pace and speaking about the restore results. Again, as you might know, a good backup policy also includes a restore routine. So it's interesting for you to have these numbers to evaluate which tool to use when planning a backup routine. Okay. Next, Vinny. So just a review on the restore machine. It had the same profile as the backup one. It's a 32 CPU machine with two NVMe disk with fixed throughput. And the tools that we used had the last version running. Okay. So let's go next. Okay. Let's talk about the restore itself. So as you can see, the numbers are in second. And MySQL dump, unfortunately, it gets most of the time from the others we've compared because it's not fair to say about that, but it's a single-threaded tool. So it's this kind of results, it's expected from our side. But the fun part is when we compare the MySQL shell and the MyLoader results, which is nice the numbers they got. So in the next slide, we are going to see it closer without the noisy of MySQL dump. Okay. So here we have the numbers and it's quite interesting because there are a few points which we would like to show for you guys. So the first one is the battle between MySQL shell and MyLoader is quite tough in terms of numbers. They are very, very close. So any of those tools that you pick to use, you'll be more personal choice. So if you look the numbers, let me see. We have around, let's see, we have two minutes. We have, yeah, we have seven minutes for MySQL shell and a little bit more for the MySQL Loader. So but the point here is also worth mentioning. It's about the MySQL bump. It shined when we did the backup because it provides the parallelism option. But then when we did the restore, the numbers goes a little bit high if you compare to the others. So that's because for the restore process, the MySQL bump, it's a single-threaded operation. So that's why, unfortunately, the numbers skyrocketed a little. And the last, but not the least, we have the extra backup running with impressive numbers. So as you can see, it took two minutes to finish the restore with no compressed data and seven minutes with compressed data sets. So it's a very good result for extra backup. So next, please. Okay. So the restore itself shows a little good numbers. So in the next slide, we'll have the total of the numbers counting backup and restore and that will help you to show, to move the decision. Next slide, please. Okay. Here, we have the backup and restore numbers summit. So as you can see, the MySQL dump, it got the highest number. So it took 27 hours to finish the process. So that's a lot of time to run a backup and restore process. But then we also have the information from MySQL shell and the other tools. On the next slide, you'll see it without the MySQL dump information. So great. So here, we can see that, again, the battle between MySQL shell and MySQL loader keeps happening. So the numbers are quite good in terms of how long did it take to finish the entire process. So again, both of those would be a good choice to use in a backup policy. But then when you check the MySQL, the MySQL pump, do it to the restore process. It unfortunately got a very, very high backup and restore process. Again, mainly because do it to its lack of feature in terms of parallelism for restoration process. And then we also have the extra backup, which again, from the other tools, if you compare, as Vinny said before, it's a physical backup. So the things are a bit different in terms of how it takes the backup and how it restores. And then it also provides a very good result against the others. So next, please. So what we can observe after all of this. As we saw, we can't look at only one side, only the backups or only the restore. There is always the two sides of the history plus your business budget, like how much space in GSC we will use. So you need to use compression, security compliance, you need to use encryption or network encryption or rest. So those are all the things. We can observe like extra backup seems to offer a good balance between backup and restore time. The two, of course, is not the fastest when doing the backup. We saw that my damper and my SQL shell are much faster, but overall, it has a good advantage, especially when using compression. Because as we can see here, the difference between a compressed extra backup and no compressed, it's only a few seconds of difference. My SQL pump, as Jean said, it is a great tool to take the backup, but it suffers because it sends everything to a single file and you need to import through MySQL. So hopefully in the future, I would like to see MySQL pumping splitting the data into different files, like my damper does. And then it should be like a very good tool to use it. Another thing is that, as I mentioned, we don't observe significant impact when using compression or SSL or TCP IP versus socket. Like when we put the entire process, there is not a big difference. And if you consider the amount of disks that you are saving, it might be a very good option for you. And lastly, like it's clear to see that parallelism, especially in these nowadays, which both CPU machines are available, parallelism is a must. We can't wait more than one day to do a backup and restore using MySQL. So it's not feasible nowadays. It is possible to squeeze more juice from this. For example, I didn't touch any MySQL settings, like we can avoid writing to the binary log, we can disable it, we can set the R&DB flush logs TRX commit to 2 or 0. We can disable the double write buffer and many other things. However, like this needs to be calculated by you guys, calculated the risk because we are going to do a trade-off between performance and reliability. So if during the process, when you disable everything, your database crashes, you will probably and I really recommend to restart the process again. This is a blog post written by Ives Trudeau, an architect from Percona, and he gave some advice on how to improve MySQL writes. On a side note, the first line is be aware, this can corrupt your data, so don't test it in production, go for your key way environment. And now the choice of the presenter. Jian, which tool you would like to use if the database would be in your hands? So it depends on the situation, of course, but generally speaking, I would choose extra backup. So first thing, some considerations first. So it's nice to see MySQL shell and MySQL loader with great numbers. It's great to have options to use throughout the database. But even for small tests or for complex ones like create a backup routine, for me personally, extra backup also helps. So I would go to extra backup at this moment. Okay, thank you. Yeah, I must say, I agree with Jian. I really like mainly the PTR option of extra backup, which is very flexible and the other tools does not have it. But because I used logical backups for a while, I got used with my damper, MySQL shell. I like to use MySQL shell only for MySQL wait. I know that Oracle guys may not agree with me, but that's my personal choice. So if I would be doing a logical backup, certainly I would use MySQL shell or my damper or my loader, but never MySQL pump or MySQL dump. As we can see, the difference is huge. And as Jian said about extra backup, his comments, and I know we are exceeding a bit of the time. So guys, if you have any questions, please feel free to ask in the chat or you can reach out to us in the social media or per corner forums and we would be glad to help. Once more from my side, thank you to be here at Fosden again. It's a huge pleasure. And Jeff. Yeah, for me as well. It's my first time. So I hope all you have enjoyed this presentation. So if you have any questions, feel free to chat us. Okay. Bye-bye. Okay. Thank you for your answer. Yeah, thank you.
 Hi everyone, thanks a lot for being here again at Fosden.  It's a great pleasure to be present even online on a second time.  And today I'm going to talk, me and my colleague, Jean, we are going to talk about backup and  restore tools and do a comparison of performance among them.  Doing our tests we found some really interesting results and I think it was a nice stop to  revisit now in 2022 because for example my SQL shell introduced new tools to do backup  and restore and I think it would be nice to do a comparison with what we had in the past  like my SQL dump until nowadays.  So first for those who doesn't know me, I'm Vinny, Vinicius, feel free to call me any  way you want.  I've been working in support team at Percona for almost five years.  I work specifically with my SQL and MongoDB.  Recently me and my colleagues again, we released a book about my SQL, learning my SQL for those  who are starting your career on this MySQL open source word.  And Jean, would you like to introduce yourself?  Thank you.  Hello everyone.  My name is Jean, I'm from Brazil as well.  I've joined Percona in 2021 in the middle of the pandemic.  So I'm joined the team since then and helping Percona with my SQL and MongoDB tasks.  And here today I hope you guys like this, our presentation.  Let's move on.  Thanks Jean.  So for us to start, let's begin with what tools do we have available to perform the backup  and restore procedure.  The tools that we chose are MySQL dump, which is the class key one.  It comes with all MySQL binaries.  It is relatively well known by everyone, I bet that everyone that is watching this,  so used MySQL dump at least once in their life.  And the other tools are MyLoader and MyDumper.  It's part of the same process.  First you use MyDumper and then MyLoader.  MySQL pump that is present on the MySQL binaries, on community version, Percona as well.  Extra backup, which is a physical backup.  It's open source tool provided by Percona.  And MySQL shell.  MySQL shell is present since MySQL 5.7 and 8 and introduced the new tools to execute these  tasks.  One clarification, I mentioned physical backups.  So quickly remembering, reminding this, there are two types of backups.  Physical backups and logical backups.  Physical backups, they literally copy the files that belong to the database.  It's like how extra backup works, copying the files.  If you shut down MySQL and do a Cp, scp or something like this, it's a code backup as  well.  It works as a physical backup.  Although as we can see, you can't copy the files while MySQL is running, because then  you will lose that and tag it.  For this we have extra backup.  Then logical backup, you don't take copy of the files.  Basically the tool will connect to the database and perform SQL operations to extract the  data and save it into one or multiple files.  So we won't have any specific files from the database like the redo logs.  A few things that we need to consider when taking a backup.  These are based on our experience.  I wrote a blog post about backup performance, which I got a few comments.  And I think it's worth to remember this.  So many things around network encryption, file encryption.  If we are going to use a single thread or a multi-threaded tool, if we are going to  compress this backup, because disk is expensive, especially for people that are on the cloud,  which each gigabyte has a cost.  Backup time and restore time, those are important for not only the DBAs, but for managers to  measure how long the operation can support to be offline because of disaster recovery  situation.  The total time, of course, we will go over the next sections.  We will see that a lot of the tools perform the backup relatively fast, but the restore  suffers a bit.  So yeah, we'll talk about this.  Of course, we talk about total time to backup and restore, point-in-time recovery, which  is a very important feature for most of the companies because they can't afford losing  more than 30 minutes of data, one hour, whatever, and performing full backups is an expensive  operation.  So usually people decide to do one full during each week and then incremental backups.  And a few others that I couldn't think of, but I'm sure you guys might have any other  items.  So let's begin with the numbers that people are most interested in.  And let's start with the backup benchmark, which is what I like to call the first side  of the history because it's where everything is born, where we literally start.  So what I used on this benchmark, I used the IWS instance, M5DN 8X large with 128GB  of memory, two Iowan disks with 5000 IOPS fixed IOPS because I didn't want those burstable  disks interfering into the results.  So I provisioned a fixed storage with IOPS, sent to 7.9, and as you can see, the versions  of the tools.  MySQL 8026, same for MySQL Shell.  This at the time of this recording, my damper, both GZIP and ZSTD, they are on 11.5, 011.5  version, and extra backup 8026.  We can notice extra backup is not working anymore, like if you have extra backup 8026  and MySQL 8027, extra backup won't work.  That was a decision that Percona made recently.  And how my data is composed.  So we have 96GB in disk of data distributed into 90 tables with different sizes, many tables  going from 3GB to a few megabytes only.  And I created this using the TPCC script.  It's available here and providing it, it's open source.  And the idea is, of course, to have more data than my currently buffer pool, because that's  what we observe for most customers.  The buffer pool contains only the hot data, while the database is much larger than what  we have configured.  So let's start with the results.  This is the backup size.  As I mentioned here, the database is 96GB.  So extra backup with no compression.  It's a bit easier to understand why, because it's a copy of the files.  It says this, except binary logs.  So if you have 100GB of binary logs, this is not going to be copied.  And if we follow, we can see extra backup using compression.  My SQL pump with no compression and LZ4, my damper, my damper without compression.  My SQL shell uses the STD by default.  So it's already compressed.  And my SQL dump not compressed, not compressed as is.  This I tried to make it a bit different.  This is the ratio, which means that the extra backup is equivalent to the database size.  And for example, my SQL shell, as we can see, it's 90% of the total size of the database.  So you can theorize if you have a database with 100GB, my SQL shell usually uses 90%.  And you are going to have around 9 to 10GB of backup.  So then you can explode to one, explode to one, that abides to whatever and gets some  similar metrics.  Now going to the runtime, this is time to execute.  Less is better.  So smaller the bars, more faster the backup run.  As we can see, my SQL dump, the only single thread to running is huge.  We can see the other ones running almost at the same time.  My SQL shell, we can see my SQL shell and my damper with the STD are the tools that are  running, are the fastest tools to perform the backup.  And because the my SQL dump took so long, the graph is not so clear.  So I took it out so you guys can have a better idea when we are talking about parallel tools.  And well, the numbers are the same, this didn't change, but I think this shows with more clarity,  like for example that extra backup took more time.  And each bar is for 16, 32 and 64 threads.  Remember that the machine has 32 cores.  So these are the results.  I got some questions and comments on the blog post that I wrote.  Extra backup currently does not support the STD.  You need to do a pipe on the backup and then compress the files manually.  However, we opened a feature request.  So hopefully we will soon see support to the STD in extra backup too.  In terms of performance, during the test, I didn't observe any impact in performance  in using socket or TCP IP protocol, both running to the same speed with one or two seconds  of difference, which for me, this is the margin of error.  And another question that I got, for example, my SQL shell uses SSL by default to connect  to the database.  Why am I done, per or not?  So I got a question like how meaningful is the impact of SSL?  And I run the tests, as we can see, SSL took four seconds more than no SSL, but I don't  consider this as a big difference.  I would say, especially when we see the total time, this is not a meaningful impact to the  overall process.  And now my colleague, Gian, will talk about the restore benchmark.  Okay.  Thank you, Vinny, and hello, everyone, again.  So let's keep the pace and speaking about the restore results.  Again, as you might know, a good backup policy also includes a restore routine.  So it's interesting for you to have these numbers to evaluate which tool to use when  planning a backup routine.  Okay.  Next, Vinny.  So just a review on the restore machine.  It had the same profile as the backup one.  It's a 32 CPU machine with two NVMe disk with fixed throughput.  And the tools that we used had the last version running.  Okay.  So let's go next.  Okay.  Let's talk about the restore itself.  So as you can see, the numbers are in second.  And MySQL dump, unfortunately, it gets most of the time from the others we've compared  because it's not fair to say about that, but it's a single-threaded tool.  So it's this kind of results, it's expected from our side.  But the fun part is when we compare the MySQL shell and the MyLoader results, which is nice  the numbers they got.  So in the next slide, we are going to see it closer without the noisy of MySQL dump.  Okay.  So here we have the numbers and it's quite interesting because there are a few points  which we would like to show for you guys.  So the first one is the battle between MySQL shell and MyLoader is quite tough in terms  of numbers.  They are very, very close.  So any of those tools that you pick to use, you'll be more personal choice.  So if you look the numbers, let me see.  We have around, let's see, we have two minutes.  We have, yeah, we have seven minutes for MySQL shell and a little bit more for the MySQL  Loader.  So but the point here is also worth mentioning.  It's about the MySQL bump.  It shined when we did the backup because it provides the parallelism option.  But then when we did the restore, the numbers goes a little bit high if you compare to the  others.  So that's because for the restore process, the MySQL bump, it's a single-threaded operation.  So that's why, unfortunately, the numbers skyrocketed a little.  And the last, but not the least, we have the extra backup running with impressive numbers.  So as you can see, it took two minutes to finish the restore with no compressed data  and seven minutes with compressed data sets.  So it's a very good result for extra backup.  So next, please.  Okay.  So the restore itself shows a little good numbers.  So in the next slide, we'll have the total of the numbers counting backup and restore  and that will help you to show, to move the decision.  Next slide, please.  Okay.  Here, we have the backup and restore numbers summit.  So as you can see, the MySQL dump, it got the highest number.  So it took 27 hours to finish the process.  So that's a lot of time to run a backup and restore process.  But then we also have the information from MySQL shell and the other tools.  On the next slide, you'll see it without the MySQL dump information.  So great.  So here, we can see that, again, the battle between MySQL shell and MySQL loader keeps  happening.  So the numbers are quite good in terms of how long did it take to finish the entire  process.  So again, both of those would be a good choice to use in a backup policy.  But then when you check the MySQL, the MySQL pump, do it to the restore process.  It unfortunately got a very, very high backup and restore process.  Again, mainly because do it to its lack of feature in terms of parallelism for restoration  process.  And then we also have the extra backup, which again, from the other tools, if you compare,  as Vinny said before, it's a physical backup.  So the things are a bit different in terms of how it takes the backup and how it restores.  And then it also provides a very good result against the others.  So next, please.  So what we can observe after all of this.  As we saw, we can't look at only one side, only the backups or only the restore.  There is always the two sides of the history plus your business budget, like how much space  in GSC we will use.  So you need to use compression, security compliance, you need to use encryption or network encryption  or rest.  So those are all the things.  We can observe like extra backup seems to offer a good balance between backup and restore  time.  The two, of course, is not the fastest when doing the backup.  We saw that my damper and my SQL shell are much faster, but overall, it has a good advantage,  especially when using compression.  Because as we can see here, the difference between a compressed extra backup and no compressed,  it's only a few seconds of difference.  My SQL pump, as Jean said, it is a great tool to take the backup, but it suffers because  it sends everything to a single file and you need to import through MySQL.  So hopefully in the future, I would like to see MySQL pumping splitting the data into  different files, like my damper does.  And then it should be like a very good tool to use it.  Another thing is that, as I mentioned, we don't observe significant impact when using  compression or SSL or TCP IP versus socket.  Like when we put the entire process, there is not a big difference.  And if you consider the amount of disks that you are saving, it might be a very good option  for you.  And lastly, like it's clear to see that parallelism, especially in these nowadays, which both  CPU machines are available, parallelism is a must.  We can't wait more than one day to do a backup and restore using MySQL.  So it's not feasible nowadays.  It is possible to squeeze more juice from this.  For example, I didn't touch any MySQL settings, like we can avoid writing to the binary log,  we can disable it, we can set the R&DB flush logs TRX commit to 2 or 0.  We can disable the double write buffer and many other things.  However, like this needs to be calculated by you guys, calculated the risk because we  are going to do a trade-off between performance and reliability.  So if during the process, when you disable everything, your database crashes, you will  probably and I really recommend to restart the process again.  This is a blog post written by Ives Trudeau, an architect from Percona, and he gave some  advice on how to improve MySQL writes.  On a side note, the first line is be aware, this can corrupt your data, so don't test  it in production, go for your key way environment.  And now the choice of the presenter.  Jian, which tool you would like to use if the database would be in your hands?  So it depends on the situation, of course, but generally speaking, I would choose extra  backup.  So first thing, some considerations first.  So it's nice to see MySQL shell and MySQL loader with great numbers.  It's great to have options to use throughout the database.  But even for small tests or for complex ones like create a backup routine, for me personally,  extra backup also helps.  So I would go to extra backup at this moment.  Okay, thank you.  Yeah, I must say, I agree with Jian.  I really like mainly the PTR option of extra backup, which is very flexible and the other  tools does not have it.  But because I used logical backups for a while, I got used with my damper, MySQL shell.  I like to use MySQL shell only for MySQL wait.  I know that Oracle guys may not agree with me, but that's my personal choice.  So if I would be doing a logical backup, certainly I would use MySQL shell or my damper  or my loader, but never MySQL pump or MySQL dump.  As we can see, the difference is huge.  And as Jian said about extra backup, his comments, and I know we are exceeding a bit of the  time.  So guys, if you have any questions, please feel free to ask in the chat or you can reach  out to us in the social media or per corner forums and we would be glad to help.  Once more from my side, thank you to be here at Fosden again.  It's a huge pleasure.  And Jeff.  Yeah, for me as well.  It's my first time.  So I hope all you have enjoyed this presentation.  So if you have any questions, feel free to chat us.  Okay.  Bye-bye.  Okay.  Thank you for your answer.  Yeah, thank you.
mysql_backup_restore_0_155.png mysql_backup_restore_156_1643.png mysql_backup_restore_1644_2291.png mysql_backup_restore_2292_2567.png mysql_backup_restore_2568_2687.png mysql_backup_restore_2688_2735.png mysql_backup_restore_2736_2819.png mysql_backup_restore_2820_3479.png mysql_backup_restore_3480_3611.png mysql_backup_restore_3612_3719.png mysql_backup_restore_3720_3851.png mysql_backup_restore_3852_6359.png mysql_backup_restore_6360_7307.png mysql_backup_restore_7308_10343.png mysql_backup_restore_10344_10523.png mysql_backup_restore_10524_10931.png mysql_backup_restore_10932_14507.png mysql_backup_restore_14508_14555.png mysql_backup_restore_14556_16931.png mysql_backup_restore_16932_17939.png mysql_backup_restore_17940_18731.png mysql_backup_restore_18732_20915.png mysql_backup_restore_20916_21035.png mysql_backup_restore_21036_21095.png mysql_backup_restore_21096_21167.png mysql_backup_restore_21168_21779.png mysql_backup_restore_21780_22499.png mysql_backup_restore_22500_23579.png mysql_backup_restore_23580_27275.png mysql_backup_restore_27276_28187.png mysql_backup_restore_28188_30539.png mysql_backup_restore_30540_32183.png mysql_backup_restore_32184_32447.png mysql_backup_restore_32448_34715.png mysql_backup_restore_34716_36623.png mysql_backup_restore_36624_36803.png mysql_backup_restore_36804_36851.png mysql_backup_restore_36852_36899.png mysql_backup_restore_36900_38051.png mysql_backup_restore_38052_40043.png mysql_backup_restore_40044_40643.png mysql_backup_restore_40644_40762.png