Open Source Database Infrastructure with Vitess
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Noach, Shlomi
This session reveals four experimental Vitess developments that automate away complex database operations. With these developments Vitess is able to run its own database infrastructure, transparently to the user, and take control of risky and elaborate situations and operations. We will briefly explain the Vitess architecture and how it supports said control, and discuss the following developments: - Throttling: pushback for massive writes. - Table life cycle: safe and lazy DROP TABLE operations. - Online DDL: automating, scheduling and managing online schema migrations. - HA, failovers and cluster healing via vitess/orchestrator (aka vtorc). Vitess is a CNCF open source database clustering system for horizontal scaling of MySQL.
 Hello and thank you for tuning in. My name is Shlominoch and this is open source database infrastructure with Vitesse. I'm an engineer at PlanetScale. I'm an author of a bunch of open source projects in the MyScale ecosystem including orchestrator, ghost, reno and others which are relevant to today's discussion on a maintainer for Vitesse. PlanetScale was co-founded by the co-authors of Vitesse and we build interesting things on top of Vitesse. But we don't own Vitesse. Vitesse is a CNCF-graduate project released under the Apache 2 license and has maintainers and contributors from around the community. PlanetScale just happens to be a major contributor to Vitesse. Vitesse is a database-clustering system for horizontal scaling of MyScale but today we're going to present a different aspect of Vitesse, a database infrastructure aspect. On our agenda are four recent developments. They're either in progress or experimental. They're in the paintbrush. You can use them but they're not officially stable at this time. To understand these we're going to need to understand the general architecture of Vitesse so as to realize how Vitesse is able to provide automation of such complex operations. Today we're going to discuss throttling, table lifecycle, online video and high availability in failures. To illustrate the Vitesse architecture basics let's begin with looking at a simple MyScale replication cluster. We have one primary, three reticas. The first thing we're going to do is to attach a VTTablet to each of the database servers. This is a demon typically co-located on the database server which can then both communicate with the demon. It can send queries and receive responses but also it can control the MyScale service so it can stop and start the service. For example to run a backup and a restore etc. In production you will have many clusters and many servers. Again you will have a one-to-one mapping between a tablet and a server. Now on top of everything we're going to put a smart proxy it's called VitiGate. VitiGate impersonates as one giant monolithic database. It speaks the MyScale protocol, it parses your queries, it knows how to route them to the backend servers and to you it looks like a MyScale server. Now in reality we'll have many VitiGates to scale out the read and write capacity but for our purposes we'll only discuss one. And the big question is how does VitiGate route queries, how does it know where to write a query. You will have different schemas, different tables, different datasets on the various MyScale servers. Some of your classes could be shotted, some unshotted. How does VitiGate know where exactly to route the query. It begins with just the query itself, right? The application will use a specific schema and select something that indicates to VitiGate where to get the data from. So if we select order ID and price from orders where customer ID equals 4, the value 4 may indicate to VitiGate that in the commerce schema the value 4 is only located in shard 0. So based on schema, based on information VitiGate could be able to know where to route the queries to. But again how does it know how do all VitiGates know that? Well we have a backend database that's the state of Vites itself. We call it Topo, essentially it's an etcd or zookeeper or console, a distributed KV store, which stores all the meta information about the Vites cluster like what schemas we have, what shards we have, who are the tablets involved, etc. So VitiGate reads that information from Topo, it's mostly static information and it's mostly cached within VitiGate. The last component I want to introduce is VitiControlDemon, which is a demon which runs ad hoc operations like reshotting, it also acts as an API server and we'll follow up later on to see how it helps us. The first feature I want to present is throttling, pushback for massive rights to protect your database cluster. It's based on Fresno, a service I call for at GitHub, which is a cooperative throttling service. It's now imported into Vites and specifically into VitiTablet. The idea is to protect against replication lag. So the moment your replication lag grows higher, you begin to push back. We want to keep replication lag low because we want our replicas to serve fresh data, we want our failovers to be happy and we use a mechanism similar to PD-HardBit where Vites injects a timestamp onto the primary MySQL server and we read the value on the replicas and we calculate replication lag on sub-second resolution. The fun thing about Vites and throttling is that Vites is aware of the topology and of the identity of the service, not all servers are created equal. We have, of course, the primary and we have replicas, but some replicas are serving production traffic whereas other replicas could be like backup servers or OLAP servers or be out for maintenance momentarily. The fun thing is that Vites knows exactly at any given time which service serves data or not. The fact of it is controls the streaming of production traffic through to those servers with Vitegate and so Vites is able to say, okay, I'm going to consult the reading replicas for replication lag but not other types and this is overreadable with user command line flag. So the primary tablet of each shard runs a continuous pull for replication lag on the relevant replicas and it serves an HTTP endpoint and the users can call on that endpoint. It returns with 200k when everything is good, meaning there is no lag or small lag on the relevant replicas or other HTTP code when there is lag. It also periodically consults with the topology service to see if there's any changes to the topology. Like, is there a new server, a new replica or maybe a replica which used to serve traffic is now down for maintenance or change role into something else and so Vites is able to say at any given time whether, you know, throttling makes sense right now or not. And the front-end service is basically internal. We don't expect to expose it to the users while it's actually visible but it's really used for some internal mechanisms. The two that I will describe today are table life cycle and all MDDL but we have more ideas for the future like automatically throttling massive updates or massive deletes. What happens if you update the entire row set of a table or try to delete a gazillion rows? The next development is table life cycle and automatic garbage collector for all tables. And this comes to solve the woes and misferchance of dropping tables in production. There's two major problems to dropping tables in production. One is that you may have missed something and some part of the application still needs the table and once you've dropped it, you're in a big problem. The other thing is that actually dropping a table on a busy production system can bring your can lock down your your primary database. That's something we've seen many times in production. And so people have come around to similar solutions in different configurations like first of all, how about you don't drop the table but we may need to something else. Application-wise, the table is gone but in case you forgot something, you can always rename it back and make application happy. Next is that dropping a full table with the gazillion rows is risky. So how about purging it first until it's completely empty? This can take a few days and actually in that process, you bring most of that table back into the even the buffer pool memory because your your access in the data in that table. So you're going to need a few more days to get those pages evacuated from the even the buffer pool. And then once the table is empty and it's being evacuated from the buffer pool, you're able to drop it. And there's more variations on this process like handling things different on the replicas like black hole in it, black hole in them or just truncating because the workload on the primary on the replicas is really very different. So this is a very difficult process to track, right? You need to know which state each table is and what's been done, what's not been done, you need to coordinate and schedule stuff. It's not trivial and Vitesse wishes to solve that. So Vitesse defines the following states in the table's lifecycle. Table can be in use, which is just normal, or it can be held, which means it's kept safe intact, like no data is impacted, and you still have time to regret and rename it back into the original name. Or it can be in the purge state where it's actively being purged of rows. And then the evac state where you just wait out for the pages to evacuate out of the buffer pool. Finally, the drop state where the table is imminently going to be dropped, and then the table is gone. And the idea is that all this is based on the table name. So there's a magical table names. Take a look at the first table name here, Vitti hold something, something, something, then some number. If you look closely, that number is actually a timestamp. So a table with this name Vitti hold something, something, means Vitesse is not going to attach this table until 2021, January 31, 9 30am. And then when that time passes, the table gets from the transitions, transitioned into the next phase. Or if the table is Vitti evac, something, something, that means the table remains in evac mode. We wait out for the pages to evacuate out of the buffer pool until February 7, 2021, 7.15am. So what about the purging state? Well, the purging state is more complex because we're actually doing something here. So for purging tables, what we do is that Vitti tablet on the primary server is charged with purging the table data, right? And it only schedules a single table at the time. And it repeatedly purges like 50 rows at the time. And it uses the tablet throttler. So we've introduced tablet throttler earlier. This is the first use case between each such purge of 50 rows, the table garbage collector consults with the tablet throttler to make sure that it's not pushing the cluster health too hard. And so with this idea of encoding the states within the table names, the process becomes stateless. There's no metadata table where we need to track the progress of each of its tables, life cycle. It's enough to take one look at the tables in your schema to know what we need to do next. Vitti has to always score with the relevant tables and will always do the right thing and converge into dropping those tables. Not all users are the same. Some users are perfectly happy with just dropping the tables, which is fine. Others say, well, I just want to hold them for a few days and then drop them. And others, like in my experience, like we have to go there, the fool, the full flow, right? We need to hold and purge and evac and drop and then some. And so that's the default and it's controlled by a command line flag. Online DDL, big development, which incorporates both previous developments discussed. So skin and changes made easy. We all know the trouble you want to alter table production. It's a big problem. It's either blocking and locking on the primary or on the replica. And it's exhausting. It's uninterruptible. And it's an old go. Most people would use being on a scheme change or ghost to run scheme changes, which is great. These are external tools, but they create an operational complexity because you need to install them somewhere. You need to ssd somewhere to run them. You need to tell them where the what the topology is. So there's this discovery issue. What's the primary water directly because you need to create an account on your nice build service. You want to schedule the migrations because you typically don't want to run two or more concurrent migrations. You need to be able to throttle somehow. You need to be able to track the migration interrupted if needed, etc. And that that takes the ownership outside of the developer's hands. And so in small companies, developers will be able to just run this in production. But as your company scales, this doesn't that's not sustainable. So you need specialized people to do that. And that's that's not so good. And Vitesse is in the unique position to solve really all of these problems because it knows about the topology. It can throttle. It can help you track in discoveries trivial. It can run external tools on your behalf. It's a run. And so Vitesse reduces the problem to this. All you need to do is set the deal strategy to ghost or PDO on a skill exchange. That's a new session variable followed up by your normal art table. The response is a bit funny with a standard direct art table. The statement can take like hours. And then you get no response. Whereas here, the query returned immediately with this weird uu ad. So that's a that's an asynchronous migration. And that's a job tracking ad, which we will use later on. So what goes on here is that the application requests scheme of change, but it doesn't talk directly to the database. Every talks to Viti gate. Now, Viti gate is under no obligation to actually pass this alter table statement onto the tablets or the back in my skillservice. Instead, it says, oh, okay, I was requested to run a ghost migration. That's the strategy. So I'm going to just store the request in topo. No Viti control the imperiality checks for those requests for such requests. And it sees our alter table request and it knows, okay, that's the commerce schema, the product table, it figures out which clusters belong to the commerce scheme, which are the shots, it picks the primary tablets for these shots, and sends them that request. And all the fun happens in the tablet, it receives the request will be controlled and persists it schedules the migration because there may already be migration running. It prepares the script and temporary directories. It creates temporary one off credentials on the MySQL server, especially for that migration, it then cleans them up afterwards. It runs ghost and PD on a scheme of change with the correct command line arguments, letting them know where the topology is. It uses the tablet thrower with ghost, it's pretty much built in with PD on a scheme of change, we override the replication land plugin to consult the throttler. It rocks the migration, it cleans up afterwards, like it will clean up the triggers for PD on a scheme change, whether the migration succeeds or fails, it will Viti tablet will ensure the triggers are removed. But how about the artifact tables like the old tables, you can't just drop them, right? So no problem, we just feed them into our garbage collection, we just rename the old tables into those magical table names to be collected by the table lifecycle garbage collection mechanism. And so we got this UU idea in response to our alter table statement, which we can use to then track the migrations like we can show the status for a given migration on these four shards, it's still running on three, it's completed on the fourth one. We can cancel ongoing migrations, we can retry, cancel or fail migrations, etc. But online DDL is more than just alter tables, we can only also use it for create and drop, create is pretty trivial. But how about drops? So this is where we integrate our table lifecycle, again, the second time in online DDL, when you drop a table in an online DDL strategy, VitiGate will transform your drop statement into a rename. So it will rename into something hold, which will, you know, put the table into a health state, you still have time to regret, and then eventually it will be picked up and purged and removed. So this puts back ownership into the hands of the developers. It's almost zero dependencies. If you're on Linux AMD64, Ghost comes pre-compiled within Vites, you don't really need to do anything to make that happen, you just need to invoke the alter statement, it will retry in case it will fail over, etc. There's a bunch of goodies for this. Future work is to integrate with ongoing reshalling. So think about immigration that keeps running while you reshot the database and while you actually repart, reparent, like promote new replicas as primaries, etc. Last development I want to discuss today is VitiWork, the orchestrated integration. So I began writing orchestrates seven years ago and throughout Outbrand and Booking.com and GitHub, knowing independently, and the idea was that when I joined PlanetSkill, I would assist in integrating Vites and orchestrate, and thankfully that did not happen. And Suguru, co-creator of Vites, took this upon himself and did the integration, and I'm ever so happy because if I were to do that, I would try to make Vites work nicely with the orchestrator, whereas Suguru made orchestrate work nicely with Vites, and that's all the difference in the world. And there's many aspects to that, but let me just describe one. One of the greatest problems with MySQL replication clusters is that MySQL is not aware of replication clusters. MySQL is only aware of a replica, reticating from a primary, but it doesn't have the concept that there's no entity called a replication cluster. In MySQL, there's no name to that, but to you and me, this means the world, right? That's our database cluster. So orchestrator's approach is mostly to observe and then act upon what it perceives to be the expected scenario. If it observes a cluster and the primary fails, then it can kick on a failover, but it still doesn't know per se that all these servers are necessarily part of the same cluster. It gives some metadata for name and et cetera, but it's still very heuristic. Just as an example, if you have a split-brain scenario and one big cluster splits into two clusters, which of these two is the real cluster? Is there a correct answer? So if orchestrator were the one to run the failover, then heuristically it says, okay, the one which I promoted is now the real cluster, but is that the correct behavior? This is just heuristic. And there's other more complex scenarios where the heuristics are not so clear. But Vitesno's. Vitesno's exactly what it wants, right? Everything is written and so forth. It keeps a state. It knows which cluster it has, which members in the cluster, what's the role of every single server in a cluster. It knows which server it wants as a primary, which is a replica. The old Vitesno orchestrator integration was based on APIs and hooks and scripts. And there was a lot of friction where both were trying to take ownership of the separation, which led to conflicts. And Vite Orr is a spin-off of orchestrator, where the orchestrator's code is actually embedded within Vites. It uses the Vites stop or it uses Vites functionality like locking clusters and locking operations, etc. And the mission of Vites is inherently different from that of orchestrator. Its mission, its goal oriented in its mission is to make the replication clusters to look like Vites expects them to. So a few examples to scenarios that Vite Orr handles that orchestrator doesn't. From left to right, there's a broken replica. Or a stop replication. Or it's just observed and said, well, there's a broken replica, but Vite Orr says, well, why is it broken? According to my records, it should be running, right? So let's kick it in. Or there's a detached replica. Orgustrator wise, it doesn't really know which cluster this detached replica belongs to, but Vites knows and Vite Orr knows and can rewire it back into place. If there's a dual primary, a co-primary scenario, it knows which of the two should be the master. Take a look at the right. There could be a fully functional cluster, right? An orchestrator would perceive it to be completely healthy. There's a primary, there's three replicas, everything seems to be working fine. But maybe Vites says, well, you know that first replica on top, that according to my records, that should be the primary, not the existing primary. So Vite Orr will actually actively promote that to be the new primary, because its goal is to make the cluster match Vites' definition. So that's super cool. There's a lot of work in progress going on with future customizable availability and durability rules, etc. So lots of things to look forward for. So with these developments, we turn Vites not into a shot in-frame work, but into a database infrastructure framework that brings in all the database complexity and solves it in one single place. And with that, I am able to take questions in chat. Thank you so much for tuning in.
 Hello and thank you for tuning in. My name is Shlominoch and this is open source database  infrastructure with Vitesse. I'm an engineer at PlanetScale. I'm an author of a bunch of  open source projects in the MyScale ecosystem including orchestrator, ghost, reno and others  which are relevant to today's discussion on a maintainer for Vitesse. PlanetScale was  co-founded by the co-authors of Vitesse and we build interesting things on top of Vitesse.  But we don't own Vitesse. Vitesse is a CNCF-graduate project released under the Apache 2 license  and has maintainers and contributors from around the community. PlanetScale just happens to be a  major contributor to Vitesse. Vitesse is a database-clustering system for horizontal scaling of MyScale  but today we're going to present a different aspect of Vitesse, a database infrastructure aspect.  On our agenda are four recent developments. They're either in progress or experimental.  They're in the paintbrush. You can use them but they're not officially stable at this time.  To understand these we're going to need to understand the general architecture of Vitesse so as to  realize how Vitesse is able to provide automation of such complex operations.  Today we're going to discuss throttling, table lifecycle, online video and high availability  in failures. To illustrate the Vitesse architecture basics let's begin with looking at a simple  MyScale replication cluster. We have one primary, three reticas. The first thing we're going to do  is to attach a VTTablet to each of the database servers. This is a demon typically co-located  on the database server which can then both communicate with the demon. It can send queries  and receive responses but also it can control the MyScale service so it can stop and start the  service. For example to run a backup and a restore etc. In production you will have many  clusters and many servers. Again you will have a one-to-one mapping between a tablet and a server.  Now on top of everything we're going to put a smart proxy it's called VitiGate. VitiGate  impersonates as one giant monolithic database. It speaks the MyScale protocol, it parses your  queries, it knows how to route them to the backend servers and to you it looks like a MyScale server.  Now in reality we'll have many VitiGates to scale out the read and write capacity but for our purposes  we'll only discuss one. And the big question is how does VitiGate route queries, how does it know  where to write a query. You will have different schemas, different tables, different datasets on  the various MyScale servers. Some of your classes could be shotted, some unshotted. How does VitiGate  know where exactly to route the query. It begins with just the query itself, right? The application  will use a specific schema and select something that indicates to VitiGate where to get the data  from. So if we select order ID and price from orders where customer ID equals 4, the value 4 may  indicate to VitiGate that in the commerce schema the value 4 is only located in shard 0. So based on  schema, based on information VitiGate could be able to know where to route the queries to. But  again how does it know how do all VitiGates know that? Well we have a backend database that's the  state of Vites itself. We call it Topo, essentially it's an etcd or zookeeper or console, a distributed  KV store, which stores all the meta information about the Vites cluster like what schemas we have,  what shards we have, who are the tablets involved, etc. So VitiGate reads that information from Topo,  it's mostly static information and it's mostly cached within VitiGate. The last component  I want to introduce is VitiControlDemon, which is a demon which runs ad hoc operations like  reshotting, it also acts as an API server and we'll follow up later on to see how it helps us.  The first feature I want to present is throttling, pushback for massive rights to protect your  database cluster. It's based on Fresno, a service I call for at GitHub, which is a cooperative  throttling service. It's now imported into Vites and specifically into VitiTablet. The idea is to  protect against replication lag. So the moment your replication lag grows higher, you begin to  push back. We want to keep replication lag low because we want our replicas to serve fresh data,  we want our failovers to be happy and we use a mechanism similar to PD-HardBit where Vites  injects a timestamp onto the primary MySQL server and we read the value on the replicas and we  calculate replication lag on sub-second resolution. The fun thing about Vites and throttling is that  Vites is aware of the topology and of the identity of the service, not all servers are created equal.  We have, of course, the primary and we have replicas, but some replicas are serving production  traffic whereas other replicas could be like backup servers or OLAP servers or be out for  maintenance momentarily. The fun thing is that Vites knows exactly at any given time which  service serves data or not. The fact of it is controls the streaming of production traffic  through to those servers with Vitegate and so Vites is able to say, okay, I'm going to consult  the reading replicas for replication lag but not other types and this is overreadable with  user command line flag. So the primary tablet of each shard runs a continuous pull for replication  lag on the relevant replicas and it serves an HTTP endpoint and the users can call on that  endpoint. It returns with 200k when everything is good, meaning there is no lag or small lag  on the relevant replicas or other HTTP code when there is lag. It also periodically consults with  the topology service to see if there's any changes to the topology. Like, is there a new server,  a new replica or maybe a replica which used to serve traffic is now down for maintenance or  change role into something else and so Vites is able to say at any given time whether, you know,  throttling makes sense right now or not. And the front-end service is basically internal. We don't  expect to expose it to the users while it's actually visible but it's really used for some  internal mechanisms. The two that I will describe today are table life cycle and all MDDL but we  have more ideas for the future like automatically throttling massive updates or massive deletes.  What happens if you update the entire row set of a table or try to delete a gazillion rows?  The next development is table life cycle and automatic garbage collector for all tables.  And this comes to solve the woes and misferchance of dropping tables in production. There's two  major problems to dropping tables in production. One is that you may have missed something and  some part of the application still needs the table and once you've dropped it, you're in a big  problem. The other thing is that actually dropping a table on a busy production system can bring  your can lock down your your primary database. That's something we've seen many times in production.  And so people have come around to similar solutions in different configurations like first of all,  how about you don't drop the table but we may need to something else. Application-wise,  the table is gone but in case you forgot something, you can always rename it back and make application  happy. Next is that dropping a full table with the gazillion rows is risky. So how about purging  it first until it's completely empty? This can take a few days and actually in that process,  you bring most of that table back into the even the buffer pool memory because your your access in  the data in that table. So you're going to need a few more days to get those pages evacuated from  the even the buffer pool. And then once the table is empty and it's being evacuated from the buffer  pool, you're able to drop it. And there's more variations on this process like handling things  different on the replicas like black hole in it, black hole in them or just truncating because the  workload on the primary on the replicas is really very different. So this is a very difficult  process to track, right? You need to know which state each table is and what's been done, what's  not been done, you need to coordinate and schedule stuff. It's not trivial and Vitesse wishes to  solve that. So Vitesse defines the following states in the table's lifecycle. Table can be in use,  which is just normal, or it can be held, which means it's kept safe intact, like no data is  impacted, and you still have time to regret and rename it back into the original name. Or it can  be in the purge state where it's actively being purged of rows. And then the evac state where you  just wait out for the pages to evacuate out of the buffer pool. Finally, the drop state where the  table is imminently going to be dropped, and then the table is gone. And the idea is that all this  is based on the table name. So there's a magical table names. Take a look at the first table name  here, Vitti hold something, something, something, then some number. If you look closely, that number  is actually a timestamp. So a table with this name Vitti hold something, something, means Vitesse  is not going to attach this table until 2021, January 31, 9 30am. And then when that time passes,  the table gets from the transitions, transitioned into the next phase. Or if the table is Vitti evac,  something, something, that means the table remains in evac mode. We wait out for the pages to evacuate  out of the buffer pool until February 7, 2021, 7.15am. So what about the purging state? Well,  the purging state is more complex because we're actually doing something here. So for purging  tables, what we do is that Vitti tablet on the primary server is charged with purging the table  data, right? And it only schedules a single table at the time. And it repeatedly purges like 50 rows  at the time. And it uses the tablet throttler. So we've introduced tablet throttler earlier. This  is the first use case between each such purge of 50 rows, the table garbage collector consults  with the tablet throttler to make sure that it's not pushing the cluster health too hard.  And so with this idea of encoding the states within the table names, the process becomes  stateless. There's no metadata table where we need to track the progress of each of its tables,  life cycle. It's enough to take one look at the tables in your schema to know what we need to do  next. Vitti has to always score with the relevant tables and will always do the right thing and  converge into dropping those tables. Not all users are the same. Some users are perfectly happy with  just dropping the tables, which is fine. Others say, well, I just want to hold them for a few days  and then drop them. And others, like in my experience, like we have to go there, the fool,  the full flow, right? We need to hold and purge and evac and drop and then some. And so that's  the default and it's controlled by a command line flag. Online DDL, big development, which incorporates  both previous developments discussed. So skin and changes made easy. We all know the trouble you  want to alter table production. It's a big problem. It's either blocking and locking on the primary  or on the replica. And it's exhausting. It's uninterruptible. And it's an old go. Most people  would use being on a scheme change or ghost to run scheme changes, which is great. These are  external tools, but they create an operational complexity because you need to install them  somewhere. You need to ssd somewhere to run them. You need to tell them where the what the topology  is. So there's this discovery issue. What's the primary water directly because you need to create  an account on your nice build service. You want to schedule the migrations because you typically  don't want to run two or more concurrent migrations. You need to be able to throttle  somehow. You need to be able to track the migration interrupted if needed, etc. And that that takes the  ownership outside of the developer's hands. And so in small companies, developers will be able to  just run this in production. But as your company scales, this doesn't that's not sustainable. So  you need specialized people to do that. And that's that's not so good. And Vitesse is in the unique  position to solve really all of these problems because it knows about the topology. It can  throttle. It can help you track in discoveries trivial. It can run external tools on your behalf.  It's a run. And so Vitesse reduces the problem to this. All you need to do is set the deal strategy  to ghost or PDO on a skill exchange. That's a new session variable followed up by your normal  art table. The response is a bit funny with a standard direct art table. The statement can take  like hours. And then you get no response. Whereas here, the query returned immediately with this  weird uu ad. So that's a that's an asynchronous migration. And that's a job tracking ad, which  we will use later on. So what goes on here is that the application requests scheme of change,  but it doesn't talk directly to the database. Every talks to Viti gate. Now, Viti gate is under  no obligation to actually pass this alter table statement onto the tablets or the back in my  skillservice. Instead, it says, oh, okay, I was requested to run a ghost migration. That's the  strategy. So I'm going to just store the request in topo. No Viti control the imperiality checks  for those requests for such requests. And it sees our alter table request and it knows, okay,  that's the commerce schema, the product table, it figures out which clusters belong to the commerce  scheme, which are the shots, it picks the primary tablets for these shots, and sends them that request.  And all the fun happens in the tablet, it receives the request will be controlled and  persists it schedules the migration because there may already be migration running. It prepares  the script and temporary directories. It creates temporary one off credentials on the MySQL server,  especially for that migration, it then cleans them up afterwards. It runs ghost and PD on a  scheme of change with the correct command line arguments, letting them know where the topology  is. It uses the tablet thrower with ghost, it's pretty much built in with PD on a scheme of change,  we override the replication land plugin to consult the throttler. It rocks the migration, it cleans  up afterwards, like it will clean up the triggers for PD on a scheme change, whether the migration  succeeds or fails, it will Viti tablet will ensure the triggers are removed. But how about the  artifact tables like the old tables, you can't just drop them, right? So no problem, we just feed  them into our garbage collection, we just rename the old tables into those magical table names to  be collected by the table lifecycle garbage collection mechanism. And so we got this UU idea  in response to our alter table statement, which we can use to then track the migrations like we  can show the status for a given migration on these four shards, it's still running on three,  it's completed on the fourth one. We can cancel ongoing migrations, we can retry,  cancel or fail migrations, etc. But online DDL is more than just alter tables, we can only also  use it for create and drop, create is pretty trivial. But how about drops? So this is where we  integrate our table lifecycle, again, the second time in online DDL, when you drop a table in an  online DDL strategy, VitiGate will transform your drop statement into a rename. So it will rename  into something hold, which will, you know, put the table into a health state, you still have time  to regret, and then eventually it will be picked up and purged and removed. So this puts back ownership  into the hands of the developers. It's almost zero dependencies. If you're on Linux AMD64,  Ghost comes pre-compiled within Vites, you don't really need to do anything to make that happen,  you just need to invoke the alter statement, it will retry in case it will fail over, etc.  There's a bunch of goodies for this. Future work is to integrate with ongoing reshalling. So think  about immigration that keeps running while you reshot the database and while you actually  repart, reparent, like promote new replicas as primaries, etc. Last development I want to  discuss today is VitiWork, the orchestrated integration. So I began writing orchestrates  seven years ago and throughout Outbrand and Booking.com and GitHub, knowing independently,  and the idea was that when I joined PlanetSkill, I would assist in integrating Vites and  orchestrate, and thankfully that did not happen. And Suguru, co-creator of Vites, took this upon  himself and did the integration, and I'm ever so happy because if I were to do that, I would try to  make Vites work nicely with the orchestrator, whereas Suguru made orchestrate work nicely with  Vites, and that's all the difference in the world. And there's many aspects to that, but let me just  describe one. One of the greatest problems with MySQL replication clusters is that MySQL is not  aware of replication clusters. MySQL is only aware of a replica, reticating from a primary,  but it doesn't have the concept that there's no entity called a replication cluster. In MySQL,  there's no name to that, but to you and me, this means the world, right? That's our database  cluster. So orchestrator's approach is mostly to observe and then act upon what it perceives to be  the expected scenario. If it observes a cluster and the primary fails, then it can kick on a  failover, but it still doesn't know per se that all these servers are necessarily part of the same  cluster. It gives some metadata for name and et cetera, but it's still very heuristic. Just as  an example, if you have a split-brain scenario and one big cluster splits into two clusters,  which of these two is the real cluster? Is there a correct answer? So if orchestrator were the one  to run the failover, then heuristically it says, okay, the one which I promoted is now the real  cluster, but is that the correct behavior? This is just heuristic. And there's other more complex  scenarios where the heuristics are not so clear. But Vitesno's. Vitesno's exactly what it wants,  right? Everything is written and so forth. It keeps a state. It knows which cluster it has,  which members in the cluster, what's the role of every single server in a cluster. It knows  which server it wants as a primary, which is a replica. The old Vitesno orchestrator integration  was based on APIs and hooks and scripts. And there was a lot of friction where both were  trying to take ownership of the separation, which led to conflicts. And Vite Orr is a spin-off of  orchestrator, where the orchestrator's code is actually embedded within Vites. It uses the Vites  stop or it uses Vites functionality like locking clusters and locking operations, etc. And the  mission of Vites is inherently different from that of orchestrator. Its mission, its goal oriented  in its mission is to make the replication clusters to look like Vites expects them to. So a few  examples to scenarios that Vite Orr handles that orchestrator doesn't. From left to right,  there's a broken replica. Or a stop replication. Or it's just observed and said, well, there's a  broken replica, but Vite Orr says, well, why is it broken? According to my records, it should be  running, right? So let's kick it in. Or there's a detached replica. Orgustrator wise, it doesn't  really know which cluster this detached replica belongs to, but Vites knows and Vite Orr knows  and can rewire it back into place. If there's a dual primary, a co-primary scenario, it knows  which of the two should be the master. Take a look at the right. There could be a fully functional  cluster, right? An orchestrator would perceive it to be completely healthy. There's a primary,  there's three replicas, everything seems to be working fine. But maybe Vites says, well,  you know that first replica on top, that according to my records, that should be the primary, not  the existing primary. So Vite Orr will actually actively promote that to be the new primary,  because its goal is to make the cluster match Vites' definition. So that's super cool. There's a lot  of work in progress going on with future customizable availability and durability rules, etc. So lots  of things to look forward for. So with these developments, we turn Vites not into a shot  in-frame work, but into a database infrastructure framework that brings in all the database complexity  and solves it in one single place. And with that, I am able to take questions in chat. Thank you so  much for tuning in.
vitess_0_179.png vitess_180_695.png vitess_696_911.png vitess_912_1595.png vitess_1596_2543.png vitess_2544_3719.png vitess_3720_6071.png vitess_6072_7535.png vitess_7536_10295.png vitess_10296_12479.png vitess_12480_13547.png vitess_13548_16571.png vitess_16572_18107.png vitess_18108_19967.png vitess_19968_20687.png vitess_20688_20867.png vitess_20868_23207.png vitess_23208_24071.png vitess_24072_24491.png vitess_24492_24731.png vitess_24732_25391.png vitess_25392_27323.png vitess_27324_27983.png vitess_27984_30095.png vitess_30096_32471.png vitess_32472_33227.png vitess_33228_33647.png vitess_33648_34055.png vitess_34056_34823.png vitess_34824_37019.png vitess_37020_37427.png vitess_37428_37895.png