A deep dive and comparison of Python drivers for Cassandra and Scylla
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Jacob, Alexys
This talk will explain the thread-per-core data architecture of Scylla and detail how we implemented "shard-awareness" in the Cassandra Python driver which allows to route queries down to the right CPU! Cassandra's and Scylla's architecture and topology rely on the usage of a consistent token ring to distribute their data evenly on the cluster. The cassandra-python driver is used widely to interact with those NoSQL databases. It implements connection pools and token awareness allowing the driver to route queries to the right node based on its knowledge of where the data is. But Scylla goes one step further as it also dedicates CPUs to a smaller portion of the data on each node (called shards). This means that **we can route CQL queries not only to the right node but to the right CPU**! This talk will give implementation details on how we have done it. Talk audience requirements: having a minimal knowledge of what Cassandra and a consistent hash ring are.
 Okay, so we're coming to our third session in this block and you've just seen messages from some of our sponsors, but some of our sponsors also do talks. And the next one I'd like to invite to us is Alexis from Numbuli. Alexis, hello. Hello, everyone. Can you hear me alright? Can you see me as well? This works perfectly and we can see you too. You are in Paris at the moment or in France? Yeah, in France, in the south of France actually, this time, but I usually live in Paris and work in Paris. Are you ready to do your presentation? Sure, I guess so. I will just start sharing my screen and prepare everything if you allow. Yes, I'll show. Okay. And let's see if that works. So hello, everyone. I'm really happy to be remotely as you understood with you today. In this talk, I will detail Cassandra and Cilla low level architecture and explain how it's used by Python Cassandra driver and show you how we extended it to write a Python driver for Cilla. There will be diagrams, emojis, Python code, and hopefully some amazing performance graphs as well. Let me introduce a bit about myself. So, how's Martin already told you and you can judge from my accent. I'm French, I'm Alexis, and I'm a CTO at number Lee weapons as well to be a proud sponsor for Europe items since 2014. And we are digital marketing experts and we help brands establish a relationship, a digital relationship with their customers. In the open source world, I gen two Linux developer, where I'm part of the cluster and container teams, which means that I spend some of my time working on packaging distributed databases such as MongoDB and Cilla, and all work on packaging cluster related tools and work with my friends on the Docker gen two Linux images. I'm also open source contributor and enthusiast. I've been contributing to MongoDB to Cilla to Apache Airflow, and I'm a Python software foundation contributing member which means that I spend a fair amount of my time working on or contributing to Python based open source projects. Before we start, I wanted to introduce you to a fact that you may not know already, but since my Europe item is using discord, I found interesting to share to you that discord is using Cilla as well. So if you are interested in understanding what they do, I invite you to check out this link here where marksmith introduces how a discord is using Cilla. It's very interesting. So check it out if you want. This is an advanced talk as advertised. So I will suppose that you are familiar with the basics of consistent hashing and Cassandra. If that's not the case, or if you simply want to know more about consistent hashing and even use it in your Python applications. I'm allowing myself to also introduce you to the talk I gave on this very subject in a previous Europe item edition, but fear not, I've still worked on not making this a big problem for this presentation. So you should be able to follow along and no problem, even if you don't know exactly the details of consistent hashing. Let's get started now. And let me introduce you to Cassandra and Cilla token reading architectures. We see all they have in common, but also what Cilla has done that makes it special and worth a dedicated Python driver. The first thing to know is that Cassandra or Cilla cluster is a cluster is a collection of nodes or instances that can be visualized as a ring. All the nodes in this ring should be homogeneous using a share nothing approach. This means that there's nothing special about the node in this topology one node one Cassandra node on the on the ring or any Cilla node on the ring has no special role or anything special about it. There is no primary or secondary or anything. They all do the same thing. The ring is called a token ring in which the positions of the nodes on the ring define token ranges and token range partitions. You can see that before if you go clockwise on the ring, the range that is preceding a node is the token range that all the partition that it is responsible for. Partition is just a subset of data that is stored on a node in SQL. The Cassandra query language a partition appears as a group of sorted rows and is the unit of access of query data. This data is usually replicated across nodes, thanks to a setting that is called the replication factor. The replication factor defines how data is replicated on notes. For example, a replication factor of two means that a given token or token range or partition will be stored on two nodes. This is the case here where partition one and two is stored on X. And you can see that partition two is also stored on Y while the partition one is stored on Z. If we were to lose node X, we could still read the data from partition one from the Z. This is how high availability is achieved and how Cassandra and Cilla favor availability and partition tolerance. They are called AP on the CAPTRM. This kind of token ring architecture is sensible to data distribution among the nodes. Queries should in theory be evenly distributed between nodes. We could get an imbalance of data and query load in the above scenario where we store data in three big partitions and each node holds one range of the previous one and one range from the next one. If one of those partitions were to grow larger than another one, we could then have an imbalance of queries and over and sort of overload on them. To counter this effect that we are calling a hot node or hot partition, we need to add more variance in partition two node allocation. And this is done using what is called virtual nodes. So instead of placing physical nodes on the ring, we will place many virtual instances of them called virtual nodes. A virtual node represents a contiguous range of tokens owned by a single node. So it's just a smaller slice of a partition. But it's more shuffled between nodes. A physical node may be assigned multiple and non contiguous. If you remember the preceding slide, it was contiguous. This time, the actual notes allow for non contiguous assignment of notes. The default is to split a node into 256 virtual nodes on the token ring. So this is true for Cassandra and this is true for Cilla as well. So if you look at how now the partitions are distributed among notes, you see that there is more variance into this, which will end up in distributing the query bear. This is it for Cassandra's data distribution, but Cilla goes one step further. On each Cilla node, tokens of a V node are further distributed among the CPU cores of the node that are called shards. This means that the data stored on a Cilla cluster is not only bound to a node, but can be traced down to one of its CPU cores. This is really interesting architecture and low level design. This is the feature that we will leverage on the Python site Cilla shadowware driver later on. And I will explain to you how. Now that we understand how data is stored and distributed on the cluster. Let's see how it's queried by clients. On the physical layer partition is the unit of data stored on a node and is identified by a partition key. You can relate a partition key to a primary key in the SQL world. A partition key is the primary means of looking up a set of rows that comprise a partition. And a partition key serves to identify the node in the cluster that stores a given partition as well as to distribute the data across nodes in the cluster. The partitioner or the partition hash function using the partition key will help us determine where the data is stored on the given node in the cluster. So you take the ID in this case, you see a colon ID, you will, this will be the partition key, you take the value, you apply a hash function on it, the partitioner hash function, which by default on Cilla is murmur hash tree. And this will give you a token. A token is like a number, which is just actually is a number that will be placed on the token ring. And from where it leads on token ring, you will find out which nodes is which node is responsible for this data. That's as simple as this. Okay, so let's recap now on Cassandra, the hash of the partition key gives you a token telling you which node has the data that you are looking for. We can see this architecture as a shard pair node architecture, because from the from a token, you get to a node so short per node on Cilla. The same hash of the partition key from the same partition key gives you the same token, but the same token not only is not only telling you which node has the data, but also which CPUs core in this node is responsible for handling it. So this is a short per core architecture. This is how it's called. So Cassandra is short per node while Cilla is short per core. Now, let's see how does a client driver query Cassandra or a Cilla cluster. Because now that we know this, we could guess and expect that client drivers uses this knowledge to find out and optimize their query plan. A knife client would go on like this when a client connects to a Cassandra or Cilla cluster, it opens a connection to every node of the cluster. When it wants to issue a query, a knife client would pick one of its connections randomly, let's say, and issue the query to the node. The node it issues the query to will be seen as the from the client's perspective, it will act as what he's called a coordinator because he's coordinating the query and he's taking the ephemeral responsibility of rooting the query internally in the cluster to the right nodes that are replicas for this data. That means that are responsible for the partition, the query belongs to and gathering this responses and then responding back to the client. But maybe this coordinator is not a replica for the query data. If it's not the case, if the coordinator way is not a replica for the query data, it has to issue the queries to all replicas itself. That means that you will add it will add an extra hop inside inside the cluster to get the responses. This is sub optimal, of course, as it consumes network and processing power on the coordinator node for something that the client could have guessed in the first place, right. Because since the partition a hash function is known, our client library can use it to predict data location on the cluster and optimize the query routing. This is what the Cassandra driver does using the token aware policy. How does it work token aware clients, apply the partition logic to select the right connection to the right node and make sure that its coordinator node is also a replica of the query data. This is cool. And this is very efficient as a result, we save network hopes, lower the cluster internal load and get reduced query latency, meaning, meaning faster queries. Let's see how the Cassandra driver does it for really internally. So the token aware policy from the point of view of the Python Cassandra driver, the partition key. It is seen as a routine key, because it will be used to route the query, right. So it is seen as a routine key which is used to determine which node are the replicas for the query. And to know how to allow our Python driver to know about the partition key of a query, the query itself must be prepared as a server side statement. So it looks on in Python Cassandra's and CLA's prepared statements that can you can see them a bit like stored procedures in the SQL world. You see, if you see statement equals session dot prepare, and then you, you express the query that you want. And when you have an argument or parameter, you just put an exclamation, an integration mark. And this is the recommended and most optimal way to query the data because when you have prepared your, your query, it is validated and it leaves on the server side so you don't have to pass it. And you just have to pass a reference to it, and then only pass the arguments and the arguments, one of them will be the mandatory routine key partition key then routine key. So statement plus routine key equals nodes. And that's very, very cool. Another thing to note, it is that just like prepared stored procedures, prepare statements are those sort of safest way, because they prevent query injection so please, in production at the bare minimal only use prepare statements when you issue queries to Cassandra or CLA clusters. And then the Python Cassandra driver defaults to the token aware to root the query and then it also defaults to a data center where round robin load balancing query routine policy it's, it's a big long, but what it means that is that it will load balance for you in a round robin fashion so one after the other after the other like this. So bear minimal load balancing algorithm there is, but it's still pretty efficient. So, don't worry if even if your cluster is not spread between multiple data center is still works it's just, it just happens to be the default so it's token aware plus data center where round robin. By doing so the query routine will not only hit the right node, holding a copy of the data that you seek. Remember it's called the replica replica, but also load balance the queries evenly between all its every class so one can think yeah this is awesome and optimal. I mean, from the Cassandra cluster point of view, it is, and we can do better than this. But not with a CLA one. Remember, CLA shards the data one way further down to node CPUs. So, having a token aware token awareness is cool. But if our client had shared awareness, it would be even cooler, because this means that a token aware client could be extended to become a shadow aware client to root his queries, not only to nodes, but right to their CPU cores. She is very interesting to do. Such drivers, they already exist as forks of the data stacks and Cassandra drivers, and it's true for the java one and the go one as well and they have been around since last year, actually. There was no shadow where driver for Python and it made it made me sad and pretty angry. So when I attended CLA submit last year in San Francisco, I did some lobbying and hard lobbying on the on and found some CLA developers that are that were willing to help in in, in, in, in, in making this happen for Python as well. So we promised each other to make a Python shadow where driver and good news is that we have obviously did. And I will now explain to you in details how it has been done. And what we found out by doing so. Very, very interesting as well. Let's start by checking out the expected structural differences between the Cassandra driver and the CLA driver fork. The first thing to see is that the token aware Cassandra driver has, when he connects for the first time to the cluster, it opens a control connection this control connection allows your customer driver to know about the cluster topology how many nodes they are, which are up which are the, what are the, the schemas, etc, etc. It needs to know all this so it opens a special connection for this and in this connection it refreshes from time to time. And then it will open one connection per node, because this is how the token aware policy will be applied to select the right of those connection based on the query. And then it will be done by the famous token calculation. And that's how it's done. All right. On the shadow where client perspective, we still need to know about the cluster topology, actually, but instead of you opening one connection per node we will be opening one connection per core per node. The token calculation will still be useful to select the right node from the token perspective, but then we will need to add a shard ID calculation because we need to go down to the shard or the CPU core to so to select the connection to the right core to root the queries. Let's transform this into a to do from the Python code perspective. First thing is, since we will be using the same kind of control connection, we don't, we just use is as is there's nothing to change here. We will need to change the connection class object, because now, when we are going to open a connection per core per node, we will need to be able to detect. If we are talking to a CLA cluster or to a Cassandra cluster. The CLA driver, we wanted to retain the maximum compatibility with Cassandra as well so you can use the CLA driver to discuss and query your Cassandra cluster as well. The host connection pool should will use those, those shadow where connections to open one connection to every every core of every note. The current calculation that selects the right node will be the same. We will just use the vanilla and already existing and efficient token aware policy. But then we will need to extend it and had in the cluster when you issue the query, we will need the cluster class to pass down the routine key to the to the connection pool. We will try the shard ID calculation and then implement the logic based on the shard ID of selecting the right connection to the right node to the right shard. Okay, sounds like a plan. Let's do this. Now, we'll get down into the code. Before we we go into this, I wanted to highlight and to introduce Israel Israel is a CLA developer from Israel. I know it's confusing. That's how it is. This is in the audience. That's makes me some kind of pressure as you can guess. I wanted to take this opportunity to thank him and giving him the credits for most of what I'm going to present now, especially the efforts he put into CI testing the CLA driver. So the first thing that that that that needed to be done is to have those shard information to the connections. And so this is how it's been done. So a connection now has a shard ID assigned to it and sharding information. This sharding information comes from the server responses when we issue a query. That's what is what is squared in red on the bottom. The logic of this looks like this. And it is what's interesting to note is that the Cassandra protocol allows only for connection message options being passed on the server response. This means that when the client initially connects to Cassandra or Silla, it has no way of passing any kind of information to the server. So we are dependent on the server response to know about shard shard information or whatever it is that we need. If we look at the, the, the message options that we get back after we have connected to the server. The first one is one of the most interesting for us because the Silla shard information tells us which shard ID or core was assigned to the connection by the server. And I'm going to say this again. This information tells us what shard ID or core, why is assigned to the connection by the server. Since we have no way of asking anything when we connect, we are dependent on the server shard allocation to the connection that we open. This is a protocol limitation. So, now we are going to change the host connection pool class. We need to, to get a connection object for every core of the node, right. So the first thing we did is that we got rid of the single connection that we had before. And replaced it with a dict where the keys are the shard ID numerical shard ID and the values are the connections that are bound to the specified job. The first time we connect, as you can see in the first square rectangle, the first connection allows us to detect if we are connecting to a shard aware cluster or a Silla cluster and for instance. And this is where we get the first glance at the sharding information and we store it. Then I, the first time I saw the initial implementation like this. And this is the second part here with the for underscore in range. We can see that we are doing an insecure news and optimistic way to get a connection to every core. Why is that we open a new connection to the node and storage shard ID plus connection object on the dict. And we will do this twice as many times there are cores on the remote node until we have a connection to all charts. Maybe. Because if you remember, the client when it connects cannot specify which shards it want to connect to. So you just happen to connect to the server and you get a shard assigned. That's why the initial implementation was trying and saying, okay, let's let's try twice as much as there are cores available on the remote node and hopefully we'll get we'll get a full connection and a connection for every shard. If we were lucky. Fine. Keep on moving. If not, we would raise an exception. No. This is the first time I saw this I understood that there was the I understood this flow in in the client not being able to request a specific shard ID because of the protocol limitation so there is no deterministic and secure way to get the connection to all remote shards of a seal and connecting also synchronously means that the startup of our application would be as slow as connecting to hopefully all shards of all nodes. Not acceptable. So, the second thing that that came to my mind was a. This also means that all the current shadow where drivers since it's the protocol limitation it's not bound to Python and whatever it's not a Python's problem. It's, it's a flow or a lack, maybe not a flow but rather lack in the protocol itself. So that means that all the current shadow drivers are lying since none of them even today can guarantee to always have a connection for a given routine key. All of this is opportunistic and optimistic. You will eventually get one, but not all your queries will be able to use the direct connection to the right child. So I wrote an RFC on the sealer they've mailing list to discuss this problem. And the good news is that the consensus to a solution was recently found. It will take the form of a new shard allocation algorithm that will be placed on the server side. And that will be made available as a new listening port on Silla. So since we want and want to see la and Cassandra on the, their default port to retain the same behavior. If we want to change a bit and add a and add on the server a new kind of allocation for allocation, we need to do it on a new on your new port. So it will be a shadow where pop, let's say it will just use a sort of modular on the clients socket source port to assign the correct and calculate and assign the correct shot to the connection. So it means that the clients on the side will just have just have to calculate and select the right socket source port to get a connection to the desired child ID. This is work in progress, but it's not done yet. So I worked on implementing a software and optimistic and asynchronous way of dealing with this problem. Let's see how it's been done. The first thing is that I wrote two functions. The first one is the optimistic one. It opens a connection, tries to open a connection and only stores it if the reported shot ID was not connected before. Else, we close it. So we are only interested in the keeping connections to missing shot IDs. But we just open it and if it works good if it doesn't. We'll see later. Then I switch the startup logic to schedule as many optimistic of those optimistic attempts to get a connection to a missing shot ID has their shots available on the remote node. So when you connect and you start connecting, if you have 50 cores on your remote server, you will issue asynchronously and schedule asynchronously 50 attempts to get a connection to shots. Maybe 25 of them will give you different and unique shot ID is connected. Maybe two of them, maybe 50 of them lucky you. But now we don't care. It's optimistic asynchronous and it will go on and on again like this as we use the driver. The result is a cluster faster. An application startup time that is as fast as the usual Cassandre driver and a and non-blocking optimistic short connections. So the cluster object now it should pass down the routine key as well. To the pool. So when you see here when you issue a query in the query function and we and the cluster looks up cluster object looks up for a connection there. We added the routine key so that we could apply the shot ID calculation. This shot ID calculation is a bit obscure and lucky lucky me Israel was there to implement it in the first place. But the first time I tried to use this pure Python implemented shot ID calculation. It was very bad on the driver performance. We were slower than the Cassandre driver. So what Israel did is to move this shot ID computation to sitan because actually the Cassandre driver is using a lot of of sitan in the background when you install it. And he managed to cut its latency impact by almost seven. So, could also again Israel it was a very, very impressive move and it made it made the difference on the driver's perspective. So now let's wrap it together and in the main shadow awareness logic in the in the host connection pool. And so here. This is basically where the connection selection happens and everything is glued together. So if we are in that's line two, if we are on a shadow where communication with a cluster, we will calculate the shot ID now from the routine key token. Then we will use the routine key, the shot ID and we will try to look up in our connection dict if we happen to have a connection to this direct shot ID so to this direct core. If we if we do triumph, we will use this direct connection to the right core to enroute the query there. Almost. That's perfect. That's the best case scenario. If not, we will pray and issue. I synchronously a new attempt to connect to missing shard. Maybe it will be the shard we were trying to look at before. Maybe it will be another one. But that means that as much as you issue queries to your cluster, the more chance you get to have a connection to all sharp and all course. And if you didn't have one, we will just random pick an existing connection. So we would be just as if we were using the customer driver. Now that we have seen the implementation details does the seller driver live up to our expectations is it fast and how did it work in production because to us. And to me, the real values and are from must be taken from production. So let's see. The first expectations that we check was the, an increase we expect an increase in the number of open connections on from the cluster perspective. And this is a check because now that we are opening not only one connection to each node, but one connection to each core of each node. We, we expected to see this increase. So you can see, we's from the annotation when we really when we deployed the seller driver. We saw this increase. The second one was also an expectation to have more CPU requirements, because you open more connections, meaning that your driver and your CPU has to handle more connections and keep alive, etc. For, for to keep those connections alive. We saw that we had to increase from on our Kubernetes deployments, a bit the CPU limits to avoid CPU saturation and throttling. But then what about the major impact we wanted, we want faster queries lower latencies right. How did that translate for real. This is how what were graph look like I was like, ha, wow, it's amazing. We gained between 15 and 25% performance boost at Lumberley we like to look at our graph on the worst case scenario possible. That means that if you check this out. This is the max of our processing. This is the worst latency that we get from our application perspectives. What's interesting to note as well is that the performance boost is progressive. Since we connect to shards in the background in an optimistic fashion. The longer our application runs, the more chance to have a connection to all shards it has. And then the lower the latency guests, because we start to have always have the right connection to the right core for every query. So you can see that right after the deployment, we get already a fair amount of performance boost, but the longer the time passes, the more shards connected, the better the latency. It was very interesting to see we can see it also from afar if we apply a moving median on another power angry process, you can clearly see the big difference that that the the CLA shard of where driver has made in our production applications. From this we got an unexpected and cool side effect is that we manage as well to, since the cluster load was reduced and the client last latency was lower. Our for the same workload we could cut by half the number of replicas on our deployment, which was very cool as well so we save actually resources on our Kubernetes cluster. Recent additions that we've done on the driver. We have added some some helpers to allow you to check for shadow awareness and to check for this opportunistic shadow where connections so you can actually see how much of it is fully or not connected. When it becomes available, we will be changing also the driver to be able to select deterministically this time, the shard ID when it connects. So there are two open pool of requests already for this, we're going to still work on improving the documentation and since it's a Cassandra driver fork, we will imagine rebase the latest improvements as well. And try the CLA driver. It's working great. It's working production for us for now. Almost almost a month and with the great impact that you've seen before. Check it out on the repository. Come chat with us as well. When your Python is over on the CLA DB clusters like we have a Pythonistas channel where you're all welcome. And that's it for me. I want to thank everyone for for attending and making this Europe. I turn a success. There is the discord channel where we can keep in touch and discuss this further or deeper if you if you want. And tomorrow we will also have a sponsor talk session where we will be talking about a lot of different aspects and and have cool guests from a number of years. Well, thank you very much. Thank you very much for the talk. We have a minute or two for questions. Do you mind some Q&A? No, of course. Yes, there's one question that has gotten some votes. The code of the driver seems to be Python 2 compliant. But is there an Asyncio part like for Python 3.5 or better? Yes. Yes, it's an old code base actually. The Cassandra driver is quite old and since we have been forking it to extend it, we inherited this as well. So yes, it is still supporting Python 2 and there are Asyncio and LibyEV as well, connection class. So you can also change your connection class. The default one is Asyncio. We have a question from Roberto. Q1 shards are pinned to CPU or what? Are there corrections for unbalanced shards, e.g. one CPU to be hit by more than the others? Is that correct? Sorry, one CPU to be hit by more than the others? Yes. If you have an unbalanced in your data distribution, you will get the same kind of problem that you can get on a node. But instead of impacting the whole node, you will be impacting a core. So yeah, this problem exists as well in this kind of architecture. This is inherent of a consistent hashing based architecture actually. Okay, thank you very much. There's a few more questions in the Discord talk channel. So you're going to find that you have raised a lot of interest. There was also an off topic question before we let you go. What's with those papers hanging there behind you? Yeah, this is a way to keep notes above my head, you know. Perfect. Thank you very much for taking the time to show everyone. Thank you for sponsoring your Python. And here's a round of applause for you and you're going to meet more questions in the chat.
 Okay, so we're coming to our third session in this block and you've just seen messages  from some of our sponsors, but some of our sponsors also do talks.  And the next one I'd like to invite to us is Alexis from Numbuli.  Alexis, hello.  Hello, everyone. Can you hear me alright? Can you see me as well?  This works perfectly and we can see you too. You are in Paris at the moment or in France?  Yeah, in France, in the south of France actually, this time, but I usually live in Paris and work in Paris.  Are you ready to do your presentation?  Sure, I guess so. I will just start sharing my screen and prepare everything if you allow.  Yes, I'll show.  Okay.  And let's see if that works.  So hello, everyone. I'm really happy to be remotely as you understood with you today.  In this talk, I will detail Cassandra and Cilla low level architecture and explain how it's used by Python Cassandra driver and show you how we extended it to write a Python driver for Cilla.  There will be diagrams, emojis, Python code, and hopefully some amazing performance graphs as well.  Let me introduce a bit about myself. So, how's Martin already told you and you can judge from my accent. I'm French, I'm Alexis, and I'm a CTO at number Lee weapons as well to be a proud sponsor for Europe items since 2014.  And we are digital marketing experts and we help brands establish a relationship, a digital relationship with their customers.  In the open source world, I gen two Linux developer, where I'm part of the cluster and container teams, which means that I spend some of my time working on packaging distributed databases such as MongoDB and Cilla, and all work on packaging  cluster related tools and work with my friends on the Docker gen two Linux images.  I'm also open source contributor and enthusiast. I've been contributing to MongoDB to Cilla to Apache Airflow, and I'm a Python software foundation contributing member which means that I spend a fair amount of my time working on or contributing to Python based open source projects.  Before we start, I wanted to introduce you to a fact that you may not know already, but since my Europe item is using discord, I found interesting to share to you that discord is using Cilla as well.  So if you are interested in understanding what they do, I invite you to check out this link here where marksmith introduces how a discord is using Cilla. It's very interesting. So check it out if you want.  This is an advanced talk as advertised. So I will suppose that you are familiar with the basics of consistent hashing and Cassandra. If that's not the case, or if you simply want to know more about consistent hashing and even use it in your Python applications.  I'm allowing myself to also introduce you to the talk I gave on this very subject in a previous Europe item edition, but fear not, I've still worked on not making this a big problem for this presentation. So you should be able to follow along and no problem, even if you don't know exactly the details of consistent hashing.  Let's get started now. And let me introduce you to Cassandra and Cilla token reading architectures. We see all they have in common, but also what Cilla has done that makes it special and worth a dedicated Python driver.  The first thing to know is that Cassandra or Cilla cluster is a cluster is a collection of nodes or instances that can be visualized as a ring.  All the nodes in this ring should be homogeneous using a share nothing approach. This means that there's nothing special about the node in this topology one node one Cassandra node on the on the ring or any Cilla node on the ring has no special role or anything special about it.  There is no primary or secondary or anything. They all do the same thing.  The ring is called a token ring in which the positions of the nodes on the ring define token ranges and token range partitions.  You can see that before if you go clockwise on the ring, the range that is preceding a node is the token range that all the partition that it is responsible for.  Partition is just a subset of data that is stored on a node in SQL.  The Cassandra query language a partition appears as a group of sorted rows and is the unit of access of query data.  This data is usually replicated across nodes, thanks to a setting that is called the replication factor.  The replication factor defines how data is replicated on notes. For example, a replication factor of two means that a given token or token range or partition will be stored on two nodes.  This is the case here where partition one and two is stored on X. And you can see that partition two is also stored on Y while the partition one is stored on Z.  If we were to lose node X, we could still read the data from partition one from the Z.  This is how high availability is achieved and how Cassandra and Cilla favor availability and partition tolerance. They are called AP on the CAPTRM.  This kind of token ring architecture is sensible to data distribution among the nodes. Queries should in theory be evenly distributed between nodes.  We could get an imbalance of data and query load in the above scenario where we store data in three big partitions and each node holds one range of the previous one and one range from the next one.  If one of those partitions were to grow larger than another one, we could then have an imbalance of queries and over and sort of overload on them.  To counter this effect that we are calling a hot node or hot partition, we need to add more variance in partition two node allocation. And this is done using what is called virtual nodes.  So instead of placing physical nodes on the ring, we will place many virtual instances of them called virtual nodes. A virtual node represents a contiguous range of tokens owned by a single node.  So it's just a smaller slice of a partition.  But it's more shuffled between nodes. A physical node may be assigned multiple and non contiguous.  If you remember the preceding slide, it was contiguous. This time, the actual notes allow for non contiguous assignment of notes.  The default is to split a node into 256 virtual nodes on the token ring.  So this is true for Cassandra and this is true for Cilla as well.  So if you look at how now the partitions are distributed among notes, you see that there is more variance into this, which will end up in distributing the query bear.  This is it for Cassandra's data distribution, but Cilla goes one step further.  On each Cilla node, tokens of a V node are further distributed among the CPU cores of the node that are called shards.  This means that the data stored on a Cilla cluster is not only bound to a node, but can be traced down to one of its CPU cores.  This is really interesting architecture and low level design.  This is the feature that we will leverage on the Python site Cilla shadowware driver later on. And I will explain to you how.  Now that we understand how data is stored and distributed on the cluster. Let's see how it's queried by clients.  On the physical layer partition is the unit of data stored on a node and is identified by a partition key.  You can relate a partition key to a primary key in the SQL world.  A partition key is the primary means of looking up a set of rows that comprise a partition.  And a partition key serves to identify the node in the cluster that stores a given partition as well as to distribute the data across nodes in the cluster.  The partitioner or the partition hash function using the partition key will help us determine where the data is stored on the given node in the cluster.  So you take the ID in this case, you see a colon ID, you will, this will be the partition key, you take the value, you apply a hash function on it, the partitioner hash function, which by default on Cilla is murmur hash tree.  And this will give you a token. A token is like a number, which is just actually is a number that will be placed on the token ring.  And from where it leads on token ring, you will find out which nodes is which node is responsible for this data.  That's as simple as this.  Okay, so let's recap now on Cassandra, the hash of the partition key gives you a token telling you which node has the data that you are looking for.  We can see this architecture as a shard pair node architecture, because from the from a token, you get to a node so short per node on Cilla.  The same hash of the partition key from the same partition key gives you the same token, but the same token not only is not only telling you which node has the data, but also which CPUs core in this node is responsible for handling it.  So this is a short per core architecture. This is how it's called. So Cassandra is short per node while Cilla is short per core.  Now, let's see how does a client driver query Cassandra or a Cilla cluster.  Because now that we know this, we could guess and expect that client drivers uses this knowledge to find out and optimize their query plan.  A knife client would go on like this when a client connects to a Cassandra or Cilla cluster, it opens a connection to every node of the cluster.  When it wants to issue a query, a knife client would pick one of its connections randomly, let's say, and issue the query to the node.  The node it issues the query to will be seen as the from the client's perspective, it will act as what he's called a coordinator because he's coordinating the query and he's taking the ephemeral responsibility of rooting the query internally in the cluster to the right  nodes that are replicas for this data. That means that are responsible for the partition, the query belongs to and gathering this responses and then responding back to the client.  But maybe this coordinator is not a replica for the query data.  If it's not the case, if the coordinator way is not a replica for the query data, it has to issue the queries to all replicas itself. That means that you will add it will add an extra hop inside inside the cluster to get the responses.  This is sub optimal, of course, as it consumes network and processing power on the coordinator node for something that the client could have guessed in the first place, right.  Because since the partition a hash function is known, our client library can use it to predict data location on the cluster and optimize the query routing.  This is what the Cassandra driver does using the token aware policy.  How does it work token aware clients, apply the partition logic to select the right connection to the right node and make sure that its coordinator node is also a replica of the query data.  This is cool. And this is very efficient as a result, we save network hopes, lower the cluster internal load and get reduced query latency, meaning, meaning faster queries.  Let's see how the Cassandra driver does it for really internally. So the token aware policy from the point of view of the Python Cassandra driver, the partition key.  It is seen as a routine key, because it will be used to route the query, right.  So it is seen as a routine key which is used to determine which node are the replicas for the query.  And to know how to allow our Python driver to know about the partition key of a query, the query itself must be prepared as a server side statement.  So it looks on in Python Cassandra's and CLA's prepared statements that can you can see them a bit like stored procedures in the SQL world.  You see, if you see statement equals session dot prepare, and then you, you express the query that you want. And when you have an argument or parameter, you just put an exclamation, an integration mark.  And this is the recommended and most optimal way to query the data because when you have prepared your, your query, it is validated and it leaves on the server side so you don't have to pass it.  And you just have to pass a reference to it, and then only pass the arguments and the arguments, one of them will be the mandatory routine key partition key then routine key.  So statement plus routine key equals nodes.  And that's very, very cool.  Another thing to note, it is that just like prepared stored procedures, prepare statements are those sort of safest way, because they prevent query injection so please, in production at the bare minimal only use prepare statements when you issue queries to Cassandra or CLA clusters.  And then the Python Cassandra driver defaults to the token aware to root the query and then it also defaults to a data center where round robin load balancing query routine policy it's, it's a big long, but what it means that is that it will load balance for you in a round robin fashion so one after the other after the other like this.  So bear minimal load balancing algorithm there is, but it's still pretty efficient. So, don't worry if even if your cluster is not spread between multiple data center is still works it's just, it just happens to be the default so it's token aware plus data center where round robin.  By doing so the query routine will not only hit the right node, holding a copy of the data that you seek. Remember it's called the replica replica, but also load balance the queries evenly between all its every class so  one can think yeah this is awesome and optimal. I mean, from the Cassandra cluster point of view, it is, and we can do better than this.  But not with a CLA one.  Remember, CLA shards the data one way further down to node CPUs. So, having a token aware token awareness is cool. But if our client had shared awareness, it would be even cooler, because this means that a token aware client could be extended to become a shadow  aware client to root his queries, not only to nodes, but right to their CPU cores. She is very interesting to do.  Such drivers, they already exist as forks of the data stacks and Cassandra drivers, and it's true for the java one and the go one as well and they have been around since last year, actually.  There was no shadow where driver for Python and it made it made me sad and pretty angry. So when I attended CLA submit last year in San Francisco, I did some lobbying and hard lobbying on the on and found some CLA developers that are that were willing to help in  in, in, in, in, in making this happen for Python as well.  So we promised each other to make a Python shadow where driver and good news is that we have obviously did. And I will now explain to you in details how it has been done.  And what we found out by doing so.  Very, very interesting as well.  Let's start by checking out the expected structural differences between the Cassandra driver and the CLA driver fork.  The first thing to see is that the token aware Cassandra driver has, when he connects for the first time to the cluster, it opens a control connection this control connection allows your customer driver to know about the cluster topology how many nodes they are, which are up which are  the, what are the, the schemas, etc, etc. It needs to know all this so it opens a special connection for this and in this connection it refreshes from time to time.  And then it will open one connection per node, because this is how the token aware policy will be applied to select the right of those connection based on the query.  And then it will be done by the famous token calculation.  And that's how it's done. All right. On the shadow where client perspective, we still need to know about the cluster topology, actually, but instead of you opening one connection per node we will be opening one connection per core per node.  The token calculation will still be useful to select the right node from the token perspective, but then we will need to add a shard ID calculation because we need to go down to the shard or the CPU core to so to select the connection to the right core to root the queries.  Let's transform this into a to do from the Python code perspective.  First thing is, since we will be using the same kind of control connection, we don't, we just use is as is there's nothing to change here.  We will need to change the connection class object, because now, when we are going to open a connection per core per node, we will need to be able to detect.  If we are talking to a CLA cluster or to a Cassandra cluster.  The CLA driver, we wanted to retain the maximum compatibility with Cassandra as well so you can use the CLA driver to discuss and query your Cassandra cluster as well.  The host connection pool should will use those, those shadow where connections to open one connection to every every core of every note.  The current calculation that selects the right node will be the same.  We will just use the vanilla and already existing and efficient token aware policy.  But then we will need to extend it and had in the cluster when you issue the query, we will need the cluster class to pass down the routine key to the to the connection pool.  We will try the shard ID calculation and then implement the logic based on the shard ID of selecting the right connection to the right node to the right shard.  Okay, sounds like a plan. Let's do this. Now, we'll get down into the code.  Before we we go into this, I wanted to highlight and to introduce Israel Israel is a CLA developer from Israel. I know it's confusing.  That's how it is.  This is in the audience. That's makes me some kind of pressure as you can guess. I wanted to take this opportunity to thank him and giving him the credits for most of what I'm going to present now, especially the efforts he put into CI testing the CLA driver.  So the first thing that that that that needed to be done is to have those shard information to the connections. And so this is how it's been done. So a connection now has a shard ID assigned to it and sharding information.  This sharding information comes from the server responses when we issue a query. That's what is what is squared in red on the bottom. The logic of this looks like this.  And it is what's interesting to note is that the Cassandra protocol allows only for connection message options being passed on the server response.  This means that when the client initially connects to Cassandra or Silla, it has no way of passing any kind of information to the server. So we are dependent on the server response to know about shard shard information or whatever it is that we need.  If we look at the, the, the message options that we get back after we have connected to the server. The first one is one of the most interesting for us because the Silla shard information tells us which shard ID or core was assigned to the connection by the server.  And I'm going to say this again. This information tells us what shard ID or core, why is assigned to the connection by the server. Since we have no way of asking anything when we connect, we are dependent on the server shard allocation to the connection that we open.  This is a protocol limitation.  So, now we are going to change the host connection pool class.  We need to, to get a connection object for every core of the node, right. So the first thing we did is that we got rid of the single connection that we had before.  And replaced it with a dict where the keys are the shard ID numerical shard ID and the values are the connections that are bound to the specified job.  The first time we connect, as you can see in the first square rectangle, the first connection allows us to detect if we are connecting to a shard aware cluster or a Silla cluster and for instance.  And this is where we get the first glance at the sharding information and we store it.  Then I, the first time I saw the initial implementation like this.  And this is the second part here with the for underscore in range.  We can see that we are doing an insecure news and optimistic way to get a connection to every core.  Why is that we open a new connection to the node and storage shard ID plus connection object on the dict.  And we will do this twice as many times there are cores on the remote node until we have a connection to all charts.  Maybe.  Because if you remember, the client when it connects cannot specify which shards it want to connect to.  So you just happen to connect to the server and you get a shard assigned.  That's why the initial implementation was trying and saying, okay, let's let's try twice as much as there are cores available on the remote node and hopefully we'll get we'll get a full connection and a connection for every shard.  If we were lucky.  Fine.  Keep on moving.  If not, we would raise an exception.  No.  This is the first time I saw this I understood that there was the I understood this flow in in the client not being able to request a specific shard ID because of the protocol limitation so there is no deterministic and secure way to get the connection to all remote shards of a seal and  connecting also synchronously means that the startup of our application would be as slow as connecting to hopefully all shards of all nodes.  Not acceptable.  So,  the second thing that that came to my mind was a.  This also means that all the current shadow where drivers since it's the protocol limitation it's not bound to Python and whatever it's not a Python's problem.  It's, it's a flow or a lack, maybe not a flow but rather lack in the protocol itself.  So that means that all the current shadow drivers are lying since none of them even today can guarantee to always have a connection for a given routine key.  All of this is opportunistic and optimistic.  You will eventually get one, but not all your queries will be able to use the direct connection to the right child.  So I wrote an RFC on the sealer they've mailing list to discuss this problem.  And the good news is that the consensus to a solution was recently found.  It will take the form of a new shard allocation algorithm that will be placed on the server side.  And that will be made available as a new listening port on Silla.  So since we want and want to see la and Cassandra on the, their default port to retain the same behavior.  If we want to change a bit and add a and add on the server a new kind of allocation for allocation, we need to do it on a new on your new port.  So it will be a shadow where pop, let's say it will just use a sort of modular on the clients socket source port to assign the correct and calculate and assign the correct shot to the connection.  So it means that the clients on the side will just have just have to calculate and select the right socket source port to get a connection to the desired child ID.  This is work in progress, but it's not done yet. So I worked on implementing a software and optimistic and asynchronous way of dealing with this problem.  Let's see how it's been done.  The first thing is that I wrote two functions. The first one is the optimistic one.  It opens a connection, tries to open a connection and only stores it if the reported shot ID was not connected before.  Else, we close it.  So we are only interested in the keeping connections to missing shot IDs.  But we just open it and if it works good if it doesn't.  We'll see later.  Then I switch the startup logic to schedule as many optimistic of those optimistic attempts to get a connection to a missing shot ID has their shots available on the remote node.  So when you connect and you start connecting, if you have 50 cores on your remote server, you will issue asynchronously and schedule asynchronously 50 attempts to get a connection to shots.  Maybe 25 of them will give you different and unique shot ID is connected.  Maybe two of them, maybe 50 of them lucky you.  But now we don't care.  It's optimistic asynchronous and it will go on and on again like this as we use the driver.  The result is a cluster faster. An application startup time that is as fast as the usual Cassandre driver and a and non-blocking optimistic short connections.  So the cluster object now it should pass down the routine key as well.  To the pool. So when you see here when you issue a query in the query function and we and the cluster looks up cluster object looks up for a connection there.  We added the routine key so that we could apply the shot ID calculation.  This shot ID calculation is a bit obscure and lucky lucky me Israel was there to implement it in the first place.  But the first time I tried to use this pure Python implemented shot ID calculation.  It was very bad on the driver performance.  We were slower than the Cassandre driver.  So what Israel did is to move this shot ID computation to sitan because actually the Cassandre driver is using a lot of of sitan in the background when you install it.  And he managed to cut its latency impact by almost seven.  So, could also again Israel it was a very, very impressive move and it made it made the difference on the driver's perspective.  So now let's wrap it together and in the main shadow awareness logic in the in the host connection pool.  And so here.  This is basically where the connection selection happens and everything is glued together.  So if we are in that's line two, if we are on a shadow where communication with a cluster, we will calculate the shot ID now from the routine key token.  Then we will use the routine key, the shot ID and we will try to look up in our connection dict if we happen to have a connection to this direct shot ID so to this direct core.  If we if we do triumph, we will use this direct connection to the right core to enroute the query there.  Almost.  That's perfect.  That's the best case scenario.  If not, we will pray and issue.  I synchronously a new attempt to connect to missing shard. Maybe it will be the shard we were trying to look at before. Maybe it will be another one.  But that means that as much as you issue queries to your cluster, the more chance you get to have a connection to all sharp and all course.  And if you didn't have one, we will just random pick an existing connection. So we would be just as if we were using the customer driver.  Now that we have seen the implementation details does the seller driver live up to our expectations is it fast and how did it work in production because to us.  And to me, the real values and are from must be taken from production. So let's see.  The first expectations that we check was the, an increase we expect an increase in the number of open connections on from the cluster perspective.  And this is a check because now that we are opening not only one connection to each node, but one connection to each core of each node. We, we expected to see this increase. So you can see, we's from the annotation when we really when we deployed the seller driver.  We saw this increase. The second one was also an expectation to have more CPU requirements, because you open more connections, meaning that your driver and your CPU has to handle more connections and keep alive, etc.  For, for to keep those connections alive. We saw that we had to increase from on our Kubernetes deployments, a bit the CPU limits to avoid CPU saturation and throttling.  But then what about the major impact we wanted, we want faster queries lower latencies right. How did that translate for real.  This is how what were graph look like I was like, ha, wow, it's amazing.  We gained between 15 and 25% performance boost at Lumberley we like to look at our graph on the worst case scenario possible. That means that if you check this out.  This is the max of our processing. This is the worst latency that we get from our application perspectives.  What's interesting to note as well is that the performance boost is progressive. Since we connect to shards in the background in an optimistic fashion.  The longer our application runs, the more chance to have a connection to all shards it has.  And then the lower the latency guests, because we start to have always have the right connection to the right core for every query.  So you can see that right after the deployment, we get already a fair amount of performance boost, but the longer the time passes, the more shards connected, the better the latency.  It was very interesting to see we can see it also from afar if we apply a moving median on another power angry process, you can clearly see the big difference that that the the CLA shard of where driver has made in our  production applications.  From this we got an unexpected and cool side effect is that we manage as well to, since the cluster load was reduced and the client last latency was lower.  Our for the same workload we could cut by half the number of replicas on our deployment, which was very cool as well so we save actually resources on our Kubernetes cluster.  Recent additions that we've done on the driver. We have added some some helpers to allow you to check for shadow awareness and to check for this opportunistic shadow where connections so you can actually see how much of it is fully or not connected.  When it becomes available, we will be changing also the driver to be able to select deterministically this time, the shard ID when it connects.  So there are two open pool of requests already for this, we're going to still work on improving the documentation and since it's a Cassandra driver fork, we will imagine rebase the latest improvements as well.  And try the CLA driver. It's working great. It's working production for us for now.  Almost almost a month and with the great impact that you've seen before.  Check it out on the repository. Come chat with us as well. When your Python is over on the CLA DB clusters like we have a Pythonistas channel where you're all welcome.  And that's it for me. I want to thank everyone for for attending and making this Europe. I turn a success. There is the discord channel where we can keep in touch and discuss this further or deeper if you if you want.  And tomorrow we will also have a sponsor talk session where we will be talking about a lot of different aspects and and have cool guests from a number of years. Well, thank you very much.  Thank you very much for the talk. We have a minute or two for questions. Do you mind some Q&A?  No, of course. Yes, there's one question that has gotten some votes. The code of the driver seems to be Python 2 compliant. But is there an Asyncio part like for Python 3.5 or better?  Yes. Yes, it's an old code base actually. The Cassandra driver is quite old and since we have been forking it to extend it, we inherited this as well.  So yes, it is still supporting Python 2 and there are Asyncio and LibyEV as well, connection class. So you can also change your connection class. The default one is Asyncio.  We have a question from Roberto. Q1 shards are pinned to CPU or what? Are there corrections for unbalanced shards, e.g. one CPU to be hit by more than the others?  Is that correct? Sorry, one CPU to be hit by more than the others?  Yes. If you have an unbalanced in your data distribution, you will get the same kind of problem that you can get on a node. But instead of impacting the whole node, you will be impacting a core.  So yeah, this problem exists as well in this kind of architecture. This is inherent of a consistent hashing based architecture actually.  Okay, thank you very much. There's a few more questions in the Discord talk channel. So you're going to find that you have raised a lot of interest.  There was also an off topic question before we let you go. What's with those papers hanging there behind you?  Yeah, this is a way to keep notes above my head, you know.  Perfect. Thank you very much for taking the time to show everyone. Thank you for sponsoring your Python. And here's a round of applause for you and you're going to meet more questions in the chat.
Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_0_155.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_156_1547.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_1548_2363.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_2364_4487.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_4488_5243.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_5244_6251.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_6252_6335.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_6336_6527.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_6528_6719.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_6720_8915.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_8916_11519.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_11520_13163.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_13164_14135.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_14136_16175.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_16176_17483.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_17484_17783.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_17784_18047.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_18048_20255.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_20256_20315.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_20316_20363.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_20364_20483.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_20484_20615.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_20616_20723.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_20724_21599.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_21600_24527.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_24528_26231.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_26232_26387.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_26388_26483.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_26484_26603.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_26604_26879.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_26880_26951.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_26952_27479.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_27480_28547.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_28548_28667.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_28668_28811.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_28812_28907.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_28908_29051.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_29052_31535.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_31536_34067.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_34068_35723.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_35724_37823.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_37824_41951.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_41952_45335.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_45336_45695.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_45696_45827.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_45828_46235.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_46236_47795.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_47796_49031.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_49032_49835.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_49836_52127.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_52128_52199.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_52200_52247.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_52248_52379.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_52380_52439.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_52440_52487.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_52488_52607.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_52608_54347.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_54348_56147.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_56148_57155.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_57156_58307.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_58308_59039.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_59040_59879.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_59880_62195.png Alexys_Jacob_-_A_deep_dive_and_comparison_of_Pytho_vWgTF5xnx8M_62196_63543.png