Performance Guillotina
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Gheem, Nathan Van
Navarro Bosch, Ramon
We are going to go through all the tricks, decisions and compromises taken with Guillotina and their addons to provide performance and keep easy usage as a priority. Topics will include object and field storage techniques, cache storage and invalidation, catalog integration strategies, queue integrations, and data streaming.
 We are going to do a technical presentation of Guillotine. Well, mostly, yesterday they explained why we created it, and I was saying all the time, we are going to explain about the SYNC-IO, we are going to explain what was the problems that we were facing, we were doing the framework two years ago, and this is the talk that we are going to go through all the different points that we had pain using blown and using ZODB and using RelaStorage. So we are going to explain all the decisions we needed to make in order to build a framework that fulfills our requirements. First of all, if you don't know, I am Ramon Navarro, I have two companies, is going to testify, I am a blown foundation member and I am Catalan, and this is the demonstration in Catalonia last week. Hi, and I am Nathan Manguim, I work at Ona as principal engineer, I am a blown foundation member and I don't really play guitar anymore, I just felt I needed that because Ramon always talks about playing music and I am American. Normally, I wouldn't say anything about that either, but you know. So we are just going to try to go topic by topic to talk about some of the different areas that we used to tweak performance and create a framework for scaling to a lot of data in Python. Some of those are, I guess I can repeat them, but you can read them as well, the async I O and then just the design of the database, caching and life cycle management, indexing, file storage and queues. So the first thing we will go over is async I O and I guess I just want to stress that async I O is really important for Python and the web in general going forward. The threaded approach, like okay, let's, we will just talk about how async I O is different than a threaded app. And a threaded app, when a request, an HTTP request comes in, it is assigned a thread and that thread is active through the duration of the request and once it finishes, the thread is returned to another request. So you are limited to the number of simultaneous connections for the number of threads that you have on your application. And Python, that usually means two because Python doesn't perform very well with lots of threads. The best way to deploy Python is with two threads per process. So you can just keep doing more processes, the scale, which is fine, but then you don't have any memory sharing or anything like that. But moreover, it is just also, it is the differences between how you block on your application. So if you have the threaded approach, a request comes in and then you communicate with say Postgres SQL and maybe it happens to be a slow query and you are blocking for three seconds to wait for that query to finish. Now that thread is consumed for that whole three seconds while that query is communicating that Python is not doing any CPU, it is not doing anything at all, but it is still blocking and it is preventing other people from being able to have their request serviced. So that is the threaded model of request per thread and that is Django right now, that is Flask and Pyramid. And so think about how this applies to situations where you are communicating with lots of services. You are sending things to a message queue, you are communicating with the Redis for caching, you are doing OAuth for authentication or any kind of authentication or authorization provider, you are doing S3 for file storage, like any of these services, they are all over like HTTP or some kind of TCP connection and now you are blocking on communication. If there is anything slow in any of those services, you are blocking and you are not doing anything. And so this is where async.io is useful, like it is non-blocking network.io now and you can communicate with all these other services and while you are communicating there is very little overhead and maintain that connection and you can really push the performance of Python where you were not able to before. And moreover just think about the context of microservices and even if you think microservices is crap, the reality is you don't have to be microservices, you are still using a lot of other services. You are connecting to other APIs in a number of ways, you are at least using like multiple database layers or like most applications you develop now are not just database only, you are going to be communicating with something, some API, maybe an authentication service or something. So like this is going to be a problem if you have any sort of substantial load. And to illustrate this even further, I guess I do have an example of some code to show like how this can be even this is going to be an example of a CPU bound application versus a network bound application. And so like one of the, I will back up a little bit because I forgot to explain the downside of async.io. So async.io is a single threaded approach to running your application. So one process, one thread, your event loop which is running the async code is only on one thread. So if there is any CPU bound code there, it is going to block on CPU because no other code can run at the same time if you are executing CPU, it is single threaded. So it is really important that you do not block on CPU. Whereas threads, if you have high CPU load, you can switch threads every once in a while and you will never have a situation where you are blocking all other code from running. So that is one downside. So if you know you have CPU bound code, you have to be careful about it. In async.io there is a way to run CPU bound code in a thread pool. So you just have to know what you are doing. But anyways, this is to demonstrate, this is some code that just crunches on numbers for half a second and we can see how it behaves. And the regular async.io approach is to request a second and the threaded approach without setting any threads to request a second. It behaves the same. Async.io with the executor, that is what I was talking about how you can throw CPU bound code in a thread pool that does slightly better than a threaded application with 20 threads, which you do not really want to do, but that is just to say like you need to try to scale it further with a single process. The problem is not that much different with the CPU bound app. But if you go to a network bound app and this is the same sort of thing, we are just connecting to a service and that service is just something, just a service that is delaying for half a second before it returns, which is in async.io service of course. And then the difference is this. This is only bound by the number of concurrent requests the library was using per host. So it could go further. That was some arbitrary limit they had on number of concurrent connections per host. And the threaded app is abysmal for the performance. Yes. Just to say that async.io is amazing for network, but there is a lot of other input-opened things that we have on our computers that the async.io is also really helpful. We are not using a Guillotine specifically because most of the things is kind of a REST API and it's mostly network communication, but we are using also a lot for, for example, accessing the disk with long operations that you need to do on the disk. Or I'm sure that there is a lot of input-opened communications with devices, with the kernel that you could build drivers for async.io in order to be also on the event loop. So everything that you need to wait async.io is amazing. And sub-processing, yeah, you just start a new process. We have processes that start a lot of different processes and wait for them. Meanwhile, they are doing other things, serving HTTP so we can monitor the process that are running. All this input-opened is so easy with async.io. And there was another thing, Python delivers with a standard event loop, a basic one that works really well for most of the cases, but there are other implementations of event loops, for example, the UV loop, that it's much faster than the standard one, that it's the same as Node.js. And there is also the Tokyo event loop, that it's written in Rust, that it's also really fast. And the system is so pluggable that if you really need this extra 20% of the speed, you can even switch the event pool to something that is much more powerful. Sorry. So, I think async.io was the main important thing that we needed to come. So it was the first thing that we decided that we were going to happen. And the second thing that we decided is that we needed a database, a place to store. We started storing things on a file system, on folders, and well, it was okay. But we needed some kind of more strong way of storing with transactions that, so there is a lot of databases that offers that. And we went on to understanding how Rela storage works. So we say Rela storage has been there for a while, it's been growing. So we want a pickle system, what we can copy. And we went to check Rela storage. And we decided, a design, that it's a bit of inspired with Rela storage, also with that database that Jim built, new DB. So, Lutinates traversal. So there is main nodes that you have on the tree. And these main nodes are registries on a table. Which this makes easy to be able to do a select on the table and to do a dump and to get all that objects out of it. And we decided that we don't have enough with a row, with a byte field on Postgres to store all the information that we want from an object, because an object maybe can grow up a lot and we need to include extreme information. So we copied the concept of annotation from Plume. And we decided to have annotation registries that links to the node object. So if you do a select on a Glutinate database, you will see rows that are the node objects, the basic nodes on the tree, and nodes that are annotations from to these node objects. Then of course we are a tree. So on this structure, we needed to define a way of defining pointers even to the children or to the parent. In our case, we decided to point to the parent. So each row has a pointer to the parent or to the node that it's an annotation for. Why we did that? Because of course it's much easier to avoid to check conflicts because you are only pointing to one. Yeah. At the beginning when we started to do the design, we did not use the JSONB indexing from Postgres. So we decided that by then we are going only to store the pickle with a byte field on Postgres and then indexing it's for another system. We are microservices, so Glutinates only taking care of storing with transactions the main objects and the notation objects. So well, these are some of the configurations on Postgres that we have on right now. Mostly it's a simple one table, one big table with all the information. So it's database that's mounted on a point of the tree. It's that table and you can have multiple tables and mount on different endpoints on Postgres and on the URLs of Glutinates. Then there is a lot of foreign keys that make sure that you cannot delete, delete the children, all the parent, all the children gets deleted. If for example you want to do a post with the PlongRest API to create an object and you define which is the ID. Even if that ID exists, it's a simple operation because there is a primary key that doesn't allow to have different IDs for the same parent. So the database itself provides consistency. But related to that, you don't know until you try to commit if there's a conflict in this way because we didn't want to, as we'll go through more, you need to cut out all possible places where you're going to do a lookup. So if on, like when you're doing a post to create a new object and you have the ID and then you, at that moment you check to say, is there an ID with this already exist on the database, that's always going to be a lookup to the database. You always need a connection to the database in order to do that. So we were like, no, we'll just wait until we actually succeed and let the database do the constraint check for us. If the database gives a conflict error, then we kick it back out and it's a conflict error and the client can handle the 409. We're a REST API, so they should know how to handle 409s and that's kind of how it's handled. Thank you. Great. So another thing that is important that we use a lot and we are really thankful that Postgres can give us to go fast is the sequence number. So the transactions ID is quite, it's really important that our sequential and Postgres itself offers us a sequential ID that allows us to have a sequence number for doing that. There is two things that this architecture makes, there is a lot of things that makes this a bit complex but two major ones. One is that we lost the history, the ZODB kind of a pen only that you are storing all the information, all the transaction, so you can go on time and decide how far you can go. So we have a snapshot of actual situation. We can implement versioning on top of the objects by the data model itself but the transactions, we support what Postgres supports transactions and we have the snapshot of the database. Of course, as the previous talk explained it, you define a specific snapshot on time on the database then you can go back if you don't remove the history. Another thing that is difficult is the relation of children and parents. The children pointing to the parent, ordering is really complex because we really need to, you cannot store the order on the children, you need to store the order in the parent or you can store the list or if you store the order on the children then you need to index that properly. So if you guys are familiar with how this is done in Plone, ordering is like they are storing a complete list of the order of the folder on the parent. So every single time you add an object or you change the order, it is modifying the parent object. We don't do ordering. The Guilty in the CMS thing does ordering and it is just studying an attribute and so it is like a fuzzy on pseudo enforced ordering. So you are still only writing to the child and that is the safest way to do it where you can actually not have a huge performance implications. There is more on writing. There is a lot of things. So just to see an image is always after explaining all these things, an image helps a lot. So this is an SQL from a table from Glutina. You see that there are two specific registries, the DDDDD and the 0000. The 0000 is the root object. So it is the first object. And DDDDD is the trash. So when we are deleting something, we are making the pointer of the parent of that element to point to DDDDD. Why? Because we discovered that the reference integrity for key deletion trigger when we were deleting kind of folders with millions of objects were kind of shutting down the database. So we decided to do the deletion on batch operations. So we point the object to the DDDDD and then there is a batch job that cleans up everything. You will see during all some of the slides that we talk about vacuuming or batch jobs to clean up. We need to do a lot of these things. We cannot do everything in real time because we want to answer fast and we want to provide integrity. Yeah. So like if you do a delete of a really large folder in blown, it might take like 30 minutes or something like that because it is doing all of that in one single transaction, making sure that transaction is synchronized across all the other transactions that are happening and then hopefully it actually works. So all we do is set this attribute and all of our cleanup re-indexing, all that stuff is in an async task or in a queue and that is how we manage doing large operations that affect large, a large amount of data at a time. Yeah. So some other differences with Ralla storage is that of course we have the object ID, it is the ZOID. I am a romantic guy. I choose to name it ZOID for romantic reasons. You may understand. The transaction ID, this is the sequential number. State size is also romantic name to store the size of the byte. Part. Part is a field that you can create an adapter so each object can define in which partition is stored. This is used for what has been explained also in the talk before that Postgres 11 supports partitioning at Postgres 10 also, 11, no? Supports partitioning so you automatically do a partition, you can partition your database table with multiple sub-tables and using this attribute to define in which partition you want to store it. Of course, just to know if it is a main object or it is just an rotation, in case you are an rotation, the transaction ID is for the cycle of commit, the two-phase commit. Parent ID is the pointer to the parent. The ID is the real ID and this is different, for example, from, I think, on Ralla storage or maybe it is the type that you have the real ID that appears on the URL. Then the type, the real type from Glutina that we have, JSON is the serialization that we want to use for indexing in case we use Postgres catalog and a state that is the Pico. One other thing about the ZOID is you can see from row 3 to 6, all of the ZOID keys start with 824. If you are using something like Cockroach, you have CockroachDB support as well. In order for it to distribute the data, it does key sorting and you want to have your data near each other if you are querying it. If you didn't have this sort of key storing which allowed Cockroach to put chunks of data on the same node, it would be having to do the queries between CockroachDB nodes and synchronize those queries and it can be really slow. That is the reason why the ZOIDs are structured that way. Yeah, the three digits are the parent first three digits and each children gets the three digits from the parent also besides its own ZOID. Everything gets ordered on CockroachDB. We already explained a bit about that. Deletion is complex. Synchronize, making sure that you have a clean system, it's complex. Right now we have two systems of auto-baccuying. My default Guillotine tries to create an AsyncIO job to clean the database. But we also have an option to disable that so you have an external job that does it manually. Okay, no catching. Yeah, so we use Redis to facilitate our caching and then an in-memory cache. I mean, I guess in that respect it's a lot like the way ZODB caches objects. We just have an LRU cache that's pretty fast and written in C. We have cache keys for all operations that are done on the database. We set the in-memory cache and then we use Redis to coordinate and validate between Guillotine application servers. Yeah, so that's kind of it. Optionally, it can also push cache out to other application servers as well. Depending on your load, that can be actually a negative performance because that can put a lot of pressure on Redis to push out all that data if you have a lot of rights going on. So how I usually deploy or how we deploy, I'm sorry, is we just use Redis for invalidating cache across the application servers and everything else is just straight in-memory cache then and we don't push out any new values with Redis. Yeah, so the L2 cache would be a reference to the Redis actually storing cache of the pickles. So I don't use that, but it is supported and if you don't have a right heavy system, it can be helpful. Yeah, so in the design, you always just want to reduce the amount of lookups ever. So every operation that you can potentially do on the database should be cacheable and you shouldn't ever need to be doing any types of operations that require lookups. So like we had mentioned before when we insert into the database, it can produce 409s that the client just needs to deal with and that's because we don't want to have, like we want to optimize for the normal case instead of the edge cases because that's what most of our requests are and then the client can have some of the logic to deal with those edge cases. Yeah, I guess I kind of already said that, yeah. Okay, so one of the cool things that Async.io has is the option to have tasks that run in parallel, meaning that they are waiting that there is some input output to be run because we cannot use more than one CPU. And one of the things that we have is request features. So you have a request and you can add tasks or functions or operations or wherever you want that is going to be done after the request is finished. So that helps for example for indexing or you want to call some audit log or do whatever you want. And well it's so easy that you just get the execute global function of Glutina and you do after this request, call this function with these arguments and it will be called after the request has already been delivered to the client. So it's not so that you can really have operations that work after the request has already been finished by the client in Async.io. It's like the pre-post, we also have pre and post commit hooks so you can, yeah, we are also romantic on that. So you can do whatever you want before a transaction or before a commit or whatever. But they are done before the request has been delivered to the client. So whatever you do there it's going to make the request a bit longer. Whatever you add on after request it's never going to affect the request time. And then we have another interesting thing that we needed to develop to make this kind of Async.io jobs work better. It's a pool and a queue of tasks. That means that we call something Async utilities also for romantic names. Utilities are single items that are running in memory that does whatever you want. And we, Glutina out of the box delivers us with a queue that you can, we can queue tasks there. I don't know. I can't get to these surveys and say that I'm alive every five minutes or call Celery and do whatever or call this queue system and do whatever. And so the queue and the pool it's used, it's like before you just get the pool and you say run this, this can run forever and it's never going to block. Well, it needs to be something that it's Async.io. If it's CPU bound, it will block all the system of course, but you can just execute wherever you want there, short, long, whatever. And this is Async.io utilities that we have. We have a lot. We have for caching, for the indexing, for working pools. We have a lot. There is a lot right now. Yeah. Transaction mechanism. So as Nathan said before, we don't want to access the database because accessing the database is slow. So we are trying to demure it as much as possible, accessing the database. So we have a traversal system that goes through all the objects. If they are in cache, we don't need to access the database. We go through the security. We validate whatever JSON web token you have. If we don't need to access any service or you need to access a service, you access it. And then we look for the view. There is some other things on the traversal. But you look for the view. We start the transaction at the moment that we are going to execute the view. And then we execute whatever is inside and we close as soon as possible when we finish the rendering. So we try always to demure it as much as possible. And we also want to try to be as smart as possible with read-only and write transactions. And we are using something that we are really proud is the get, the HTTP verbs. So we have get, hit, pause, delete, patch. So why don't we use these verbs to define on advance that the connection is read-only or is write? So what we do is if you do any get or any get on Glutina, we even don't open writing transaction. We just do a read-only transaction that we are not going to commit. For sure. You cannot write the database on any get or head operation. And we use the pause, the get, add the pause, the patch, and the delete. And then we open transactions. So that helps us because then you can tune that you have, for example, a slave of Postgres that it's only for read operations that all the get go there. And then you avoid overloading the main database. You can do a lot of things if you have this kind of tooling so you can define read operations and write operations. Yeah, another important thing is that we have our concept of what's a transaction that we are reusing on our tasks and we are using now in 3.7 the context bars to store these transactions. But the transaction in Glutina is not a transaction on database. So we start a Glutina transaction at the beginning of the request and we end at the end. But as I said before, we try to make as far as possible, as small as possible the transaction. And if we don't need to, the database, we even don't open the transaction. Yeah, finally, in this block of lifecycle management, kind of what's happening, we have an option that is called xdebug. If you call Glutina and it's enabled with a parameter that is called xdebug, you get really interesting debug headers where you can see the timing of each of the phases that you have on the traversal and the performance of the cache, the memory cache, and how many queries you are doing too as well. It's a really interesting tool too. So when you finish, okay, this call, where am I spending more time? No? Oh, yeah, we also support, we have packages for Prometheus to send metrics to Prometheus and to on a StatsD. So you can deliver all this information to see if there is any problem. Indexing. Okay. We will go faster because we are running out of time. So indexing, it's a really complex problem. And if you want to, we are storing the pickles on the database, so we wanted to make something that is fast, but it's also synchronous. So the first approach that we went is to index things on elastic search. What means is that we have a process that extracts the indexing information and sends to elastic search completely out of the transaction. That makes that the index may be out of sync with the main database. So you may have objects that are not indexed or information on the index that is not on the object, but was a really fast, fast first approach. And it fixed, well, maybe we were losing some information on the index, maybe it was late. For example, now if you try to use elastic search on guillotine and you want to use Volto, you create an object and you go to the folder and you don't see until one second because elastic search needs one second to index that information. So yeah, this is synchronous even from guillotine a point of view, even from elastic search or whatever you're using to communicate with elastic search, it's a problem. So you need to have two jobs that are every some days going through all your data making sure that everything is indexing or indexing all this information. We will explain it before a good solution for this problem with Kafka, but we will explain something if we have time. Then we decided to create also transaction tie at indexing, so it means that there is as we are storing on JSONB on Postgres, that means that it's on the same transaction and if the transaction fails, the information is not stored on the index. It's really goes really fast. It's scalable using JSONB. We have indexes for all the fields that we have used to index. The problem is that the full text search is not as powerful as elastic search and if your needs require elastic search, then you need elastic search. Yeah, so there is API to initialize the indexing, to repopulate the indexing, to delete the indexing. It's not only a container. Container is what we should be a site. In guillotine, it's an index. You can define that any folder is a subindex. You can define multiple subindex for each folder of your site if you want. Yeah, file storage, storing files, it's a bit easier than indexing. We are lucky that there is a lot of services that provide this kind of backend in order to support storing files. We have S3 Google Cloud Storage and Minio that it's on-premise S3 that you can install in your Kubernetes cluster. You can also use database, but as we said before and talked before, it's not recommended to have large volumes of blocked files on Postgres. Oh, yeah, we are building a new driver for guillotine to store also on the file system if it's needed. But what's important is that as it's a syncio, any file coming in and out of guillotine, even because it's coming from the storage to the client or from the client to the storage, it's a stream. It means that we are chunking, wherever if you're sending altogether or with TOS, we are chunking with buffers of 256 or up to four megabytes of bytes, and we are sending this information to the service and get on the other way around. So I remember that in Zoop maybe a long time ago, I needed to download files locally and then return to the client. It's so easy with a syncio to have this buffering of input or output, meanwhile, you're doing other things with other tasks on the same thread. Yeah, it's also important to know that if you're using S3 or go-to-couch storage or Minio, you need to have a vacuuming in case there has been some un-sync information between the storage and guillotine. Yeah, I'm going to go really fast. As is the syncio, queues are really friendly with the syncio, so we support AM2P, mostly designed when you really need to ensure that the task is being done, so you need to have an easy case this is being done. You have Kafka support, that means that you can even log in, use sequence for indexing and being able to have a consumer that indexes that on Elasticsearch or to also assign tasks, more kind of an streaming service, and then you have NATS and STAN that is the best of AM2P and the best of Kafka together. Sorry, I'm using that. Yeah, some other things that we found out in tuning guillotine is, well, here, we don't have to do that all the time, but anyways, be careful with reference cycles, so objects that end up referencing through a chain of references of objects themselves, so it's really hard for Python to garbage collect reference cycles. It takes a lot of garbage collection attempts before it actually gets it out of memory. Also, an asyncio, each task is this object that doesn't get cleaned up immediately, and there's some exception, even if it's like, I don't know, some like a conflict error or whatever, it might be that exception, an exception object usually wraps the local context of what happened in that exception in it, and so that might have references to objects inside of it, and that might not be cleaned up right away, because that task, even though it may have been finished, the object that holds the information about the task hasn't been cleaned up. I don't know the rules behind when, it's like a weak ref, I don't know the rules behind when that actually does get cleaned up in asyncio, but there's like this weak ref dictionary of tasks or something that doesn't always get cleaned up, and that can be a lot of memory sitting around potentially. There's also the built-in profile in Python that's really nice that you should use to know profile like CPU intensive code, and then there's this really nice library called line profiler, and that will give you the full time of every single line of code that you just decide to profile, and so like, you can look at a line, and then that took so many microseconds, and then you can go into like what function that called, and then you can see it all split into like where inside of that function, like you can go, keep going down and see like where the time was spent, and it's actually time, not CPU time, so this will include if you're like connecting to a database and that database is slower or whatever, you can, this can be really helpful to identify slower parts of your code. Yeah, in Guiltinga comes with built-in support for both of them. There's minus minus line profiler, command option and also just profile command option, and then is, oh maybe, oh yeah, so you can kind of see, this is the line profiler output, but if you want to actually profile a function, you have to add that app-profileable decorator, and then once you use this, it will register that this needs to be profiled, and this is the line profiler output, though it tells you line by line how much time was spent on a real line. That's all, thank you. Any questions? Hi, I noticed you turned off the autobacuum on PostgreSQL. Oh, okay, so you're out of the vacuum, not because it's not safe. Okay, you answered my question, thank you. Yeah, because when we delete, we just mark an object as being deleted, and so like, there's another, there's an application vacuuming process where we're actually going through those and doing, going all the way, going in a shallow way and deleting them individually instead of relying on referential integrity to clean them all up, because referential integrity, if it's a substantial amount of objects that need to be cleaned up, can like, take just gigabytes of memory and destroy it. Hi, maybe this is a crazy question, but how much SQL are you really using in guillotine? I mean, if it is just a little part of SQL, for more formats, you can bring more and more using directly or writing and reading from files using the PostgreSQL libraries, for example. Better would be to use a database built with the sync.io directly so it can be plugged in on your application. This would get the best performance, but I don't know if it exists, so I'm asking if you, when you did your research, if something like this can happen or exist. Yeah, I guess at the time, PostgreSQL was really by far the best option. There was this really nice async.pg library that's an async.io driver for PostgreSQL that's really fast. And even when we started this, there weren't, not all databases had async.io drivers. And PostgreSQL is really fast, as long as you use it correctly. It's really fast. And then if you're going to use a database that spoke HTTP, there's like overhead with that and they might be able to scale horizontally, but then regular operations aren't necessarily really fast. And there's no database that's just async.io. In the end, it's still Python. It's not going to be as fast as C or Rust or C++ or Go, but we use Python to do what we like it. And there's other benefits to using Python. So this is fast in Python. We mean Python fast. You can still do a lot with Python. Instagram uses Python and async.io. They're pretty big. It's just a matter of choices and trade-offs. Other question? Comments? By the way, Instagram also uses Postgres. You use buildout, right? No. No, what do you use then? Just pip. I saw Slashbin in example. Slashbin? Yeah. Oh, I just, the virtual environment. Next try. Other question? Okay. All right. Sure, I agree with the Europe lookup prevention schedule because that takes time. The actual issue there is race conditions because first you select then you update. Yeah. So we have a transaction ID that we look up to make sure that it's a safe operation. So that would work in your case specifically, yeah. But there could still be another operation that we'll insert and then you get a 409 because it will be invalid. And then you get another 409 or something. Yeah. So the actual client needs to take care of that anyhow. Yeah. Cool. Yeah, retries. We have built in retries too for database conflicts. So I mean, yeah, like just like ZODB has retries, it's kind of the same way in that respect, but the client still needs to handle it in some cases. So that's what I think. Okay. Thanks. Thank you. Bye. Bye. Bye. Bye. Bye.
 We are going to do a technical presentation of Guillotine.  Well, mostly, yesterday they explained why we created it, and I was saying all the time,  we are going to explain about the SYNC-IO, we are going to explain what was the problems  that we were facing, we were doing the framework two years ago, and this is the talk that we  are going to go through all the different points that we had pain using blown and using  ZODB and using RelaStorage.  So we are going to explain all the decisions we needed to make in order to build a framework  that fulfills our requirements.  First of all, if you don't know, I am Ramon Navarro, I have two companies, is going to  testify, I am a blown foundation member and I am Catalan, and this is the demonstration  in Catalonia last week.  Hi, and I am Nathan Manguim, I work at Ona as principal engineer, I am a blown foundation  member and I don't really play guitar anymore, I just felt I needed that because Ramon always  talks about playing music and I am American.  Normally, I wouldn't say anything about that either, but you know.  So we are just going to try to go topic by topic to talk about some of the different  areas that we used to tweak performance and create a framework for scaling to a lot of  data in Python.  Some of those are, I guess I can repeat them, but you can read them as well, the async I  O and then just the design of the database, caching and life cycle management, indexing,  file storage and queues.  So the first thing we will go over is async I O and I guess I just want to stress that  async I O is really important for Python and the web in general going forward.  The threaded approach, like okay, let's, we will just talk about how async I O is different  than a threaded app.  And a threaded app, when a request, an HTTP request comes in, it is assigned a thread  and that thread is active through the duration of the request and once it finishes, the thread  is returned to another request.  So you are limited to the number of simultaneous connections for the number of threads that  you have on your application.  And Python, that usually means two because Python doesn't perform very well with lots  of threads.  The best way to deploy Python is with two threads per process.  So you can just keep doing more processes, the scale, which is fine, but then you don't  have any memory sharing or anything like that.  But moreover, it is just also, it is the differences between how you block on your application.  So if you have the threaded approach, a request comes in and then you communicate with say  Postgres SQL and maybe it happens to be a slow query and you are blocking for three  seconds to wait for that query to finish.  Now that thread is consumed for that whole three seconds while that query is communicating  that Python is not doing any CPU, it is not doing anything at all, but it is still blocking  and it is preventing other people from being able to have their request serviced.  So that is the threaded model of request per thread and that is Django right now, that  is Flask and Pyramid.  And so think about how this applies to situations where you are communicating with lots of services.  You are sending things to a message queue, you are communicating with the Redis for caching,  you are doing OAuth for authentication or any kind of authentication or authorization  provider, you are doing S3 for file storage, like any of these services, they are all over  like HTTP or some kind of TCP connection and now you are blocking on communication.  If there is anything slow in any of those services, you are blocking and you are not  doing anything.  And so this is where async.io is useful, like it is non-blocking network.io now and you  can communicate with all these other services and while you are communicating there is very  little overhead and maintain that connection and you can really push the performance of  Python where you were not able to before.  And moreover just think about the context of microservices and even if you think microservices  is crap, the reality is you don't have to be microservices, you are still using a lot  of other services.  You are connecting to other APIs in a number of ways, you are at least using like multiple  database layers or like most applications you develop now are not just database only,  you are going to be communicating with something, some API, maybe an authentication service  or something.  So like this is going to be a problem if you have any sort of substantial load.  And to illustrate this even further, I guess I do have an example of some code to show  like how this can be even this is going to be an example of a CPU bound application versus  a network bound application.  And so like one of the, I will back up a little bit because I forgot to explain the downside  of async.io.  So async.io is a single threaded approach to running your application.  So one process, one thread, your event loop which is running the async code is only on  one thread.  So if there is any CPU bound code there, it is going to block on CPU because no other  code can run at the same time if you are executing CPU, it is single threaded.  So it is really important that you do not block on CPU.  Whereas threads, if you have high CPU load, you can switch threads every once in a while  and you will never have a situation where you are blocking all other code from running.  So that is one downside.  So if you know you have CPU bound code, you have to be careful about it.  In async.io there is a way to run CPU bound code in a thread pool.  So you just have to know what you are doing.  But anyways, this is to demonstrate, this is some code that just crunches on numbers  for half a second and we can see how it behaves.  And the regular async.io approach is to request a second and the threaded approach without  setting any threads to request a second.  It behaves the same.  Async.io with the executor, that is what I was talking about how you can throw CPU bound  code in a thread pool that does slightly better than a threaded application with 20 threads,  which you do not really want to do, but that is just to say like you need to try to scale  it further with a single process.  The problem is not that much different with the CPU bound app.  But if you go to a network bound app and this is the same sort of thing, we are just connecting  to a service and that service is just something, just a service that is delaying for half a  second before it returns, which is in async.io service of course.  And then the difference is this.  This is only bound by the number of concurrent requests the library was using per host.  So it could go further.  That was some arbitrary limit they had on number of concurrent connections per host.  And the threaded app is abysmal for the performance.  Yes.  Just to say that async.io is amazing for network, but there is a lot of other input-opened things  that we have on our computers that the async.io is also really helpful.  We are not using a Guillotine specifically because most of the things is kind of a REST  API and it's mostly network communication, but we are using also a lot for, for example,  accessing the disk with long operations that you need to do on the disk.  Or I'm sure that there is a lot of input-opened communications with devices, with the kernel  that you could build drivers for async.io in order to be also on the event loop.  So everything that you need to wait async.io is amazing.  And sub-processing, yeah, you just start a new process.  We have processes that start a lot of different processes and wait for them.  Meanwhile, they are doing other things, serving HTTP so we can monitor the process that  are running.  All this input-opened is so easy with async.io.  And there was another thing, Python delivers with a standard event loop, a basic one that  works really well for most of the cases, but there are other implementations of event loops,  for example, the UV loop, that it's much faster than the standard one, that it's the  same as Node.js.  And there is also the Tokyo event loop, that it's written in Rust, that it's also really  fast.  And the system is so pluggable that if you really need this extra 20% of the speed, you  can even switch the event pool to something that is much more powerful.  Sorry.  So, I think async.io was the main important thing that we needed to come.  So it was the first thing that we decided that we were going to happen.  And the second thing that we decided is that we needed a database, a place to store.  We started storing things on a file system, on folders, and well, it was okay.  But we needed some kind of more strong way of storing with transactions that, so there  is a lot of databases that offers that.  And we went on to understanding how Rela storage works.  So we say Rela storage has been there for a while, it's been growing.  So we want a pickle system, what we can copy.  And we went to check Rela storage.  And we decided, a design, that it's a bit of inspired with Rela storage, also with that  database that Jim built, new DB.  So, Lutinates traversal.  So there is main nodes that you have on the tree.  And these main nodes are registries on a table.  Which this makes easy to be able to do a select on the table and to do a dump and to get all  that objects out of it.  And we decided that we don't have enough with a row, with a byte field on Postgres to store  all the information that we want from an object, because an object maybe can grow up a lot and  we need to include extreme information.  So we copied the concept of annotation from Plume.  And we decided to have annotation registries that links to the node object.  So if you do a select on a Glutinate database, you will see rows that are the node objects,  the basic nodes on the tree, and nodes that are annotations from to these node objects.  Then of course we are a tree.  So on this structure, we needed to define a way of defining pointers even to the children  or to the parent.  In our case, we decided to point to the parent.  So each row has a pointer to the parent or to the node that it's an annotation for.  Why we did that?  Because of course it's much easier to avoid to check conflicts because you are only pointing  to one.  Yeah.  At the beginning when we started to do the design, we did not use the JSONB indexing  from Postgres.  So we decided that by then we are going only to store the pickle with a byte field on Postgres  and then indexing it's for another system.  We are microservices, so Glutinates only taking care of storing with transactions the main  objects and the notation objects.  So well, these are some of the configurations on Postgres that we have on right now.  Mostly it's a simple one table, one big table with all the information.  So it's database that's mounted on a point of the tree.  It's that table and you can have multiple tables and mount on different endpoints on  Postgres and on the URLs of Glutinates.  Then there is a lot of foreign keys that make sure that you cannot delete, delete the children,  all the parent, all the children gets deleted.  If for example you want to do a post with the PlongRest API to create an object and you  define which is the ID.  Even if that ID exists, it's a simple operation because there is a primary key that doesn't  allow to have different IDs for the same parent.  So the database itself provides consistency.  But related to that, you don't know until you try to commit if there's a conflict in  this way because we didn't want to, as we'll go through more, you need to cut out all possible  places where you're going to do a lookup.  So if on, like when you're doing a post to create a new object and you have the ID and  then you, at that moment you check to say, is there an ID with this already exist on  the database, that's always going to be a lookup to the database.  You always need a connection to the database in order to do that.  So we were like, no, we'll just wait until we actually succeed and let the database do  the constraint check for us.  If the database gives a conflict error, then we kick it back out and it's a conflict error  and the client can handle the 409.  We're a REST API, so they should know how to handle 409s and that's kind of how it's  handled.  Thank you.  Great.  So another thing that is important that we use a lot and we are really thankful that  Postgres can give us to go fast is the sequence number.  So the transactions ID is quite, it's really important that our sequential and Postgres  itself offers us a sequential ID that allows us to have a sequence number for doing that.  There is two things that this architecture makes, there is a lot of things that makes  this a bit complex but two major ones.  One is that we lost the history, the ZODB kind of a pen only that you are storing all  the information, all the transaction, so you can go on time and decide how far you can  go.  So we have a snapshot of actual situation.  We can implement versioning on top of the objects by the data model itself but the transactions,  we support what Postgres supports transactions and we have the snapshot of the database.  Of course, as the previous talk explained it, you define a specific snapshot on time  on the database then you can go back if you don't remove the history.  Another thing that is difficult is the relation of children and parents.  The children pointing to the parent, ordering is really complex because we really need to,  you cannot store the order on the children, you need to store the order in the parent  or you can store the list or if you store the order on the children then you need to  index that properly.  So if you guys are familiar with how this is done in Plone, ordering is like they are  storing a complete list of the order of the folder on the parent.  So every single time you add an object or you change the order, it is modifying the  parent object.  We don't do ordering.  The Guilty in the CMS thing does ordering and it is just studying an attribute and so  it is like a fuzzy on pseudo enforced ordering.  So you are still only writing to the child and that is the safest way to do it where  you can actually not have a huge performance implications.  There is more on writing.  There is a lot of things.  So just to see an image is always after explaining all these things, an image helps a lot.  So this is an SQL from a table from Glutina.  You see that there are two specific registries, the DDDDD and the 0000.  The 0000 is the root object.  So it is the first object.  And DDDDD is the trash.  So when we are deleting something, we are making the pointer of the parent of that element  to point to DDDDD.  Why?  Because we discovered that the reference integrity for key deletion trigger when we  were deleting kind of folders with millions of objects were kind of shutting down the  database.  So we decided to do the deletion on batch operations.  So we point the object to the DDDDD and then there is a batch job that cleans up everything.  You will see during all some of the slides that we talk about vacuuming or batch jobs  to clean up.  We need to do a lot of these things.  We cannot do everything in real time because we want to answer fast and we want to provide  integrity.  Yeah.  So like if you do a delete of a really large folder in blown, it might take like 30 minutes  or something like that because it is doing all of that in one single transaction, making  sure that transaction is synchronized across all the other transactions that are happening  and then hopefully it actually works.  So all we do is set this attribute and all of our cleanup re-indexing, all that stuff  is in an async task or in a queue and that is how we manage doing large operations that  affect large, a large amount of data at a time.  Yeah.  So some other differences with Ralla storage is that of course we have the object ID, it  is the ZOID.  I am a romantic guy.  I choose to name it ZOID for romantic reasons.  You may understand.  The transaction ID, this is the sequential number.  State size is also romantic name to store the size of the byte.  Part.  Part is a field that you can create an adapter so each object can define in which partition  is stored.  This is used for what has been explained also in the talk before that Postgres 11 supports  partitioning at Postgres 10 also, 11, no?  Supports partitioning so you automatically do a partition, you can partition your database  table with multiple sub-tables and using this attribute to define in which partition you  want to store it.  Of course, just to know if it is a main object or it is just an rotation, in case you are  an rotation, the transaction ID is for the cycle of commit, the two-phase commit.  Parent ID is the pointer to the parent.  The ID is the real ID and this is different, for example, from, I think, on Ralla storage  or maybe it is the type that you have the real ID that appears on the URL.  Then the type, the real type from Glutina that we have, JSON is the serialization that  we want to use for indexing in case we use Postgres catalog and a state that is the Pico.  One other thing about the ZOID is you can see from row 3 to 6, all of the ZOID keys  start with 824.  If you are using something like Cockroach, you have CockroachDB support as well.  In order for it to distribute the data, it does key sorting and you want to have your  data near each other if you are querying it.  If you didn't have this sort of key storing which allowed Cockroach to put chunks of data  on the same node, it would be having to do the queries between CockroachDB nodes and  synchronize those queries and it can be really slow.  That is the reason why the ZOIDs are structured that way.  Yeah, the three digits are the parent first three digits and each children gets the three  digits from the parent also besides its own ZOID.  Everything gets ordered on CockroachDB.  We already explained a bit about that.  Deletion is complex.  Synchronize, making sure that you have a clean system, it's complex.  Right now we have two systems of auto-baccuying.  My default Guillotine tries to create an AsyncIO job to clean the database.  But we also have an option to disable that so you have an external job that does it manually.  Okay, no catching.  Yeah, so we use Redis to facilitate our caching and then an in-memory cache.  I mean, I guess in that respect it's a lot like the way ZODB caches objects.  We just have an LRU cache that's pretty fast and written in C.  We have cache keys for all operations that are done on the database.  We set the in-memory cache and then we use Redis to coordinate and validate between Guillotine  application servers.  Yeah, so that's kind of it.  Optionally, it can also push cache out to other application servers as well.  Depending on your load, that can be actually a negative performance because that can put  a lot of pressure on Redis to push out all that data if you have a lot of rights going  on.  So how I usually deploy or how we deploy, I'm sorry, is we just use Redis for invalidating  cache across the application servers and everything else is just straight in-memory  cache then and we don't push out any new values with Redis.  Yeah, so the L2 cache would be a reference to the Redis actually storing cache of the  pickles.  So I don't use that, but it is supported and if you don't have a right heavy system,  it can be helpful.  Yeah, so in the design, you always just want to reduce the amount of lookups ever.  So every operation that you can potentially do on the database should be cacheable and  you shouldn't ever need to be doing any types of operations that require lookups.  So like we had mentioned before when we insert into the database, it can produce 409s that  the client just needs to deal with and that's because we don't want to have, like we want  to optimize for the normal case instead of the edge cases because that's what most of  our requests are and then the client can have some of the logic to deal with those edge  cases.  Yeah, I guess I kind of already said that, yeah.  Okay, so one of the cool things that Async.io has is the option to have tasks that run in  parallel, meaning that they are waiting that there is some input output to be run because  we cannot use more than one CPU.  And one of the things that we have is request features.  So you have a request and you can add tasks or functions or operations or wherever you  want that is going to be done after the request is finished.  So that helps for example for indexing or you want to call some audit log or do whatever  you want.  And well it's so easy that you just get the execute global function of Glutina and you  do after this request, call this function with these arguments and it will be called  after the request has already been delivered to the client.  So it's not so that you can really have operations that work after the request has already been  finished by the client in Async.io.  It's like the pre-post, we also have pre and post commit hooks so you can, yeah, we are  also romantic on that.  So you can do whatever you want before a transaction or before a commit or whatever.  But they are done before the request has been delivered to the client.  So whatever you do there it's going to make the request a bit longer.  Whatever you add on after request it's never going to affect the request time.  And then we have another interesting thing that we needed to develop to make this kind  of Async.io jobs work better.  It's a pool and a queue of tasks.  That means that we call something Async utilities also for romantic names.  Utilities are single items that are running in memory that does whatever you want.  And we, Glutina out of the box delivers us with a queue that you can, we can queue tasks  there.  I don't know.  I can't get to these surveys and say that I'm alive every five minutes or call Celery  and do whatever or call this queue system and do whatever.  And so the queue and the pool it's used, it's like before you just get the pool and you  say run this, this can run forever and it's never going to block.  Well, it needs to be something that it's Async.io.  If it's CPU bound, it will block all the system of course, but you can just execute  wherever you want there, short, long, whatever.  And this is Async.io utilities that we have.  We have a lot.  We have for caching, for the indexing, for working pools.  We have a lot.  There is a lot right now.  Yeah.  Transaction mechanism.  So as Nathan said before, we don't want to access the database because accessing the  database is slow.  So we are trying to demure it as much as possible, accessing the database.  So we have a traversal system that goes through all the objects.  If they are in cache, we don't need to access the database.  We go through the security.  We validate whatever JSON web token you have.  If we don't need to access any service or you need to access a service, you access it.  And then we look for the view.  There is some other things on the traversal.  But you look for the view.  We start the transaction at the moment that we are going to execute the view.  And then we execute whatever is inside and we close as soon as possible when we finish  the rendering.  So we try always to demure it as much as possible.  And we also want to try to be as smart as possible with read-only and write transactions.  And we are using something that we are really proud is the get, the HTTP verbs.  So we have get, hit, pause, delete, patch.  So why don't we use these verbs to define on advance that the connection is read-only  or is write?  So what we do is if you do any get or any get on Glutina, we even don't open writing  transaction.  We just do a read-only transaction that we are not going to commit.  For sure.  You cannot write the database on any get or head operation.  And we use the pause, the get, add the pause, the patch, and the delete.  And then we open transactions.  So that helps us because then you can tune that you have, for example, a slave of Postgres  that it's only for read operations that all the get go there.  And then you avoid overloading the main database.  You can do a lot of things if you have this kind of tooling so you can define read operations  and write operations.  Yeah, another important thing is that we have our concept of what's a transaction that we  are reusing on our tasks and we are using now in 3.7 the context bars to store these  transactions.  But the transaction in Glutina is not a transaction on database.  So we start a Glutina transaction at the beginning of the request and we end at the end.  But as I said before, we try to make as far as possible, as small as possible the transaction.  And if we don't need to, the database, we even don't open the transaction.  Yeah, finally, in this block of lifecycle management, kind of what's happening, we have  an option that is called xdebug.  If you call Glutina and it's enabled with a parameter that is called xdebug, you get  really interesting debug headers where you can see the timing of each of the phases that  you have on the traversal and the performance of the cache, the memory cache, and how many  queries you are doing too as well.  It's a really interesting tool too.  So when you finish, okay, this call, where am I spending more time?  No?  Oh, yeah, we also support, we have packages for Prometheus to send metrics to Prometheus  and to on a StatsD.  So you can deliver all this information to see if there is any problem.  Indexing.  Okay.  We will go faster because we are running out of time.  So indexing, it's a really complex problem.  And if you want to, we are storing the pickles on the database, so we wanted to make something  that is fast, but it's also synchronous.  So the first approach that we went is to index things on elastic search.  What means is that we have a process that extracts the indexing information and sends  to elastic search completely out of the transaction.  That makes that the index may be out of sync with the main database.  So you may have objects that are not indexed or information on the index that is not on  the object, but was a really fast, fast first approach.  And it fixed, well, maybe we were losing some information on the index, maybe it was late.  For example, now if you try to use elastic search on guillotine and you want to use Volto,  you create an object and you go to the folder and you don't see until one second because  elastic search needs one second to index that information.  So yeah, this is synchronous even from guillotine a point of view, even from elastic search or  whatever you're using to communicate with elastic search, it's a problem.  So you need to have two jobs that are every some days going through all your data making  sure that everything is indexing or indexing all this information.  We will explain it before a good solution for this problem with Kafka, but we will explain  something if we have time.  Then we decided to create also transaction tie at indexing, so it means that there is  as we are storing on JSONB on Postgres, that means that it's on the same transaction and  if the transaction fails, the information is not stored on the index.  It's really goes really fast.  It's scalable using JSONB.  We have indexes for all the fields that we have used to index.  The problem is that the full text search is not as powerful as elastic search and if your  needs require elastic search, then you need elastic search.  Yeah, so there is API to initialize the indexing, to repopulate the indexing, to delete the  indexing.  It's not only a container.  Container is what we should be a site.  In guillotine, it's an index.  You can define that any folder is a subindex.  You can define multiple subindex for each folder of your site if you want.  Yeah, file storage, storing files, it's a bit easier than indexing.  We are lucky that there is a lot of services that provide this kind of backend in order  to support storing files.  We have S3 Google Cloud Storage and Minio that it's on-premise S3 that you can install  in your Kubernetes cluster.  You can also use database, but as we said before and talked before, it's not recommended  to have large volumes of blocked files on Postgres.  Oh, yeah, we are building a new driver for guillotine to store also on the file system  if it's needed.  But what's important is that as it's a syncio, any file coming in and out of guillotine, even  because it's coming from the storage to the client or from the client to the storage,  it's a stream.  It means that we are chunking, wherever if you're sending altogether or with TOS, we  are chunking with buffers of 256 or up to four megabytes of bytes, and we are sending  this information to the service and get on the other way around.  So I remember that in Zoop maybe a long time ago, I needed to download files locally and  then return to the client.  It's so easy with a syncio to have this buffering of input or output, meanwhile, you're doing  other things with other tasks on the same thread.  Yeah, it's also important to know that if you're using S3 or go-to-couch storage or  Minio, you need to have a vacuuming in case there has been some un-sync information between  the storage and guillotine.  Yeah, I'm going to go really fast.  As is the syncio, queues are really friendly with the syncio, so we support AM2P, mostly  designed when you really need to ensure that the task is being done, so you need to have  an easy case this is being done.  You have Kafka support, that means that you can even log in, use sequence for indexing  and being able to have a consumer that indexes that on Elasticsearch or to also assign tasks,  more kind of an streaming service, and then you have NATS and STAN that is the best of  AM2P and the best of Kafka together.  Sorry, I'm using that.  Yeah, some other things that we found out in tuning guillotine is, well, here, we don't  have to do that all the time, but anyways, be careful with reference cycles, so objects  that end up referencing through a chain of references of objects themselves, so it's  really hard for Python to garbage collect reference cycles.  It takes a lot of garbage collection attempts before it actually gets it out of memory.  Also, an asyncio, each task is this object that doesn't get cleaned up immediately, and  there's some exception, even if it's like, I don't know, some like a conflict error or  whatever, it might be that exception, an exception object usually wraps the local context of  what happened in that exception in it, and so that might have references to objects inside  of it, and that might not be cleaned up right away, because that task, even though it may  have been finished, the object that holds the information about the task hasn't been  cleaned up.  I don't know the rules behind when, it's like a weak ref, I don't know the rules behind  when that actually does get cleaned up in asyncio, but there's like this weak ref dictionary  of tasks or something that doesn't always get cleaned up, and that can be a lot of memory  sitting around potentially.  There's also the built-in profile in Python that's really nice that you should use to  know profile like CPU intensive code, and then there's this really nice library called  line profiler, and that will give you the full time of every single line of code that  you just decide to profile, and so like, you can look at a line, and then that took so  many microseconds, and then you can go into like what function that called, and then you  can see it all split into like where inside of that function, like you can go, keep going  down and see like where the time was spent, and it's actually time, not CPU time, so this  will include if you're like connecting to a database and that database is slower or  whatever, you can, this can be really helpful to identify slower parts of your code.  Yeah, in Guiltinga comes with built-in support for both of them.  There's minus minus line profiler, command option and also just profile command option,  and then is, oh maybe, oh yeah, so you can kind of see, this is the line profiler output,  but if you want to actually profile a function, you have to add that app-profileable decorator,  and then once you use this, it will register that this needs to be profiled, and this is  the line profiler output, though it tells you line by line how much time was spent on  a real line.  That's all, thank you.  Any questions?  Hi, I noticed you turned off the autobacuum on PostgreSQL.  Oh, okay, so you're out of the vacuum, not because it's not safe.  Okay, you answered my question, thank you.  Yeah, because when we delete, we just mark an object as being deleted, and so like,  there's another, there's an application vacuuming process where we're actually going through  those and doing, going all the way, going in a shallow way and deleting them individually  instead of relying on referential integrity to clean them all up, because referential  integrity, if it's a substantial amount of objects that need to be cleaned up, can like,  take just gigabytes of memory and destroy it.  Hi, maybe this is a crazy question, but how much SQL are you really using in guillotine?  I mean, if it is just a little part of SQL, for more formats, you can bring more and more  using directly or writing and reading from files using the PostgreSQL libraries, for example.  Better would be to use a database built with the sync.io directly so it can be plugged in  on your application.  This would get the best performance, but I don't know if it exists, so I'm asking if  you, when you did your research, if something like this can happen or exist.  Yeah, I guess at the time, PostgreSQL was really by far the best option.  There was this really nice async.pg library that's an async.io driver for PostgreSQL that's  really fast.  And even when we started this, there weren't, not all databases had async.io drivers.  And PostgreSQL is really fast, as long as you use it correctly.  It's really fast.  And then if you're going to use a database that spoke HTTP, there's like overhead with  that and they might be able to scale horizontally, but then regular operations aren't necessarily  really fast.  And there's no database that's just async.io.  In the end, it's still Python.  It's not going to be as fast as C or Rust or C++ or Go, but we use Python to do what we  like it.  And there's other benefits to using Python.  So this is fast in Python.  We mean Python fast.  You can still do a lot with Python.  Instagram uses Python and async.io.  They're pretty big.  It's just a matter of choices and trade-offs.  Other question?  Comments?  By the way, Instagram also uses Postgres.  You use buildout, right?  No.  No, what do you use then?  Just pip.  I saw Slashbin in example.  Slashbin?  Yeah.  Oh, I just, the virtual environment.  Next try.  Other question?  Okay.  All right.  Sure, I agree with the Europe lookup prevention schedule because that takes time.  The actual issue there is race conditions because first you select then you update.  Yeah.  So we have a transaction ID that we look up to make sure that it's a safe operation.  So that would work in your case specifically, yeah.  But there could still be another operation that we'll insert and then you get a 409 because  it will be invalid.  And then you get another 409 or something.  Yeah.  So the actual client needs to take care of that anyhow.  Yeah.  Cool.  Yeah, retries.  We have built in retries too for database conflicts.  So I mean, yeah, like just like ZODB has retries, it's kind of the same way in that respect,  but the client still needs to handle it in some cases.  So that's what I think.  Okay.  Thanks.  Thank you.  Bye.  Bye.  Bye.  Bye.  Bye.
Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_0_47.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_48_95.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_96_155.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_156_215.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_216_263.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_264_311.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_312_371.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_372_419.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_420_587.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_588_659.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_660_1499.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_1500_2459.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_2460_3251.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_3252_8507.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_8508_10607.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_10608_11963.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_11964_12599.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_12600_20495.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_20496_34175.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_34176_35003.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_35004_35567.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_35568_36791.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_36792_37007.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_37008_47855.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_47856_48251.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_48252_49079.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_49080_49403.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_49404_52211.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_52212_52463.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_52464_52991.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_52992_53111.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_53112_53831.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_53832_56711.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_56712_57191.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_57192_57575.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_57576_57779.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_57780_61583.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_61584_62291.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_62292_62351.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_62352_62399.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_62400_62459.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_62460_62507.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_62508_62627.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_62628_62675.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_62676_62723.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_62724_62831.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_62832_62879.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_62880_62927.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_62928_63023.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_63024_63071.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_63072_63119.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_63120_63239.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_63240_63323.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_63324_63395.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_63396_63731.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_63732_63983.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_63984_64031.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_64032_64127.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_64128_64499.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_64500_67919.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_67920_68279.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_68280_69599.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_69600_69647.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_69648_69695.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_69696_69755.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_69756_69851.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_69852_69899.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_69900_69947.png Ramon_Navarro_Bosch_and_Nathan_Van_Gheem_-_Perform_buFNeUowUDo_69948_70002.png