Yet another package for multi-tenancy in Django
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Peña, Lorenzo
Django is a popular, solid web framework for perfectionists with deadlines, with a wide ecosystem of packages that extend its powers in multiple directions. In the era of peaking popularity of Node/Deno, microservices, and heavyweight browser rendered webapps, Django still remains a triumphant monolith maker, very capable of major undertakings in the web arena. For all you Django-lovers out there, it seems to me like a matter of time before you have to do some form of multi-tenancy in Django. Taking a solution that works well for one tenant and extending it to multiple tenants should still be a problem for perfectionists with deadlines. Interestingly, when it comes to covering all the many facets of multi-tenancy, Django can be not so batteries included, as one might end up working around or 'hacking' the framework in order to get things done. In this talk I will walk you through the challenges of bringing multi-tenancy to a Django project. We'll cover the fundamental plumbing required to make it work reliably, securely, and elegantly. You will be expected to have a basic knowledge of Django (models, settings, users, URL reversing), and you will learn the working logic behind popular multi-tenancy packages.
 Okay, so Lorenzo is joining us from Cuba. This is your first aeropitone, Lorenzo? Yeah, my first aeropitone ever. Cool, welcome. It's also for me. It's also my first aeropitone. It's not my first Python conference, but it's my first aeropitone. So thank you very much for presenting. A reminder for everyone, you can ask questions in the Q&A, and I will ask those questions after the talk. This talk is 30 minutes. So Lorenzo, all yours. You're ready to go. Well thank you so much, and good afternoon or good night, good morning. Thank you for joining me today in this journey to discover yet another package for multi-tenancy in Django. My name is Lorenzo Peña. That's my handle where you can find me pretty much everywhere. I have been a software developer for over a decade, and I have around 11 years of experience in Django with around three packages in the PIP registry. So I hope you're having a great time, and this online version of EuroPython 2020, and I hope as long as my connection behaves properly, that you're going to enjoy this talk as well. So yet another package for multi-tenancy in Django. Let's play this title kind of backwards, and let's begin with Django. And firstly, let's get all on the same page in that 2020 has gotten a little bit out of control. Yeah. And with the rise of GPT-3, future is uncertain in some aspects. You know, probably we're thinking whether we're going to be keeping our jobs by the end of the year, but I honestly think we don't have to be worried too much, worried about it, because we are Pythonistas and Django-NOTs, and we are in the world of Django, the web framework for perfectionists with deadlines. We just turned 15 years old, which is a great milestone, is mature, solid, and battle tested. It has an amazing community and a great momentum, and is getting stronger than ever with more async and reactivity as of late. So just in case you're wondering whether it's still a valid investment to be thinking in Django and multi-tenancy as of 2020, I think it is. And I think Django can perfectly handle the rest of the year and the upcoming decade and provide for you and for me, and we're actually rooting for it. So multi-tenancy. What is exactly multi-tenancy? Let's play by an example. Supposedly it's a customer red, which has a problem, and you develop a solution for that customer. You deploy the solution, it's working great, but now customers blue, green, and yellow have exactly the same problem. So what to do? You have a solution that is already working for the customer red. There are two things you could do. You could just copy and paste that solution and provide multiple single tenant solutions, or you could make the jump into a multi-tenant solution, which is single instance of the software that can provide for the needs of all your customers. That's exactly what multi-tenancy is. It's the software architecture in which a single instance of the software serves multiple tenants. And as an example, you can see Dropbox, Shopify, Slack, and WordPress. So what are tenants? Well, tenants are the isolated spaces in which users with specific privileges interact. There is the accounts of Dropbox, work spaces at Slack, blogs in WordPress, servers at Discord, or Shopify, or sites slash communities as stack exchange. So all shaping good so far. Now, how do we actually get to implement multi-tenancy in Django? One does not simply add it to the project. Why? Because if you want to implement it from scratch, there's a number of things to do and do right. And even if you're going to use any of the existing packages out there, it takes some knowledge to properly decide and determine which one is going to be suitable for your needs. So about this thing of packages, will you actually end up needing a package for multi-tenancy? Well, most likely, yes. And there are many of them. I made one of those. I contributed towards the entropy, which is kind of a fork, actually, with some conceptual changes. But the truth is that there is no one size fits all. Packages tend to be very opinionated in a number of architectural decisions that need to be taken. And so there is no silver bullet for us when selecting packages for multi-tenancy. So am I really going to give you yet another package for multi-tenancy in Django? Well, not exactly. Instead, we're going to take a look at the building blocks, the pieces that form the foundation of multi-tenancy in Django itself. And instead of taking a package-first approach, we're going to be taking a knowledge-first approach. We're going to be pretending that we are implementing multi-tenancy from scratch without actually doing. And I hope with this knowledge, you will be able to select, understand, debug, tweak, contribute back to just any existing package. So this is not just yet another package. This is the package to rule every other package. And if you think of it, it could actually be the ultimate package. So my dear audience, I give you the ultimate package for multi-tenancy in Django. I hope you're ready to get your hands happily dirty in the concepts and notions we are about to cover right away. But the first of these notions is the concept of the active tenant. Suppose we are inside Stack Exchange in some of the internal sites, and we run this query, question objects all. Okay, but where are we expecting to get questions from? Because Stack Exchange is too big. It could be Stack Overflow. It could be ServerFault, SuperUser, maybe Area 51. So welcome to this new concept, the active tenant. And the idea is that one tenant has to be the active tenant, and the framework needs to be aware that it's going to be operating in this scope, even in database access, you're reversing admin side, cache, pretty much every part of the framework. But notice that this has to be in place even outside the request response cycle, because we have things here like management commands and salary tasks, where you don't have a request object to interrogate for the active tenant. Django has a couple of APIs currently that you're probably familiar with, where there is this notion of the active something. It's the time zone and language, and you probably know these couple of functions get current and activate where you don't have to be using a request at all. So we could take some inspiration in this and actually create a couple of functions, get current tenant and activate tenant so that we are able to enter the scope of a tenant by activating it and then retrieving the active tenant further on. This is a possible implementation of the active tenant. This we're using here as QVref local, which is a drop in replacement from trading locals, but without getting too deep, this is actually a global variable that is thread safe, but is still a global variable. And the use of globals is generally discourage. Why? Well, there is a reason why this pattern is actually so scar in Django code base itself. And the reason is that the more you depend on globals, the more couple becomes your code, the harder to test in isolation, and therefore it's not super recommended and Django hasn't followed this pattern except when it's absolutely necessary. So in this case, I would consider that this would be a perfectly valid case for this, but it would be a pattern that is thrown upon otherwise, so please don't go out of this type just creating globals for the sake of it. So there are two important questions about the active tenant and those are what is the type of a tenant object? When we get and set a tenant as the active tenant, what type of object we are working with? And the other question is what happens if for some operation, there is just no active tenant? Is that a bug? Is that a possible situation? Is that a wild card scenario in which you are actually hoping to run the operation in multiple tenants? So these are all valid questions, but it's more of foot for thought. So as we get settled this concept of the active tenant, there are three architectural choices that we need to make, and these three architectural choices are users and tenants, database architecture and tenant routing. Regarding users and tenants, there are three types of relationships between these. One is the type in which users exist outside the context of tenants. That is, you can have tenants, you can have users, there could be relationships between them, but this relationship is kind of loose, non-strict. An example, WordPress, Shopify and Discord. Third type is the one in which users are bound and constrained within the scope and the context of a tenant. The perfect example is Slack, where you cannot think of a user outside the concept of a workspace. And there is a third type in which users and tenants are pretty much the same thing. And this is an example of this is Gmail Dropbox and the like. So which one to pick will completely depend on your use case and how do you expect your users and tenants to be interacting? The baseline question here is how many tenants do you expect a user to be related with? The second architectural choice to be making is the database architecture itself. And there are typically three approaches. One is the isolated approach in which you have multiple databases, one for tenant. The other one is the shared database approach in which you have a single database and a tenant column and on entry-level tables. And the third one, which is kind of reason, is the one in which you have one database, but you use Postgres schemas to do the tenant, to store your tenant information. So how would that be if you were to implement multi-tenancy with isolated databases? Well, Django is compatible with multi-database configurations, so there's nothing stopping you from having multiple databases there. The only thing you would have to do is that you would have to be switching between databases when accessing and storing your data. Here I am using a translation function from a tenant, from an active tenant, into a database alias. And everything you would have to do is just using that alias for saving your objects or for filtering your queries or even for generating already scope managers so that all your subsequent queries are already in the context of that database alias. This could be somehow offloaded to database routers since we have this function to get the current tenant and we also have a translation function from a tenant into a database alias. We could be providing default values here so that we don't have to do the previous thing. The router makes it so that whenever you do a data access, this would be default value of a database. And you could still resort to doing manual specific scoping if you need to override the default behavior. The good thing about the isolated databases is that you are optimized for isolation. The bad is that there are no relationships across databases because Django doesn't allow it and that adding tenants requires reconfiguring the project. Why? Because your tenant catalog is actually living in your settings so every time you're adding a new tenant, you have to update your settings. The not so funny thing about the isolated databases approach is that as you scale, it's going to become quite expensive with operational costs. So unless you're planning to have a number of tenants in the lower 10s or unless you're planning to have Scrooge, Madoc as your billing manager, this is generally not recommended approach. As for the shared database approach, it's the one in which remember we have mixed records in a single database. So in this case, entry level 10 specific models will require a pointer to the tenant they belong to and notice that I'm saying tenant specific models because not every model needs to be tenant specific. You could have models that are used to share information across tenants and notice that I'm also saying entry level because models determine relationships between them. So you don't actually need to provide a pointer for each one of those as long as you're able to reach the tenant with a reasonable number of joins. And this is an example of a hypothetical question model following the stock exchange example where you're having a foreign key to the site, which is kind of the tenant in this example. In this case, you still have to rely on the active tenant in order to complete the missing part of your queries because you will have to use that tenant in order to create your records if you haven't provided a tenant agnostic interface to the client and you still have to filter by tenant no matter if you're actually filtering a model that has the tenant pointer right away or if you're doing a number of joins in order to get to the tenant. You could do some of these automatically and you could try to assign the tenant automatically by means of using a default value for the field that holds the tenant which could be a callable and in this case, if appropriate, could be get current tenant or you could have a custom field with a pre-safe hook. If you're using a foreign key, you could even subclass that a field. Finally, you could resort to having a signal on relevant models in order to complete the model prior to saving it. As for querying, well, you could use custom managers and custom queries in order to pre-filter by your active tenant and therefore the subsequent queries account that these are already filtered. There could be some annotations on subqueries where these customization probably doesn't work so you still have to be open to be doing manual scoping from time to time. The good thing about shared databases is that you are optimized for scalability because adding tenants is just a matter of adding rows into your tables. The bad is that your data isolation will take extra effort on the development side and the not so funny thing about the shared databases is that it's very easy to just forget to scope any specific query by the tenant. Therefore, if you don't want to wake up in the middle of the night with your brain wondering whether you have filtered by the active tenant or not, my recommendation is that you bookmark all your tenant scope queries that you make automated tests for each one of them and you make sure that they are returning res holds in the scope of the active tenant and even that you take a step further and you make an automated test to test that you tested each one of them. So it's kind of a riddle but it's going to save you at some point. Remember that tests are the softest pillow you could have in general for software development but specifically if you're going to share database approach for multi-tenancy. As for semi-isolated databases approach, this is going to rely in possible schemas in order to isolate the tenant within a single database. So what are schemas? Well, this is a specific Postgres concept. They are a layer of abstraction between databases and tables. They can be thought of as name spaces but the good thing about these name spaces is that they are not mutually exclusive. So you can organize them on fine tables in many of them by means of properly configuring your search path which is also another Postgres specific concept. In this case, your queries remain practically unchanged. It seems like magic but you're going to pay the price with an increased technical challenge somewhere else. And this is where you will have to do two major things or it's required two major things in order to get this approach working. First, we're going to require a custom database backend based in the Postgres backend or any other backend that is Postgres friendly. And this backend will have to be able to convert the active tenant into a sequence of schemas and then you will have to run a query to activate those schemas by means of setting the search path. The other interesting challenge and this is kind of the biggest challenge is that we have to teach Django to do migrations again. Why? Because Django doesn't know anything about schemas. Django knows about databases and Django, you can pass a database in order to run your migrations but when it comes to schemas, that is not part of the world view of change. Django itself. So the idea here is that we are also using a database router in order to use the allow migrate hook in order to tell Django whether or not it's legal to migrate a specific model based on the active tenant. And this also requires the migrate command itself to be kind of tweaked because the migrate need to operate at the schema level. The good thing about semi isolated database approach is that you're optimized for isolation and at the same time you have increased scalability because since you have a single database you can scale faster by means of adding schemas. The bad is that it's going to take extra effort to understand and control how schemas interact. The not so funny part about the semi isolated database approach is that since you're able to scale faster, you're going to be hitting the thousands of tenants very soon and therefore thousands of schemas and you still have to run your migrations on each one of those schemas. So you're going to be paying the price with an increased time to run your migration. So please be advised about this. Which one is the best? Well, neither is it. Why? Because there are pros and cons in each one of them and it will depend on your specific use case and each one of those approaches is capable of shining in their specific use case. If you want to engage in a respectful ice cream fight, you can hit me in the breakout and we can continue the debate there. The third architectural choice is tenant routing. That is how do you expect to take an incoming request and generate an active tenant out of it? So it's generally possible to do that because your tenant is somehow encoded in the incoming request by means of the user session or headers or it could be even better if your tenant can be inferred from the URL itself, you could do the translation from the domain, subfolder or query parameter. So where is the perfect place to do this translation from an incoming request into an active tenant? Well, that place is the middleware. It's a perfect place to do it because you're capable of taking the request, the incoming request, just activate the tenant and therefore the rest of your request and respond cycle for that request is going to be guaranteed to be in the scope of that active tenant. This is a possible implementation of that middleware, or one of the middleware. And in this case, this middleware is translating from a session into a tenant. Notice that the major complexity here is actually creating the translation function that I like to take something from the session and convert it into a tenant to activate because the rest is pretty much checking if there is no active tenant already and then activate it. The good part about this is that you only need to provide different translation functions for different retrieval methods and you could have similar functions for users, headers, domain, subfolder parameters and you could still chain one middleware after the other if you want to combine the power of these retrieval methods. So you only have to take care that the order will determine the precedence of those retrieval methods. Now, what if you want the opposite? What if you are already in the scope of a tenant and you just want to generate a canonical URL that is capable to give you, I mean, if you share that URL, it can take you not only to a specific path within your URLs but also to a specific tenant. Well, this is not going to be possible in some cases because, I mean, user sessions and header are not easily encodable as part of the URL but you can totally do that if you are inferring your tenants from domains of folder or query parameter because even though Django only reverses the path part of the URL, you can still prepend, interpolate or append the tenant in order to provide the canonical URL to land in a specific path of a specific tenant. And this, of course, will require a little tweaking in the URL reversing process. As a bonus, I also tell you that as I also give you that it's possible to provide custom URL comes based on the active tenant. So if you have different types of tenants and you want to provide different URLs for those tenants, you could also provide this translation function in order to convert a tenant into a URL comes module Django already provides this hook where you can just assign your outcome module into this variable in the request and you will be using that different module. So these are the three major architectural choices you have to make. And you'll see there are multiple choices for each one of those. Some combinations make sense, play nice together while others don't make much sense at all. So it's going to be up to you to determine whether or not things are playing nice for you. But there is more to Django than just those three parts and those three architectural choices. So let's take a quick look at the scope of everything else. And while this list is not going to be comprehensive, let's please at least see one of the five major places where this dynamic of the active tenant is going to be an important thing. So management commands, it's going to be very valuable to be able to run management commands in the scope of a specific tenant. And for new management commands, you could just include a tenant argument so that this command is capable of first activating the tenant and then performing the operation that the management command is expected to do. Now for existing non-tenant award commands, you will have to, I mean, these commands are basically everything that chips with Django and third party applications. So for this case, you will have to define a special command wrapper that basically takes the tenant as an argument, but also a new command to be running. And this wrapper should be in charge of activating the tenant and then calling the inner command. You will find some packages existing out there that even take a step further in elegance and combine the arguments of both commands. But this is kind of trickier to implement, but it's possible, completely possible to do. As for file storage, it's also going to be extremely valuable to be able to somehow organize the files by tenant. And in this case, you can totally define a custom file storage in order to perhaps prepend your URLs with a string representation of your tenant, et cetera. Notice that I'm not providing any code example here because there are multiple types of file storage depending on your backend. It could be file system or it could be something in the cloud. So this actually takes a little bit of more specific use case thinking. Now in higher security contexts where you cannot afford one tenant accessing files in a large tenant, there are two kind of workarounds you can do in order to increase the security level. One is generating pre-signed URLs so that your static or media URLs are short-lived and therefore it makes it harder to visit tenant-specific URLs for files. And the other one is actually using a proxy view so that the view acts as a middleman and determines whether or not the incoming request has security clearance to access the specific file storage. Again, there are some interesting packages that are doing most part of this themselves so this is more like the underlying concept. As for cache, well, as a buck, the cache that comes with Django is tenant agnostic. So if two tenants are using the same cache key, there will be a clash. But most dangerously, you could be leaking information by means of one tenant storing some data in the cache and then other tenant retrieving that. That's a leak. So one thing you could do is actually generate a special key function that you could then use as part of your cache configuration and that key function, the only things that need to do is just augment the string representation of the key itself with the current tenant. As for salary task, well, it's more a matter of discipline. The idea is that you pass an identifier of your tenant so that the first thing you do in your tenant specific salary task is to activate the tenant and then you can resort to doing the rest of the task itself. And finally, for channels, well, you're going to require a separate middleware for your WebSockets, whatever you do for your regular request, you're going to be doing, you'll have to be doing that also for your middleware, for your channels because they are not fully compatible. And if you want to activate tenants from the incoming scope, you're going to be kind of duplicating your middleware because they are not compatible. And if you're using channel layers, you will have to also do a similar approach like we did with cash with your consumer group's name because if you use a tenant agnostic name, you will also end up leaking messages between the groups of multiple tenants. So for everything else that was not covered, like admin site or template, tenant specific templates, I am certain that the principles here are generally extensible. So I'm almost sure that you will be able to extend the principles in order to cover pretty much everywhere in the framework. Otherwise, feel free to continue the discussion in the breakout. Now some of the packages that actually implement multi-tenancy and help you in integrating multi-tenancy with your project, which you could choose from. Here I'm displaying four packages. Notice that they are kind of opinionated based on the database approach. These packages were taken from the multi-tenancy greed of Django packages.org, which I consider to be the market for Django packages. Please visit there. You could find more interesting things there. Just make sure you see whether or not the package is Python 3 compatible, whether or not it's production ready, and even if it's actively maintained, because otherwise you might end up in a very difficult situation. So if you want to contribute back to any of those packages, the ones I mentioned and the ones I didn't, by means of reporting bugs, implementing new features, or even improving documentation, I will dare to speak on behalf of everybody, of all the package maintainers, and tell you, please come. We need you. You're more than welcome to do so. It could be a gold opportunity for you to put in practice everything you may have learned here or everything you have learned in your own experience and progress through multi-tenancy in Django. So that's it. We can keep in touch for more. And I would like to give a special thanks to Russel Keith McGee, Orlando Williams, and Rafael Michele, who were of a huge help in the preparation of this talk. And lastly, thank you, my dear audience, for joining me today in this journey. Thank you.
 Okay, so Lorenzo is joining us from Cuba.  This is your first aeropitone, Lorenzo?  Yeah, my first aeropitone ever.  Cool, welcome.  It's also for me.  It's also my first aeropitone.  It's not my first Python conference, but it's my first aeropitone.  So thank you very much for presenting.  A reminder for everyone, you can ask questions in the Q&A, and I will ask those questions  after the talk.  This talk is 30 minutes.  So Lorenzo, all yours.  You're ready to go.  Well thank you so much, and good afternoon or good night, good morning.  Thank you for joining me today in this journey to discover yet another package for multi-tenancy  in Django.  My name is Lorenzo Peña.  That's my handle where you can find me pretty much everywhere.  I have been a software developer for over a decade, and I have around 11 years of experience  in Django with around three packages in the PIP registry.  So I hope you're having a great time, and this online version of EuroPython 2020, and  I hope as long as my connection behaves properly, that you're going to enjoy this talk as well.  So yet another package for multi-tenancy in Django.  Let's play this title kind of backwards, and let's begin with Django.  And firstly, let's get all on the same page in that 2020 has gotten a little bit out of  control.  Yeah.  And with the rise of GPT-3, future is uncertain in some aspects.  You know, probably we're thinking whether we're going to be keeping our jobs by the  end of the year, but I honestly think we don't have to be worried too much, worried about  it, because we are Pythonistas and Django-NOTs, and we are in the world of Django, the web  framework for perfectionists with deadlines.  We just turned 15 years old, which is a great milestone, is mature, solid, and battle tested.  It has an amazing community and a great momentum, and is getting stronger than ever with more  async and reactivity as of late.  So just in case you're wondering whether it's still a valid investment to be thinking in  Django and multi-tenancy as of 2020, I think it is.  And I think Django can perfectly handle the rest of the year and the upcoming decade and  provide for you and for me, and we're actually rooting for it.  So multi-tenancy.  What is exactly multi-tenancy?  Let's play by an example.  Supposedly it's a customer red, which has a problem, and you develop a solution for that  customer.  You deploy the solution, it's working great, but now customers blue, green, and yellow  have exactly the same problem.  So what to do?  You have a solution that is already working for the customer red.  There are two things you could do.  You could just copy and paste that solution and provide multiple single tenant solutions,  or you could make the jump into a multi-tenant solution, which is single instance of the  software that can provide for the needs of all your customers.  That's exactly what multi-tenancy is.  It's the software architecture in which a single instance of the software serves multiple  tenants.  And as an example, you can see Dropbox, Shopify, Slack, and WordPress.  So what are tenants?  Well, tenants are the isolated spaces in which users with specific privileges interact.  There is the accounts of Dropbox, work spaces at Slack, blogs in WordPress, servers at Discord,  or Shopify, or sites slash communities as stack exchange.  So all shaping good so far.  Now, how do we actually get to implement multi-tenancy in Django?  One does not simply add it to the project.  Why?  Because if you want to implement it from scratch, there's a number of things to do and do right.  And even if you're going to use any of the existing packages out there, it takes some  knowledge to properly decide and determine which one is going to be suitable for your  needs.  So about this thing of packages, will you actually end up needing a package for multi-tenancy?  Well, most likely, yes.  And there are many of them.  I made one of those.  I contributed towards the entropy, which is kind of a fork, actually, with some conceptual  changes.  But the truth is that there is no one size fits all.  Packages tend to be very opinionated in a number of architectural decisions that need  to be taken.  And so there is no silver bullet for us when selecting packages for multi-tenancy.  So am I really going to give you yet another package for multi-tenancy in Django?  Well, not exactly.  Instead, we're going to take a look at the building blocks, the pieces that form the  foundation of multi-tenancy in Django itself.  And instead of taking a package-first approach, we're going to be taking a knowledge-first  approach.  We're going to be pretending that we are implementing multi-tenancy from scratch without actually  doing.  And I hope with this knowledge, you will be able to select, understand, debug, tweak,  contribute back to just any existing package.  So this is not just yet another package.  This is the package to rule every other package.  And if you think of it, it could actually be the ultimate package.  So my dear audience, I give you the ultimate package for multi-tenancy in Django.  I hope you're ready to get your hands happily dirty in the concepts and notions we are about  to cover right away.  But the first of these notions is the concept of the active tenant.  Suppose we are inside Stack Exchange in some of the internal sites, and we run this query,  question objects all.  Okay, but where are we expecting to get questions from?  Because Stack Exchange is too big.  It could be Stack Overflow.  It could be ServerFault, SuperUser, maybe Area 51.  So welcome to this new concept, the active tenant.  And the idea is that one tenant has to be the active tenant, and the framework needs to be  aware that it's going to be operating in this scope, even in database access, you're  reversing admin side, cache, pretty much every part of the framework.  But notice that this has to be in place even outside the request response cycle, because  we have things here like management commands and salary tasks, where you don't have a request  object to interrogate for the active tenant.  Django has a couple of APIs currently that you're probably familiar with, where there  is this notion of the active something.  It's the time zone and language, and you probably know these couple of functions get current  and activate where you don't have to be using a request at all.  So we could take some inspiration in this and actually create a couple of functions,  get current tenant and activate tenant so that we are able to enter the scope of a tenant  by activating it and then retrieving the active tenant further on.  This is a possible implementation of the active tenant.  This we're using here as QVref local, which is a drop in replacement from trading locals,  but without getting too deep, this is actually a global variable that is thread safe, but  is still a global variable.  And the use of globals is generally discourage.  Why?  Well, there is a reason why this pattern is actually so scar in Django code base itself.  And the reason is that the more you depend on globals, the more couple becomes your code,  the harder to test in isolation, and therefore it's not super recommended and Django hasn't  followed this pattern except when it's absolutely necessary.  So in this case, I would consider that this would be a perfectly valid case for this,  but it would be a pattern that is thrown upon otherwise, so please don't go out of this  type just creating globals for the sake of it.  So there are two important questions about the active tenant and those are what is the  type of a tenant object?  When we get and set a tenant as the active tenant, what type of object we are working  with?  And the other question is what happens if for some operation, there is just no active  tenant?  Is that a bug?  Is that a possible situation?  Is that a wild card scenario in which you are actually hoping to run the operation in  multiple tenants?  So these are all valid questions, but it's more of foot for thought.  So as we get settled this concept of the active tenant, there are three architectural choices  that we need to make, and these three architectural choices are users and tenants, database architecture  and tenant routing.  Regarding users and tenants, there are three types of relationships between these.  One is the type in which users exist outside the context of tenants.  That is, you can have tenants, you can have users, there could be relationships between  them, but this relationship is kind of loose, non-strict.  An example, WordPress, Shopify and Discord.  Third type is the one in which users are bound and constrained within the scope and the context  of a tenant.  The perfect example is Slack, where you cannot think of a user outside the concept of a workspace.  And there is a third type in which users and tenants are pretty much the same thing.  And this is an example of this is Gmail Dropbox and the like.  So which one to pick will completely depend on your use case and how do you expect your  users and tenants to be interacting?  The baseline question here is how many tenants do you expect a user to be related with?  The second architectural choice to be making is the database architecture itself.  And there are typically three approaches.  One is the isolated approach in which you have multiple databases, one for tenant.  The other one is the shared database approach in which you have a single database and a  tenant column and on entry-level tables.  And the third one, which is kind of reason, is the one in which you have one database,  but you use Postgres schemas to do the tenant, to store your tenant information.  So how would that be if you were to implement multi-tenancy with isolated databases?  Well, Django is compatible with multi-database configurations, so there's nothing stopping  you from having multiple databases there.  The only thing you would have to do is that you would have to be switching between databases  when accessing and storing your data.  Here I am using a translation function from a tenant, from an active tenant, into a database  alias.  And everything you would have to do is just using that alias for saving your objects or  for filtering your queries or even for generating already scope managers so that all your subsequent  queries are already in the context of that database alias.  This could be somehow offloaded to database routers since we have this function to get  the current tenant and we also have a translation function from a tenant into a database alias.  We could be providing default values here so that we don't have to do the previous thing.  The router makes it so that whenever you do a data access, this would be default value  of a database.  And you could still resort to doing manual specific scoping if you need to override the  default behavior.  The good thing about the isolated databases is that you are optimized for isolation.  The bad is that there are no relationships across databases because Django doesn't allow  it and that adding tenants requires reconfiguring the project.  Why?  Because your tenant catalog is actually living in your settings so every time you're adding  a new tenant, you have to update your settings.  The not so funny thing about the isolated databases approach is that as you scale, it's  going to become quite expensive with operational costs.  So unless you're planning to have a number of tenants in the lower 10s or unless you're  planning to have Scrooge, Madoc as your billing manager, this is generally not recommended  approach.  As for the shared database approach, it's the one in which remember we have mixed records  in a single database.  So in this case, entry level 10 specific models will require a pointer to the tenant they  belong to and notice that I'm saying tenant specific models because not every model needs  to be tenant specific.  You could have models that are used to share information across tenants and notice that  I'm also saying entry level because models determine relationships between them.  So you don't actually need to provide a pointer for each one of those as long as you're able  to reach the tenant with a reasonable number of joins.  And this is an example of a hypothetical question model following the stock exchange example  where you're having a foreign key to the site, which is kind of the tenant in this example.  In this case, you still have to rely on the active tenant in order to complete the missing  part of your queries because you will have to use that tenant in order to create your  records if you haven't provided a tenant agnostic interface to the client and you still have  to filter by tenant no matter if you're actually filtering a model that has the tenant pointer  right away or if you're doing a number of joins in order to get to the tenant.  You could do some of these automatically and you could try to assign the tenant automatically  by means of using a default value for the field that holds the tenant which could be  a callable and in this case, if appropriate, could be get current tenant or you could have  a custom field with a pre-safe hook.  If you're using a foreign key, you could even subclass that a field.  Finally, you could resort to having a signal on relevant models in order to complete the  model prior to saving it.  As for querying, well, you could use custom managers and custom queries in order to pre-filter  by your active tenant and therefore the subsequent queries account that these are already filtered.  There could be some annotations on subqueries where these customization probably doesn't  work so you still have to be open to be doing manual scoping from time to time.  The good thing about shared databases is that you are optimized for scalability because  adding tenants is just a matter of adding rows into your tables.  The bad is that your data isolation will take extra effort on the development side and the  not so funny thing about the shared databases is that it's very easy to just forget to scope  any specific query by the tenant.  Therefore, if you don't want to wake up in the middle of the night with your brain wondering  whether you have filtered by the active tenant or not, my recommendation is that you bookmark  all your tenant scope queries that you make automated tests for each one of them and you  make sure that they are returning res holds in the scope of the active tenant and even  that you take a step further and you make an automated test to test that you tested each  one of them.  So it's kind of a riddle but it's going to save you at some point.  Remember that tests are the softest pillow you could have in general for software development  but specifically if you're going to share database approach for multi-tenancy.  As for semi-isolated databases approach, this is going to rely in possible schemas in order  to isolate the tenant within a single database.  So what are schemas?  Well, this is a specific Postgres concept.  They are a layer of abstraction between databases and tables.  They can be thought of as name spaces but the good thing about these name spaces is  that they are not mutually exclusive.  So you can organize them on fine tables in many of them by means of properly configuring  your search path which is also another Postgres specific concept.  In this case, your queries remain practically unchanged.  It seems like magic but you're going to pay the price with an increased technical challenge  somewhere else.  And this is where you will have to do two major things or it's required two major things  in order to get this approach working.  First, we're going to require a custom database backend based in the Postgres backend or any  other backend that is Postgres friendly.  And this backend will have to be able to convert the active tenant into a sequence of schemas  and then you will have to run a query to activate those schemas by means of setting the search  path.  The other interesting challenge and this is kind of the biggest challenge is that we  have to teach Django to do migrations again.  Why?  Because Django doesn't know anything about schemas.  Django knows about databases and Django, you can pass a database in order to run your  migrations but when it comes to schemas, that is not part of the world view of change.  Django itself.  So the idea here is that we are also using a database router in order to use the allow  migrate hook in order to tell Django whether or not it's legal to migrate a specific model  based on the active tenant.  And this also requires the migrate command itself to be kind of tweaked because the migrate  need to operate at the schema level.  The good thing about semi isolated database approach is that you're optimized for isolation  and at the same time you have increased scalability because since you have a single database you  can scale faster by means of adding schemas.  The bad is that it's going to take extra effort to understand and control how schemas interact.  The not so funny part about the semi isolated database approach is that since you're able  to scale faster, you're going to be hitting the thousands of tenants very soon and therefore  thousands of schemas and you still have to run your migrations on each one of those schemas.  So you're going to be paying the price with an increased time to run your migration.  So please be advised about this.  Which one is the best?  Well, neither is it.  Why?  Because there are pros and cons in each one of them and it will depend on your specific  use case and each one of those approaches is capable of shining in their specific use  case. If you want to engage in a respectful ice cream fight, you can hit me in the breakout  and we can continue the debate there.  The third architectural choice is tenant routing.  That is how do you expect to take an incoming request and generate an active tenant out  of it?  So it's generally possible to do that because your tenant is somehow encoded in the incoming  request by means of the user session or headers or it could be even better if your tenant can  be inferred from the URL itself, you could do the translation from the domain, subfolder  or query parameter.  So where is the perfect place to do this translation from an incoming request into an active tenant?  Well, that place is the middleware.  It's a perfect place to do it because you're capable of taking the request, the incoming  request, just activate the tenant and therefore the rest of your request and respond cycle  for that request is going to be guaranteed to be in the scope of that active tenant.  This is a possible implementation of that middleware, or one of the middleware.  And in this case, this middleware is translating from a session into a tenant.  Notice that the major complexity here is actually creating the translation function  that I like to take something from the session and convert it into a tenant to activate because  the rest is pretty much checking if there is no active tenant already and then activate  it.  The good part about this is that you only need to provide different translation functions  for different retrieval methods and you could have similar functions for users, headers,  domain, subfolder parameters and you could still chain one middleware after the other  if you want to combine the power of these retrieval methods.  So you only have to take care that the order will determine the precedence of those retrieval  methods.  Now, what if you want the opposite?  What if you are already in the scope of a tenant and you just want to generate a canonical  URL that is capable to give you, I mean, if you share that URL, it can take you not only  to a specific path within your URLs but also to a specific tenant.  Well, this is not going to be possible in some cases because, I mean, user sessions and  header are not easily encodable as part of the URL but you can totally do that if you  are inferring your tenants from domains of folder or query parameter because even though  Django only reverses the path part of the URL, you can still prepend, interpolate or  append the tenant in order to provide the canonical URL to land in a specific path of  a specific tenant.  And this, of course, will require a little tweaking in the URL reversing process.  As a bonus, I also tell you that as I also give you that it's possible to provide custom  URL comes based on the active tenant.  So if you have different types of tenants and you want to provide different URLs for  those tenants, you could also provide this translation function in order to convert a  tenant into a URL comes module Django already provides this hook where you can just assign  your outcome module into this variable in the request and you will be using that different  module.  So these are the three major architectural choices you have to make.  And you'll see there are multiple choices for each one of those.  Some combinations make sense, play nice together while others don't make much sense at all.  So it's going to be up to you to determine whether or not things are playing nice for  you.  But there is more to Django than just those three parts and those three architectural  choices.  So let's take a quick look at the scope of everything else.  And while this list is not going to be comprehensive, let's please at least see one of the five  major places where this dynamic of the active tenant is going to be an important thing.  So management commands, it's going to be very valuable to be able to run management  commands in the scope of a specific tenant.  And for new management commands, you could just include a tenant argument so that this  command is capable of first activating the tenant and then performing the operation that  the management command is expected to do.  Now for existing non-tenant award commands, you will have to, I mean, these commands are  basically everything that chips with Django and third party applications.  So for this case, you will have to define a special command wrapper that basically takes  the tenant as an argument, but also a new command to be running.  And this wrapper should be in charge of activating the tenant and then calling the inner command.  You will find some packages existing out there that even take a step further in elegance  and combine the arguments of both commands.  But this is kind of trickier to implement, but it's possible, completely possible to  do.  As for file storage, it's also going to be extremely valuable to be able to somehow organize  the files by tenant.  And in this case, you can totally define a custom file storage in order to perhaps prepend  your URLs with a string representation of your tenant, et cetera.  Notice that I'm not providing any code example here because there are multiple types of file  storage depending on your backend.  It could be file system or it could be something in the cloud.  So this actually takes a little bit of more specific use case thinking.  Now in higher security contexts where you cannot afford one tenant accessing files in  a large tenant, there are two kind of workarounds you can do in order to increase the security  level.  One is generating pre-signed URLs so that your static or media URLs are short-lived and  therefore it makes it harder to visit tenant-specific URLs for files.  And the other one is actually using a proxy view so that the view acts as a middleman  and determines whether or not the incoming request has security clearance to access the  specific file storage.  Again, there are some interesting packages that are doing most part of this themselves  so this is more like the underlying concept.  As for cache, well, as a buck, the cache that comes with Django is tenant agnostic.  So if two tenants are using the same cache key, there will be a clash.  But most dangerously, you could be leaking information by means of one tenant storing  some data in the cache and then other tenant retrieving that.  That's a leak.  So one thing you could do is actually generate a special key function that you could then  use as part of your cache configuration and that key function, the only things that need  to do is just augment the string representation of the key itself with the current tenant.  As for salary task, well, it's more a matter of discipline.  The idea is that you pass an identifier of your tenant so that the first thing you do  in your tenant specific salary task is to activate the tenant and then you can resort  to doing the rest of the task itself.  And finally, for channels, well, you're going to require a separate middleware for your  WebSockets, whatever you do for your regular request, you're going to be doing, you'll  have to be doing that also for your middleware, for your channels because they are not fully  compatible.  And if you want to activate tenants from the incoming scope, you're going to be kind of  duplicating your middleware because they are not compatible.  And if you're using channel layers, you will have to also do a similar approach like we  did with cash with your consumer group's name because if you use a tenant agnostic name,  you will also end up leaking messages between the groups of multiple tenants.  So for everything else that was not covered, like admin site or template, tenant specific  templates, I am certain that the principles here are generally extensible.  So I'm almost sure that you will be able to extend the principles in order to cover pretty  much everywhere in the framework.  Otherwise, feel free to continue the discussion in the breakout.  Now some of the packages that actually implement multi-tenancy and help you in integrating  multi-tenancy with your project, which you could choose from.  Here I'm displaying four packages.  Notice that they are kind of opinionated based on the database approach.  These packages were taken from the multi-tenancy greed of Django packages.org, which I consider  to be the market for Django packages.  Please visit there.  You could find more interesting things there.  Just make sure you see whether or not the package is Python 3 compatible, whether or  not it's production ready, and even if it's actively maintained, because otherwise you  might end up in a very difficult situation.  So if you want to contribute back to any of those packages, the ones I mentioned and the  ones I didn't, by means of reporting bugs, implementing new features, or even improving  documentation, I will dare to speak on behalf of everybody, of all the package maintainers,  and tell you, please come.  We need you.  You're more than welcome to do so.  It could be a gold opportunity for you to put in practice everything you may have learned  here or everything you have learned in your own experience and progress through multi-tenancy  in Django.  So that's it.  We can keep in touch for more.  And I would like to give a special thanks to Russel Keith McGee, Orlando Williams, and  Rafael Michele, who were of a huge help in the preparation of this talk.  And lastly, thank you, my dear audience, for joining me today in this journey.  Thank you.
Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_0_155.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_156_203.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_204_335.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_336_431.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_432_503.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_504_551.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_552_623.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_624_695.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_696_875.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_876_923.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_924_1055.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_1056_1175.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_1176_2303.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_2304_2927.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_2928_3275.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_3276_3431.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_3432_3995.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_3996_6419.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_6420_23759.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_23760_27143.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_27144_27575.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_27576_40667.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_40668_41495.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_41496_42755.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_42756_42815.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_42816_42863.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_42864_42911.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_42912_42959.png Lorenzo_Pena_-_Yet_another_package_for_multi-tenan_eYpcPJaE19I_42960_43059.png