Configuring uWSGI for Production: The defaults are all wrong
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Sperl, Peter
Two years ago, we began migrating from a proprietary service framework to a WSGI-compliant one. We chose uWSGI as our host because of its performance and feature set. But, while powerful, uWSGI's defaults are driven by backward compatibility and are not ideal for new deployments. Powerful features can be overlooked due to the sheer magnitude of its feature set and spotty documentation. As we've scaled up the number of services hosted by uWSGI over the last year, we've had to tweak our standard configuration. In this talk, we'll present the base uWSGI configuration we use as a starting point for all services, as well as some tips to avoid known gotchas and provide a base level of defensiveness and high reliability. This base configuration makes use of several ""no-cost"" uWSGI features that help protect services from common, yet difficult to prevent issues - some of which we discovered the hard way. We'll also talk about some programmatic uWSGI features which can be leveraged to improve reliability and improve outage response. Some of the topics we'll cover include: - Mitigating memory leaks - Mitigating stuck, hung, or infinitely looping processes - Preventing misconfigurations - Preventing wasted development effort - Improving outage response License: This video is licensed under the CC BY-NC-SA 3.0 license:-nc-sa/3.0/ Please see our speaker release agreement for details:</a
 Thank you very much guys. Thanks for coming. The brief agenda. So I'll have a brief introduction. We're going to talk about the punch on the title, like basic configuration options of UISI and why you should use them. And then we're going to talk about some other stuff. Basically UISI provides worker management features above and beyond what most UISI hosts do and I want to talk to you about them because they can protect you in production against things you might not be expecting. And then we'll talk about additional features that you won't get from other UISI hosts and then potentially have questions. So I apologize for the clickbaity title of like trying to like make some controversy about UISI. I love the thing. It's great but you do have to get some basics right. So why are we here? We're here to be successful, right? And that means something different for all of us. Some people here are language people. Some people are data scientists. I'm a distributed systems engineer. I like, I build reliable performance systems. It gives you happy users, a profitable business and maybe you're not a for profit organization. Maybe you're a nonprofit. Well, it'll make your organization more impactful. My apologies. So yeah, so building a reliable performance system is important to be successful in whatever enterprise you are running. So this is a talk about distributed systems architecture. Specifically, we're going to zoom all the way into the individual service level and UISG is a tool we'll be doing to do that. I bring this up because I often talk about different parts of the distributed systems stack. In 2017, I gave a talk at PyGotham about stateless systems and how stateless systems can make your system more testable and more performance. And so this is the exact opposite of that. We go all the way into the very smallest level and talk about each individual microservice and how it's hosted. So why UISG? Well, it is very powerful. The list of features is tremendous and it's very fast. But it was written in a different time back in 2008 when UISG was written. People did things differently and a lot of the defaults represent that. There will be a lot of quotes in here from Unbit, the developers of UISG. Is anyone from Unbit here? I was hoping they would be. I was also scared they would be. All right. So they're not. Thank goodness. But they have production customers, people that pay them money and a lot of the decisions they make are about catering to those customers, rightly so. So I want to talk about how to avoid those problems and then move on to the things that UISG can provide above and beyond other UISG hosts. Almost every item I'm going to present today, at least in the first third, caused an issue for us at some point, a varying difficulty. And they might seem like things that are like, wow, how did that hit you? Well, I have a very large team of developers between 40 and 50 at any time working on services hosted by UISG. And those guys are productive. They get a lot of work done. They stumble across corners. So if you're about to start a project with a UISG project, maybe with Flask or Django or anything for that matter, you should consider UISG as your host. And you should use this as a reference. Now UISG 2.1 is supposed to be released at any moment. Back in July 2017, the Unbit developers posted to a mailing list that they have decided to fix all the bad defaults, especially for the Python plugin, because little known secret, UISG hosts Java services, Ruby services, anything you can imagine, really, in the 2.1 branch. The 2.1 branch has not been released as of June 2019. I know this because I check it every morning leading up to this talk, worried that they would. The first reference to UISG 2.1 was in 2014. So who knows when it will get released. Maybe that will provide some respite to these problems. But as it is now, you should use the contents of this talk as a basis if you're going to start any development of a UISG service using UISG. There's an official things to note, doc, that UISG provides that is very valuable. I recommend you read it. I'm not going to say much more about that other than that. And I reference it every once in a while throughout this talk. I would also like to say that we have published the contents of this talk in an article at techatballomburg.com about an hour ago. So if you go to this website on your phone or on your laptop, you will see in markdown form what we're going to talk about today. So in the presentation, things will be abbreviated so there's not too much text on the screen. But at the website, it is very, very, it's expanded on very greatly so you can kind of understand our reasoning for each of the items and like with specific examples. So it's a much better reference than the slides are today. All right, so let's get started. That's it. I can leave now. Do this. Right? Now I will explain each one of them. But yeah, this is what you want to do. Let's just get right into it. So the master process is what makes you UISG special. You want it on. I mean, this is kind of obvious. We should just get past this point. I want to bring it out though because there are some circumstances where you want it off, specifically when you're debugging, if you want to test your service and you turn the master process off and you run in a single process, which you have to do with the master processes off, then the PID that is created immediately after you start UISG is where your code will be running. So you can use tools like S-Trace, which we talked, we heard about earlier in the conference, or any sort of command line profiling tools like Valgrind, for example, and you won't have to deal with all the forking options, like follow fork child and all the stuff that you normally have to do when you're using a tool like Valgrind to debug or to profile a Python service. So otherwise, not that interesting. Strict config parsing. Okay. So by default, UISG lets you put anything you want in the configuration file. And the justification for that is that you can add non-existent options to your config files as place holders, customs options, or app-related configuration items. But the truth is that most of us don't do that stuff. Who was the last time you ever passed through application-specific configuration options in a framework configuration file? Never. You shouldn't do this. So you should turn this off. Why? Because it happened to us. Somebody meant to type something in the config file to control behavior in a way, and they fat-fingered it. They put a wrong character in. They moved the production. The service works, right? It's not like it didn't work. It's just that that option wasn't set properly. Service moves the production, and then something doesn't behave the way you want, right? This is an example of UISG having some default behavior that can allow you to do something that you don't intend. So you should set this. Now if you do want to use this feature of passing through configuration options through the UISG INI file, absolutely feel free to do it. But that's not the default, right? Or at least it shouldn't be, in my opinion. The next option is vacuum. So this basically makes UISG clean up after itself. Common example is if you run your service on a local UNIX socket. Without this option, UNIX socket will not be deleted for you. This sounds harmless, right? No big deal? Well, in my environments, we have multiple users operating in the same machine. And if they start a development instance of their service, the socket will get their username, and they might not have set their mask properly so that all users in the systems can read and write those and delete those files. So if another user tries to start up a service on that socket location, they will get a failure because they don't have file permissions to delete that socket. So another thing that you should just set by default, there's no real reason to leave these temporary files around. It's just one of those things you don't need. If your application is special for some reason and you need it, then feel free to move this to false. But by default, it should just be there. And if you have a large development team, it might save someone cursing another one of your employees one day in the future because that is what happens. All right. So now we're going to enter a section where there's a common theme, and that is UISG default options changing the expected behavior of your code. By default, UISG disables Python threading, which therefore removes the gil. The explanation for this, these are all quotes, by the way. These are gray background sections, quotes from the UISG developers directly. They do this because they think that it will improve performance. I don't know if it does or not. Maybe it does. All the applications that I have that have sufficient complexity end up using threads for something in the background, and so I had to turn this on right away. I pointed out because while it's relatively easy to find, the first time you come across this, your instinct isn't to think, well, my service host must be messing up my threads. Your instinct is like, ah, I screwed something up in my threading code, right? Just start looking at your code. And then depending on how good your employee is at Googling or yourself, it can take anywhere from five minutes to a few hours to figure this out. So just turn it on by default. If you're super performance sensitive and you don't use threads, you can opt to turn it on intentionally, but it's not a very good default behavior. Another one is the single interpreter parameter. UISG really has a lot of features. This one allows you to run multiple interpreters in every worker process. UISG is the pre-forking model, so if you want to hold to run 60 requests concurrently, it'll spin up 60 processes. If you allow multiple interpreters, UISG will allow you to run more than one web application in each process. Basically, it's just a way to condense your processes and keep your PIDs down. I don't really think it's necessary in the modern environment where people are running containers or in virtualization, but back when they wrote UISG and they were working on shared physical hardware, keeping the number of PIDs down could end up being important. So this usually has no side effects, but there are some third-party C modules that do not work well in multiple interpreters. They make some assumptions about, like, globals in C memory space, which isn't validated when you're multiple Python interpreters. I actually don't remember how this caught us. I assume it must have. Otherwise, we wouldn't have just found it randomly. So sorry, I can't give you some background on how we stumbled onto this one. And then the last one, which is just frustrating, is die-on-term. So by default, UISG treats the SIG term signal as reload the application stack. Maybe you have updated your code and you want to, you know, refresh what the server is hosting. Sure, fine. That's a great feature. I think you should have it. But why would you use SIG term? We all collectively agreed on what SIG term should do. It should kill something. And for some reason, they decided that they wanted to do something else with it. So I just don't get it. But the first time you start using UISG, someone's going to try and literally type kill the PID number because they're, you know, for whatever reason, they lost their terminal and the service kept running in the background. And it's just not going to stop. And they're going to be like, what's going on? And then if you're smart, you'll do kill-9 and get past it. But someone might want to figure out what's going on and then you have to find this parameter. So again, it's just the common theme of UISG deciding to change behavior away from what we all expect programs to do to something else for their use case, which we no longer need. All right? So you have the service in 2019 and you want to use UISG as your host, which you should, just by all the things I'm saying, set as a default. And this, sorry, I get excited about this one. This is the worst to me. This is the most egregious configuration parameter I think I've ever seen in any project. Without this parameter, UISG will start if it cannot load your application. That means if you have syntax errors in your application or some other dependency of your application cannot be loaded, it just starts anyway. And guess what it does? It serves 500 and 400s to everybody because that's the only thing it can do. Now what's insidious about this is that a lot of systems will just check to make sure that the server's awake, right? You have like a health checking system that's really basic or you have some sort of like, you know, a lot of these systems are pretty naive. Your service starts. It looks fine. Then you start redirecting production traffic to it and everything goes boom. The reason they did this is back in 2008, it used to be a common pattern to start UISG with nothing loaded and then dynamically inject applications. I can't imagine what the world was like back then although technically I was operating as an engineer professionally. I don't remember doing it this way. But UISG did it that way back then. So nowadays you want to set need app equals true and what that will do is if you try to start your application and there's a syntax error, it will fail right away. I don't know if you guys saw this talk today about the guy doing the lookmon no HTTP, but his whole demo relied upon him changing code and then having an auto loader restart his service every time his code was changed so that he could see the errors he made immediately. If you were using UISG, the whole thing would be for naught, right? Because it would start up and it would look like you were fine. So don't do it. This one is actually just an opinion. It's not actually anything that I can pin on the UISG developers. UISG is very verbose with logging. Most of us, if we're developing an application, will have very concise but meaningful application specific logging. Probably because we're feeding our logs into some grander distributed grep system where we have to pay by byte and we don't want to necessarily spam the thing. So I recommend disabling UISG's default logging. I don't particularly like the format, although that's all arbitrary, right? We all have our own arbitrarily preferred log formats. But what I do think you should do, and that's really what I wanted to talk about, is you should enable the UISG default logging for 400 and 500 error codes. And why do I say I do that? Because it is very, very difficult as an application developer to guarantee that you will always catch an exception or an error and log it properly. It is far more likely that some percentage of the time you will intend it to capture all your errors, but you won't because it will occur before your logging handler or some other situation that your logging handler can't anticipate. So I recommend turning this on to make sure you always have some indication that some traffic had failed, even if you also do that in your application logger. Two log lines aren't going to hurt you, but zero will. And that's kind of it. So if you stick to this, and you're running a decent sized application team or you might in the future, you can potentially avoid a lot of significant issues that will either waste developer time or cause potentially major production outages. After this, though, we're going to get a little bit more positive. And again, I see for you guys taking pictures, check out techapploomurg.com. All this content will be there with basically all the things that I said written down instead of bare bones like the slides. One of the things I want to talk to you about is worker management. I feel passionately about this, and most Whiskey hosts don't even bother. They're just like, you need 60. Be sure I'll give you 60. And that's kind of like they stop talking about workers at that point. You Whiskey gives you a real powerful wealth of features in this area. The most dearest time of heart is worker recycling. What this basically does is make sure your workers don't get too old. And there's many metrics for that. I have three different methods here. MaxRequest basically tells you Whiskey to restart your workers after 1,000 requests. That number is pretty small, depending on your situation. At Bloomberg, we're doing data services. So there's a lot of financial calculations that take a lot of time. So 1,000 requests actually might take a little while. If it's been alive for more than an hour, or if the process has allocated more than two gigs of memory, resident memory, and that all of those go into effect after the request is finished. So if you have a request that goes from, let's say, from one gig to eight gigs in one request, it will wait until it's finished, and we'll clean it up after it reaches eight gigs. I point that out because this is very valuable to help you prevent a certain class of errors. If you have a slow memory leak and you recycle your workers, you might not ever figure it out. If you have a few bytes that are leaked to everyone else, because an object somewhere that's not cleaned up, and you turn on all these options here, probably won't be a problem. Now obviously, you should find that slow memory leak anyway and fix it. But that doesn't mean you should let it cause a problem for you in production. So these features are great, and you should turn them on by default. As far as I can tell, there is really no cost to this, except for maybe if your application has a huge startup time. These workers are forked off of the parent one anyway, so that even isn't really a big problem, but I suppose if you had a huge amount of page tables, the forked time might be significant. I don't know. I have never personally observed any issue with this whatsoever. And then the worker reload mercy allows you to configure how long to wait before you forcefully close your workers if for some reason they're holding onto a resource that isn't gracefully giving up its control. Then there's dynamic worker scaling. So this again is not strictly required. This is a feature that I love because I happen to run multiple WISB services on the same physical host or in the same virtual machine. And I also run multiple versions of the same service. If you see my talk about stateless services, we run multiple versions of the same stateless services and then basically repry requests to all of them to compare differences and responses between them. So if you were to look at one of my machines, this is obviously an exaggeration. We don't run 500 processes, but we do often run 96 or 128. And if you have 10 or 20 services with that many processes each, your top or your PS can be pretty busy. So dynamic worker scaling allows you to start up with a lower number of workers and then scale up and down depending on the load in the system. And it is anticipatory. Anticipatory? Anticipative? I don't know. It anticipates how much traffic it expects based on how much traffic you're getting at the moment and generally leaves enough of a buffer that spikes won't outpace it. You can see in the bottom there's some parameters about the backlog. That feature in particular makes sure that if it's not scaling up fast enough, it will create emergency workers to make sure you can handle it. And again, as far as I can tell, this has never been a problem for us. It scales fast enough. It's not like a risk. It's just another no cost feature that you should use. It's also pretty sophisticated. There's a lot of options, a lot of different algorithms. This is the configuration that I like, but you should definitely, if you're going to do this, you should understand your application and check out the other algorithms other than the business one that I have selected. Okay, hard timeouts. This is a really powerful feature that I have not seen in other Whiskey hosts. You Whiskey, if you turn on Hara Kiri, will forcibly kill your worker process if it is stuck. I point this out because this happens all the time and this can basically make that a non-issue. I mean, when it happens, you should log it and alert on it and you should figure out why and fix it. But we had another talk earlier today. I forget the gentleman's name, but he was talking about using async IOM production and some of his war stories. Granted, he was talking about asynchronous cases and it's a little different, but he explained a situation where he had a spike of traffic come in or conversely, one of his downstream dependencies was slowing down or reaching its limit and his service basically backed up with requests. He had thousands and thousands of requests going on in parallel and they could never catch up. Even if traffic went down to zero, it would take hours for everything to recover. This is what this feature is for. If a spike of traffic comes in and your system for some reason locks up, maybe not because of a deadlock, but because of just limited resources, HaraCury will clear all that out after 60 seconds. When traffic does recover, your system will too. Granted, you should understand why that queue situation occurred and you should expand capacity of the limited resources and make your system more reliable in that regard, but there's no reason you have to get woken up in the middle of the night every time. That's what this is for, right? CYA, protect yourself. No cause made of strategies because you made a mistake. We make mistakes. Use the tools to protect you. You should use HaraCury. We use 60 seconds. That's a long time for most people, but we're doing, like I said, numerical calculations on mortgage backed securities and other large financial instruments. Some of our services go much higher than 60. The lower you can set this, the better because that's basically your period in which you can have a queue condition. Once that period turns over, all the stuff requests will clear out. This is an analog to this. Hard timeouts are kind of drastic because you're literally kill-9ing your workers. You really don't want to get to that. This should have been in the first section, but I put it here so we could contrast it. This UISC configuration option allows worker processes to receive signals. Without it, UISC does not allow this to happen. I have no idea why. I actually couldn't tell you. I couldn't find the justification for this. Hence there's no quote from them. The reason that you want this is that you might want to receive a signal. For example, in the Python signal module, you can call signal.alarm and tell the Python interpreter to wake up this handler if you have a, after a certain amount of time. One example would be, you set a hard timeout at 60 seconds and you use the signal module to set a soft timeout at 59 seconds. Oftentimes I'm serving requests where I have partial responses I can return, even if I can't return the full thing. In our system, we use the signal module to wake up any waiting process at 59 seconds, gather whatever data we have, and return it to the client. If that fails for some reason, for example, say we have an actual stuff process, this will kick in, sorry, Hara Kiri will kick in and kill the process to make sure we continue to operate properly. So these two things kind of go together, but this is really a default. You should just set this. There's really no harm in it. If no one sends your process signals, then fine. If you want to, this is another one of those situations where it's like, how good are you at Googling to figure this out? This one is very important as well. So we had a conversation, sorry, a talk at this conference about observability, about tracing and logging and monitoring, and how observability is more than just the three things that he talked about, which was logging, metrics, and tracing. This is another area you can get observability to your software. By default, when you start a service host, most of them give you this boring output. That's just the command line that you use to invoke the service, which is literally useless practically. I mean, I guess it tells you the Python module that you loaded, but that's really it. Uwizgi's aware of this, and they give you an option called auto-procname that you can put in the config file that will do something like this. It will tell you which process is the master, which process is the workers, and which worker number it is. This is still a little naive, right? What if you have more than one Uwizgi service in the same box? You might end up with 10 Uwizgi masters, and which one is the one you want. So you can do a prefix, you can say prop name prefix, and give it a service name. That's much better, but it's still not the best we can do. There is an API you can invoke from within your code to specify dynamically what you want the prefix to be. Here's an example. This is contrived. I'm not actually hosting. This isn't how we actually do it, but it gets the example across, where you can put context about what that process is doing. In this case, you can specify the service, the username, and then the URI they are accessing. What this helps you with is sometimes you have a problem where there's a lot of slowness on your hosts, and there's a lot of stuck processes. Very quickly, but it's running top or PS, you can see that it's a pattern, that it's all the same user, or that it's all the same URI. There's something common about that problem, which otherwise would perhaps take you a little while digging through trace or digging through logs. You can kind of get a quick snapshot of it. So it's not a replacement for anything else. It's just a straight, free, zero-cost additional tool that you can add to your system. That section I want to talk to you about is additional UISG features. This is a controversial section, because if you use any of these features, you have now lost compatibility with other UISG hosts. You can no longer switch back to G-Unicorn if you use these, because you are now leveraging UISG specific functionality in your code. That said, it can still be worth doing. Honestly, we don't switch hosts that often once we get a system in production. These features can help you solve or provide a solution to something with much less complexity than you would be able to otherwise. It's a trade-off between reducing complexity and having interchangeability of your UISG hosts. We end up using some of these, because the reduction in complexity is just tremendous when you're running a large system. The first one I want to talk to you about is a Kron and Timer. It's literally what it sounds like, running Kron's and Timer. You can say, do something every first of the month, or do something every 20 minutes. It does not let you do one-time timers. If you wanted to say, do something in 60 seconds, you should use the signal module. Why would they write something that already is in the standard library? I point this out as being useful, because sometimes you want your periodic tasks to be synchronized with the software version, the code that you're deploying. Sometimes you have this problem where you want to put it in a new service that has some functionality you want to run on a timer, and you have to update your Kron system separately. Now you can deploy those things in an atomic manner. If you're going forward, this isn't usually too big of a problem, because you can deploy the new features and then change the Kron timer later. But if you're going backwards, actually, it could be dangerous, because you've now moved your service forward. There's some bug in it. You want to roll back, but now you have some Kron job that expects that behavior to be there. This can help you deploy software in a more atomic fashion. Also, if you just don't like using Kron or any of those things, you can do it in the service itself. You can also do it with a decorator, instead of with doing it at the global level. So you can put a decorator around a function, and you will call that periodically. Next thing I want to talk to you about is locks. This is actually very difficult to do. If you don't have a framework-supplied lock like this, you have to try and get some code to run before the pre-forking portion of you, Izgy, forks all your processes and create a lock in global space and then share it around, or use an external lock in Redis or something. It's actually a real pain. I don't think you should write software that needs a lock. If you have to do this, you probably messed up, but if you do, this is probably a better way to do it than all of the alternatives. So I'm just making you aware that it's there, even though I would kind of feel dirty if I had to use it. Then there's the cache system. So this is super cool. This is a cache that you, Izgy, provides so that all the workers can share information. You could obviously use Memcache or Redis as well, right? Like, of course you could. But then you have to create a separate system to start those things and to monitor them, so it's kind of nice to have it all packed together. We use this for rate limiting, actually. What we do is every time a request comes in, we store in the Izgy cache how many requests a given user is doing, and then we reject requests that they've reached some threshold. You might say, well, why don't you have HA, Proxy, or NGINX do that? Well, the reason is that we want to throttle people in a more sophisticated manner than one of those Proxies can. We actually want to look at the HTTP request in pretty detailed ways in order to do that. So this is a pretty powerful feature. It's got some gotchas, for example, keys or values that are too big for the cache will suddenly fail to insert. So you need to kind of make sure you have a good test suite around it. But once it's working, it's bulletproof, and it's much simpler than starting up a remote or an external Memcache or Redis instance to get the same job done. And then there's mules. So mules are worker processes that are not workers. They're not servicing or requesting your clients, but they're there to do stuff. We use this, for example, to aggregate metrics. So all the worker processes will send the mules metrics, and then the mules will offload those metrics to your metrics engine, whether you use Datadog or Metric Tank or Grafana or whatnot. So you can put a little decorator on that and say target equals mule. If you don't say target equals mule, it'll run in just a random worker process, or all worker processes. So that's how you specify that. And then there's ways to send data there as well, which I include in the longer form article that I posted on tech at bulletinberg.com, if you want to see that. So that's it. I am hiring. I am looking for an SRE. And just to qualify, when I say an SRE, I mean a distributed systems engineer who doesn't get work from product people who want features built, whose job it is to optimize, monitor, and improve the performance of our system and its reliability for the sake of itself and not to get some featured on for a business guy. We are a pretty great place to work. Please talk to me. I would love to tell you more about the role. We have some people already doing this role, so you'd be joining a team. You wouldn't be bootstrapping an organization. But you'd have to come live in New York, which I know is terrible. Who wants to live in New York? That's it. I think we have a few minutes for questions, if there are any. Thank you. Yeah, we have a few minutes for questions. Again, microphones are there, and there. Please line up behind the microphone and ask away. Hello? Hello? Hi. Could you tell me please more about the hierarchy option? Why should I use it? Why should I just not set up like worker load time? So the worker process recycling will, I think, will kick in. I'm not 100% sure about this, but I think it will kick in and restart workers. But generally, your worker process reloading is going to have a long time interval. You might say an hour or a thousand requests. So if you set up a request, well, it will never reach that, obviously. If you set up a memory, it will never reach that either. So it's only the time one that will catch it. And then it's the question of, do you want to wait an hour or 60 seconds? That's really it. To me, I want to wait 60 seconds. If I wait for that hour to happen, many, many bad things would happen. This actually happens to people. You have a bug in your code in every worker deadlocks. It happens. So the hierarchy is a way to help prevent that. Whereas if you use just the normal worker process reloading, the time interval will be too long. All right. Over there. Hi. What about lazy apps loading? So loading app and forking versus forking and the loading apps. After all, have you been bitten by that? Or anything? So we prefer the post fork option. So we would want to load everything and then fork after. That's a great question. I don't think we have yet. Doing it lazily makes the process reloading take much longer. Because now you have to wait to load all of your application code, which can take seconds sometimes. So we have not yet. But that's a great feature to be aware of. If there's something about your application that you can't load it once and fork trivially, you can enable the lazy apps option and that will prevent that from being a problem. Usually that kind of stuff happens if you import something and it initializes something globally that's also like C extension and then you're doomed for lazy stuff. Absolutely. Thank you. So thanks for the talk. And I wanted to ask about your deployment environment because I noticed you have a huge number of workers. And is it containerized? There are small machines or big machines with a lot of CPUs. In our particular situation, we generally have dual socket Linux servers, physical servers, which somewhere around between 20 and 28 cores, generally about half a terabyte to a quarter of a terabyte of RAM. We have some virtual deployments on VMs, but we don't really prefer them because we run a very high duty cycle in these machines. We're doing like Monte Carlo simulations of mortgage backed securities, which is why I have so many workers. And we have a cluster of 500 machines that we run at 100% CPU for 20 hours a day. And so if we were to go to a virtual environment, it would just cost more money. We need more machines once you get the overhead there. So we are capable of running in a containerized environment if we need to, but for money's sake and efficiency's sake, we run a physical hardware. Thank you. Cool. One last question very quickly. Do you have any recommendations or gotchas and so on for Emperor mode or would any of your recommendations here change with that except for, I think, one option you mentioned? To be totally honest, I don't understand what like Emperor mode and Zerg mode and these things are for. I mean, we have a microservice environment where each service can run independently and can be deployed independently. So I honestly have a completely ignorant to all of that stuff. Sorry. But you kind of almost recommend against using Emperor mode at all. I don't know why you would use it. Right. So I haven't seen the use case for it. I suspect it's one of these back in the day that was the best that they could do. Now things have progressed. Thanks. All right folks, let's thank all our speaker again.
 Thank you very much guys.  Thanks for coming.  The brief agenda.  So I'll have a brief introduction.  We're going to talk about the punch on the title, like basic configuration options of  UISI and why you should use them.  And then we're going to talk about some other stuff.  Basically UISI provides worker management features above and beyond what most UISI hosts  do and I want to talk to you about them because they can protect you in production against  things you might not be expecting.  And then we'll talk about additional features that you won't get from other UISI hosts and  then potentially have questions.  So I apologize for the clickbaity title of like trying to like make some controversy  about UISI.  I love the thing.  It's great but you do have to get some basics right.  So why are we here?  We're here to be successful, right?  And that means something different for all of us.  Some people here are language people.  Some people are data scientists.  I'm a distributed systems engineer.  I like, I build reliable performance systems.  It gives you happy users, a profitable business and maybe you're not a for profit organization.  Maybe you're a nonprofit.  Well, it'll make your organization more impactful.  My apologies.  So yeah, so building a reliable performance system is important to be successful in whatever  enterprise you are running.  So this is a talk about distributed systems architecture.  Specifically, we're going to zoom all the way into the individual service level and  UISG is a tool we'll be doing to do that.  I bring this up because I often talk about different parts of the distributed systems  stack.  In 2017, I gave a talk at PyGotham about stateless systems and how stateless systems  can make your system more testable and more performance.  And so this is the exact opposite of that.  We go all the way into the very smallest level and talk about each individual microservice  and how it's hosted.  So why UISG?  Well, it is very powerful.  The list of features is tremendous and it's very fast.  But it was written in a different time back in 2008 when UISG was written.  People did things differently and a lot of the defaults represent that.  There will be a lot of quotes in here from Unbit, the developers of UISG.  Is anyone from Unbit here?  I was hoping they would be.  I was also scared they would be.  All right.  So they're not.  Thank goodness.  But they have production customers, people that pay them money and a lot of the decisions  they make are about catering to those customers, rightly so.  So I want to talk about how to avoid those problems and then move on to the things that  UISG can provide above and beyond other UISG hosts.  Almost every item I'm going to present today, at least in the first third, caused an issue  for us at some point, a varying difficulty.  And they might seem like things that are like, wow, how did that hit you?  Well, I have a very large team of developers between 40 and 50 at any time working on services  hosted by UISG.  And those guys are productive.  They get a lot of work done.  They stumble across corners.  So if you're about to start a project with a UISG project, maybe with Flask or Django  or anything for that matter, you should consider UISG as your host.  And you should use this as a reference.  Now UISG 2.1 is supposed to be released at any moment.  Back in July 2017, the Unbit developers posted to a mailing list that they have decided to  fix all the bad defaults, especially for the Python plugin, because little known secret,  UISG hosts Java services, Ruby services, anything you can imagine, really, in the 2.1 branch.  The 2.1 branch has not been released as of June 2019.  I know this because I check it every morning leading up to this talk, worried that they  would.  The first reference to UISG 2.1 was in 2014.  So who knows when it will get released.  Maybe that will provide some respite to these problems.  But as it is now, you should use the contents of this talk as a basis if you're going to  start any development of a UISG service using UISG.  There's an official things to note, doc, that UISG provides that is very valuable.  I recommend you read it.  I'm not going to say much more about that other than that.  And I reference it every once in a while throughout this talk.  I would also like to say that we have published the contents of this talk in an article at  techatballomburg.com about an hour ago.  So if you go to this website on your phone or on your laptop, you will see in markdown  form what we're going to talk about today.  So in the presentation, things will be abbreviated so there's not too much text on the screen.  But at the website, it is very, very, it's expanded on very greatly so you can kind of  understand our reasoning for each of the items and like with specific examples.  So it's a much better reference than the slides are today.  All right, so let's get started.  That's it.  I can leave now.  Do this.  Right?  Now I will explain each one of them.  But yeah, this is what you want to do.  Let's just get right into it.  So the master process is what makes you UISG special.  You want it on.  I mean, this is kind of obvious.  We should just get past this point.  I want to bring it out though because there are some circumstances where you want it off,  specifically when you're debugging, if you want to test your service and you turn the  master process off and you run in a single process, which you have to do with the master  processes off, then the PID that is created immediately after you start UISG is where your  code will be running.  So you can use tools like S-Trace, which we talked, we heard about earlier in the conference,  or any sort of command line profiling tools like Valgrind, for example, and you won't  have to deal with all the forking options, like follow fork child and all the stuff that  you normally have to do when you're using a tool like Valgrind to debug or to profile  a Python service.  So otherwise, not that interesting.  Strict config parsing.  Okay.  So by default, UISG lets you put anything you want in the configuration file.  And the justification for that is that you can add non-existent options to your config  files as place holders, customs options, or app-related configuration items.  But the truth is that most of us don't do that stuff.  Who was the last time you ever passed through application-specific configuration options  in a framework configuration file?  Never.  You shouldn't do this.  So you should turn this off.  Why?  Because it happened to us.  Somebody meant to type something in the config file to control behavior in a way, and they  fat-fingered it.  They put a wrong character in.  They moved the production.  The service works, right?  It's not like it didn't work.  It's just that that option wasn't set properly.  Service moves the production, and then something doesn't behave the way you want, right?  This is an example of UISG having some default behavior that can allow you to do something  that you don't intend.  So you should set this.  Now if you do want to use this feature of passing through configuration options through  the UISG INI file, absolutely feel free to do it.  But that's not the default, right?  Or at least it shouldn't be, in my opinion.  The next option is vacuum.  So this basically makes UISG clean up after itself.  Common example is if you run your service on a local UNIX socket.  Without this option, UNIX socket will not be deleted for you.  This sounds harmless, right?  No big deal?  Well, in my environments, we have multiple users operating in the same machine.  And if they start a development instance of their service, the socket will get their username,  and they might not have set their mask properly so that all users in the systems can read  and write those and delete those files.  So if another user tries to start up a service on that socket location, they will get a failure  because they don't have file permissions to delete that socket.  So another thing that you should just set by default, there's no real reason to leave  these temporary files around.  It's just one of those things you don't need.  If your application is special for some reason and you need it, then feel free to move this  to false.  But by default, it should just be there.  And if you have a large development team, it might save someone cursing another one  of your employees one day in the future because that is what happens.  All right.  So now we're going to enter a section where there's a common theme, and that is UISG default  options changing the expected behavior of your code.  By default, UISG disables Python threading, which therefore removes the gil.  The explanation for this, these are all quotes, by the way.  These are gray background sections, quotes from the UISG developers directly.  They do this because they think that it will improve performance.  I don't know if it does or not.  Maybe it does.  All the applications that I have that have sufficient complexity end up using threads  for something in the background, and so I had to turn this on right away.  I pointed out because while it's relatively easy to find, the first time you come across  this, your instinct isn't to think, well, my service host must be messing up my threads.  Your instinct is like, ah, I screwed something up in my threading code, right?  Just start looking at your code.  And then depending on how good your employee is at Googling or yourself, it can take anywhere  from five minutes to a few hours to figure this out.  So just turn it on by default.  If you're super performance sensitive and you don't use threads, you can opt to turn  it on intentionally, but it's not a very good default behavior.  Another one is the single interpreter parameter.  UISG really has a lot of features.  This one allows you to run multiple interpreters in every worker process.  UISG is the pre-forking model, so if you want to hold to run 60 requests concurrently, it'll  spin up 60 processes.  If you allow multiple interpreters, UISG will allow you to run more than one web application  in each process.  Basically, it's just a way to condense your processes and keep your PIDs down.  I don't really think it's necessary in the modern environment where people are running  containers or in virtualization, but back when they wrote UISG and they were working  on shared physical hardware, keeping the number of PIDs down could end up being important.  So this usually has no side effects, but there are some third-party C modules that do not  work well in multiple interpreters.  They make some assumptions about, like, globals in C memory space, which isn't validated  when you're multiple Python interpreters.  I actually don't remember how this caught us.  I assume it must have.  Otherwise, we wouldn't have just found it randomly.  So sorry, I can't give you some background on how we stumbled onto this one.  And then the last one, which is just frustrating, is die-on-term.  So by default, UISG treats the SIG term signal as reload the application stack.  Maybe you have updated your code and you want to, you know, refresh what the server is hosting.  Sure, fine.  That's a great feature.  I think you should have it.  But why would you use SIG term?  We all collectively agreed on what SIG term should do.  It should kill something.  And for some reason, they decided that they wanted to do something else with it.  So I just don't get it.  But the first time you start using UISG, someone's going to try and literally type kill the  PID number because they're, you know, for whatever reason, they lost their terminal  and the service kept running in the background.  And it's just not going to stop.  And they're going to be like, what's going on?  And then if you're smart, you'll do kill-9 and get past it.  But someone might want to figure out what's going on and then you have to find this parameter.  So again, it's just the common theme of UISG deciding to change behavior away from what  we all expect programs to do to something else for their use case, which we no longer  need.  All right?  So you have the service in 2019 and you want to use UISG as your host, which you should,  just by all the things I'm saying, set as a default.  And this, sorry, I get excited about this one.  This is the worst to me.  This is the most egregious configuration parameter I think I've ever seen in any project.  Without this parameter, UISG will start if it cannot load your application.  That means if you have syntax errors in your application or some other dependency of your  application cannot be loaded, it just starts anyway.  And guess what it does?  It serves 500 and 400s to everybody because that's the only thing it can do.  Now what's insidious about this is that a lot of systems will just check to make sure  that the server's awake, right?  You have like a health checking system that's really basic or you have some sort of like,  you know, a lot of these systems are pretty naive.  Your service starts.  It looks fine.  Then you start redirecting production traffic to it and everything goes boom.  The reason they did this is back in 2008, it used to be a common pattern to start UISG  with nothing loaded and then dynamically inject applications.  I can't imagine what the world was like back then although technically I was operating  as an engineer professionally.  I don't remember doing it this way.  But UISG did it that way back then.  So nowadays you want to set need app equals true and what that will do is if you try to  start your application and there's a syntax error, it will fail right away.  I don't know if you guys saw this talk today about the guy doing the lookmon no HTTP, but  his whole demo relied upon him changing code and then having an auto loader restart his  service every time his code was changed so that he could see the errors he made immediately.  If you were using UISG, the whole thing would be for naught, right?  Because it would start up and it would look like you were fine.  So don't do it.  This one is actually just an opinion.  It's not actually anything that I can pin on the UISG developers.  UISG is very verbose with logging.  Most of us, if we're developing an application, will have very concise but meaningful application  specific logging.  Probably because we're feeding our logs into some grander distributed grep system where  we have to pay by byte and we don't want to necessarily spam the thing.  So I recommend disabling UISG's default logging.  I don't particularly like the format, although that's all arbitrary, right?  We all have our own arbitrarily preferred log formats.  But what I do think you should do, and that's really what I wanted to talk about, is you  should enable the UISG default logging for 400 and 500 error codes.  And why do I say I do that?  Because it is very, very difficult as an application developer to guarantee that you will always  catch an exception or an error and log it properly.  It is far more likely that some percentage of the time you will intend it to capture  all your errors, but you won't because it will occur before your logging handler or some  other situation that your logging handler can't anticipate.  So I recommend turning this on to make sure you always have some indication that some  traffic had failed, even if you also do that in your application logger.  Two log lines aren't going to hurt you, but zero will.  And that's kind of it.  So if you stick to this, and you're running a decent sized application team or you might  in the future, you can potentially avoid a lot of significant issues that will either  waste developer time or cause potentially major production outages.  After this, though, we're going to get a little bit more positive.  And again, I see for you guys taking pictures, check out techapploomurg.com.  All this content will be there with basically all the things that I said written down instead  of bare bones like the slides.  One of the things I want to talk to you about is worker management.  I feel passionately about this, and most Whiskey hosts don't even bother.  They're just like, you need 60.  Be sure I'll give you 60.  And that's kind of like they stop talking about workers at that point.  You Whiskey gives you a real powerful wealth of features in this area.  The most dearest time of heart is worker recycling.  What this basically does is make sure your workers don't get too old.  And there's many metrics for that.  I have three different methods here.  MaxRequest basically tells you Whiskey to restart your workers after 1,000 requests.  That number is pretty small, depending on your situation.  At Bloomberg, we're doing data services.  So there's a lot of financial calculations that take a lot of time.  So 1,000 requests actually might take a little while.  If it's been alive for more than an hour, or if the process has allocated more than  two gigs of memory, resident memory, and that all of those go into effect after the request  is finished.  So if you have a request that goes from, let's say, from one gig to eight gigs in one request,  it will wait until it's finished, and we'll clean it up after it reaches eight gigs.  I point that out because this is very valuable to help you prevent a certain class of errors.  If you have a slow memory leak and you recycle your workers, you might not ever figure it  out.  If you have a few bytes that are leaked to everyone else, because an object somewhere  that's not cleaned up, and you turn on all these options here, probably won't be a problem.  Now obviously, you should find that slow memory leak anyway and fix it.  But that doesn't mean you should let it cause a problem for you in production.  So these features are great, and you should turn them on by default.  As far as I can tell, there is really no cost to this, except for maybe if your application  has a huge startup time.  These workers are forked off of the parent one anyway, so that even isn't really a big  problem, but I suppose if you had a huge amount of page tables, the forked time might be significant.  I don't know.  I have never personally observed any issue with this whatsoever.  And then the worker reload mercy allows you to configure how long to wait before you forcefully  close your workers if for some reason they're holding onto a resource that isn't gracefully  giving up its control.  Then there's dynamic worker scaling.  So this again is not strictly required.  This is a feature that I love because I happen to run multiple WISB services on the same  physical host or in the same virtual machine.  And I also run multiple versions of the same service.  If you see my talk about stateless services, we run multiple versions of the same stateless  services and then basically repry requests to all of them to compare differences and  responses between them.  So if you were to look at one of my machines, this is obviously an exaggeration.  We don't run 500 processes, but we do often run 96 or 128.  And if you have 10 or 20 services with that many processes each, your top or your PS can  be pretty busy.  So dynamic worker scaling allows you to start up with a lower number of workers and then  scale up and down depending on the load in the system.  And it is anticipatory.  Anticipatory?  Anticipative?  I don't know.  It anticipates how much traffic it expects based on how much traffic you're getting at  the moment and generally leaves enough of a buffer that spikes won't outpace it.  You can see in the bottom there's some parameters about the backlog.  That feature in particular makes sure that if it's not scaling up fast enough, it will  create emergency workers to make sure you can handle it.  And again, as far as I can tell, this has never been a problem for us.  It scales fast enough.  It's not like a risk.  It's just another no cost feature that you should use.  It's also pretty sophisticated.  There's a lot of options, a lot of different algorithms.  This is the configuration that I like, but you should definitely, if you're going to  do this, you should understand your application and check out the other algorithms other than  the business one that I have selected.  Okay, hard timeouts.  This is a really powerful feature that I have not seen in other Whiskey hosts.  You Whiskey, if you turn on Hara Kiri, will forcibly kill your worker process if it is  stuck.  I point this out because this happens all the time and this can basically make that a  non-issue.  I mean, when it happens, you should log it and alert on it and you should figure out  why and fix it.  But we had another talk earlier today.  I forget the gentleman's name, but he was talking about using async IOM production  and some of his war stories.  Granted, he was talking about asynchronous cases and it's a little different, but he  explained a situation where he had a spike of traffic come in or conversely, one of his  downstream dependencies was slowing down or reaching its limit and his service basically  backed up with requests.  He had thousands and thousands of requests going on in parallel and they could never  catch up.  Even if traffic went down to zero, it would take hours for everything to recover.  This is what this feature is for.  If a spike of traffic comes in and your system for some reason locks up, maybe not because  of a deadlock, but because of just limited resources, HaraCury will clear all that out  after 60 seconds.  When traffic does recover, your system will too.  Granted, you should understand why that queue situation occurred and you should expand capacity  of the limited resources and make your system more reliable in that regard, but there's  no reason you have to get woken up in the middle of the night every time.  That's what this is for, right?  CYA, protect yourself.  No cause made of strategies because you made a mistake.  We make mistakes.  Use the tools to protect you.  You should use HaraCury.  We use 60 seconds.  That's a long time for most people, but we're doing, like I said, numerical calculations  on mortgage backed securities and other large financial instruments.  Some of our services go much higher than 60.  The lower you can set this, the better because that's basically your period in which you  can have a queue condition.  Once that period turns over, all the stuff requests will clear out.  This is an analog to this.  Hard timeouts are kind of drastic because you're literally kill-9ing your workers.  You really don't want to get to that.  This should have been in the first section, but I put it here so we could contrast it.  This UISC configuration option allows worker processes to receive signals.  Without it, UISC does not allow this to happen.  I have no idea why.  I actually couldn't tell you.  I couldn't find the justification for this.  Hence there's no quote from them.  The reason that you want this is that you might want to receive a signal.  For example, in the Python signal module, you can call signal.alarm and tell the Python  interpreter to wake up this handler if you have a, after a certain amount of time.  One example would be, you set a hard timeout at 60 seconds and you use the signal module  to set a soft timeout at 59 seconds.  Oftentimes I'm serving requests where I have partial responses I can return, even if I  can't return the full thing.  In our system, we use the signal module to wake up any waiting process at 59 seconds,  gather whatever data we have, and return it to the client.  If that fails for some reason, for example, say we have an actual stuff process, this  will kick in, sorry, Hara Kiri will kick in and kill the process to make sure we continue  to operate properly.  So these two things kind of go together, but this is really a default.  You should just set this.  There's really no harm in it.  If no one sends your process signals, then fine.  If you want to, this is another one of those situations where it's like, how good are you  at Googling to figure this out?  This one is very important as well.  So we had a conversation, sorry, a talk at this conference about observability, about  tracing and logging and monitoring, and how observability is more than just the three  things that he talked about, which was logging, metrics, and tracing.  This is another area you can get observability to your software.  By default, when you start a service host, most of them give you this boring output.  That's just the command line that you use to invoke the service, which is literally useless  practically.  I mean, I guess it tells you the Python module that you loaded, but that's really it.  Uwizgi's aware of this, and they give you an option called auto-procname that you can  put in the config file that will do something like this.  It will tell you which process is the master, which process is the workers, and which worker  number it is.  This is still a little naive, right?  What if you have more than one Uwizgi service in the same box?  You might end up with 10 Uwizgi masters, and which one is the one you want.  So you can do a prefix, you can say prop name prefix, and give it a service name.  That's much better, but it's still not the best we can do.  There is an API you can invoke from within your code to specify dynamically what you  want the prefix to be.  Here's an example.  This is contrived.  I'm not actually hosting.  This isn't how we actually do it, but it gets the example across, where you can put context  about what that process is doing.  In this case, you can specify the service, the username, and then the URI they are accessing.  What this helps you with is sometimes you have a problem where there's a lot of slowness  on your hosts, and there's a lot of stuck processes.  Very quickly, but it's running top or PS, you can see that it's a pattern, that it's  all the same user, or that it's all the same URI.  There's something common about that problem, which otherwise would perhaps take you a little  while digging through trace or digging through logs.  You can kind of get a quick snapshot of it.  So it's not a replacement for anything else.  It's just a straight, free, zero-cost additional tool that you can add to your system.  That section I want to talk to you about is additional UISG features.  This is a controversial section, because if you use any of these features, you have now  lost compatibility with other UISG hosts.  You can no longer switch back to G-Unicorn if you use these, because you are now leveraging  UISG specific functionality in your code.  That said, it can still be worth doing.  Honestly, we don't switch hosts that often once we get a system in production.  These features can help you solve or provide a solution to something with much less complexity  than you would be able to otherwise.  It's a trade-off between reducing complexity and having interchangeability of your UISG  hosts.  We end up using some of these, because the reduction in complexity is just tremendous  when you're running a large system.  The first one I want to talk to you about is a Kron and Timer.  It's literally what it sounds like, running Kron's and Timer.  You can say, do something every first of the month, or do something every 20 minutes.  It does not let you do one-time timers.  If you wanted to say, do something in 60 seconds, you should use the signal module.  Why would they write something that already is in the standard library?  I point this out as being useful, because sometimes you want your periodic tasks to  be synchronized with the software version, the code that you're deploying.  Sometimes you have this problem where you want to put it in a new service that has some  functionality you want to run on a timer, and you have to update your Kron system separately.  Now you can deploy those things in an atomic manner.  If you're going forward, this isn't usually too big of a problem, because you can deploy  the new features and then change the Kron timer later.  But if you're going backwards, actually, it could be dangerous, because you've now moved  your service forward.  There's some bug in it.  You want to roll back, but now you have some Kron job that expects that behavior to be  there.  This can help you deploy software in a more atomic fashion.  Also, if you just don't like using Kron or any of those things, you can do it in the  service itself.  You can also do it with a decorator, instead of with doing it at the global level.  So you can put a decorator around a function, and you will call that periodically.  Next thing I want to talk to you about is locks.  This is actually very difficult to do.  If you don't have a framework-supplied lock like this, you have to try and get some code  to run before the pre-forking portion of you, Izgy, forks all your processes and create  a lock in global space and then share it around, or use an external lock in Redis or something.  It's actually a real pain.  I don't think you should write software that needs a lock.  If you have to do this, you probably messed up, but if you do, this is probably a better  way to do it than all of the alternatives.  So I'm just making you aware that it's there, even though I would kind of feel dirty if  I had to use it.  Then there's the cache system.  So this is super cool.  This is a cache that you, Izgy, provides so that all the workers can share information.  You could obviously use Memcache or Redis as well, right?  Like, of course you could.  But then you have to create a separate system to start those things and to monitor them,  so it's kind of nice to have it all packed together.  We use this for rate limiting, actually.  What we do is every time a request comes in, we store in the Izgy cache how many requests  a given user is doing, and then we reject requests that they've reached some threshold.  You might say, well, why don't you have HA, Proxy, or NGINX do that?  Well, the reason is that we want to throttle people in a more sophisticated manner than  one of those Proxies can.  We actually want to look at the HTTP request in pretty detailed ways in order to do that.  So this is a pretty powerful feature.  It's got some gotchas, for example, keys or values that are too big for the cache will  suddenly fail to insert.  So you need to kind of make sure you have a good test suite around it.  But once it's working, it's bulletproof, and it's much simpler than starting up a remote  or an external Memcache or Redis instance to get the same job done.  And then there's mules.  So mules are worker processes that are not workers.  They're not servicing or requesting your clients, but they're there to do stuff.  We use this, for example, to aggregate metrics.  So all the worker processes will send the mules metrics, and then the mules will offload  those metrics to your metrics engine, whether you use Datadog or Metric Tank or Grafana  or whatnot.  So you can put a little decorator on that and say target equals mule.  If you don't say target equals mule, it'll run in just a random worker process, or all  worker processes.  So that's how you specify that.  And then there's ways to send data there as well, which I include in the longer form  article that I posted on tech at bulletinberg.com, if you want to see that.  So that's it.  I am hiring.  I am looking for an SRE.  And just to qualify, when I say an SRE, I mean a distributed systems engineer who doesn't  get work from product people who want features built, whose job it is to optimize, monitor,  and improve the performance of our system and its reliability for the sake of itself  and not to get some featured on for a business guy.  We are a pretty great place to work.  Please talk to me.  I would love to tell you more about the role.  We have some people already doing this role, so you'd be joining a team.  You wouldn't be bootstrapping an organization.  But you'd have to come live in New York, which I know is terrible.  Who wants to live in New York?  That's it.  I think we have a few minutes for questions, if there are any.  Thank you.  Yeah, we have a few minutes for questions.  Again, microphones are there, and there.  Please line up behind the microphone and ask away.  Hello?  Hello?  Hi.  Could you tell me please more about the hierarchy option?  Why should I use it?  Why should I just not set up like worker load time?  So the worker process recycling will, I think, will kick in.  I'm not 100% sure about this, but I think it will kick in and restart workers.  But generally, your worker process reloading is going to have a long time interval.  You might say an hour or a thousand requests.  So if you set up a request, well, it will never reach that, obviously.  If you set up a memory, it will never reach that either.  So it's only the time one that will catch it.  And then it's the question of, do you want to wait an hour or 60 seconds?  That's really it.  To me, I want to wait 60 seconds.  If I wait for that hour to happen, many, many bad things would happen.  This actually happens to people.  You have a bug in your code in every worker deadlocks.  It happens.  So the hierarchy is a way to help prevent that.  Whereas if you use just the normal worker process reloading, the time interval will be  too long.  All right.  Over there.  Hi.  What about lazy apps loading?  So loading app and forking versus forking and the loading apps.  After all, have you been bitten by that?  Or anything?  So we prefer the post fork option.  So we would want to load everything and then fork after.  That's a great question.  I don't think we have yet.  Doing it lazily makes the process reloading take much longer.  Because now you have to wait to load all of your application code, which can take seconds  sometimes.  So we have not yet.  But that's a great feature to be aware of.  If there's something about your application that you can't load it once and fork trivially,  you can enable the lazy apps option and that will prevent that from being a problem.  Usually that kind of stuff happens if you import something and it initializes something  globally that's also like C extension and then you're doomed for lazy stuff.  Absolutely.  Thank you.  So thanks for the talk.  And I wanted to ask about your deployment environment because I noticed you have a huge  number of workers.  And is it containerized?  There are small machines or big machines with a lot of CPUs.  In our particular situation, we generally have dual socket Linux servers, physical servers,  which somewhere around between 20 and 28 cores, generally about half a terabyte to a quarter  of a terabyte of RAM.  We have some virtual deployments on VMs, but we don't really prefer them because we run  a very high duty cycle in these machines.  We're doing like Monte Carlo simulations of mortgage backed securities, which is why I  have so many workers.  And we have a cluster of 500 machines that we run at 100% CPU for 20 hours a day.  And so if we were to go to a virtual environment, it would just cost more money.  We need more machines once you get the overhead there.  So we are capable of running in a containerized environment if we need to, but for money's  sake and efficiency's sake, we run a physical hardware.  Thank you.  Cool.  One last question very quickly.  Do you have any recommendations or gotchas and so on for Emperor mode or would any of  your recommendations here change with that except for, I think, one option you mentioned?  To be totally honest, I don't understand what like Emperor mode and Zerg mode and these  things are for.  I mean, we have a microservice environment where each service can run independently and  can be deployed independently.  So I honestly have a completely ignorant to all of that stuff.  Sorry.  But you kind of almost recommend against using Emperor mode at all.  I don't know why you would use it.  Right.  So I haven't seen the use case for it.  I suspect it's one of these back in the day that was the best that they could do.  Now things have progressed.  Thanks.  All right folks, let's thank all our speaker again.
Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_0_131.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_132_191.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_192_311.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_312_359.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_360_515.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_516_1079.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_1080_1331.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_1332_1871.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_1872_2123.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_2124_2579.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_2580_3563.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_3564_4343.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_4344_5399.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_5400_5675.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_5676_6659.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_6660_7799.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_7800_9383.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_9384_10787.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_10788_12347.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_12348_13895.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_13896_15563.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_15564_17687.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_17688_19475.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_19476_20543.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_20544_23003.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_23004_25355.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_25356_28151.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_28152_28223.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_28224_28379.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_28380_29195.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_29196_29255.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_29256_29915.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_29916_29987.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_29988_30311.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_30312_31511.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_31512_32927.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_32928_34019.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_34020_35483.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_35484_35567.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_35568_35639.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_35640_36635.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_36636_38291.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_38292_39251.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_39252_40115.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_40116_40379.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_40380_40535.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_40536_40583.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_40584_40799.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_40800_41939.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_41940_46523.png Peter_Sperl_-_Configuring_uWSGI_for_Production__The_defaults_are_all_wr_46524_46568.png