Beyond grep: Practical Logging and Metrics
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Schlawack, Hynek
Hynek Schlawack - Beyond grep: Practical Logging and Metrics Knowing that your application is up and running is great. However in order to make informed decisions about the future, you also need to know in what state your application currently is and how its state is developing over time. This talk combines two topics that are usually discussed separately. However I do believe that they have a lot of overlap and ultimately a similar goal: giving you vital insights about your system in production. We'll have a look at their commonalities, differences, popular tools, and how to apply everything in your own systems while avoiding some common pitfalls.
 Why hello everybody. I'm from the Internet. You may or may not have heard of me. I do quite a bit of writing and coding and whatnot. But I have little time and I'm not that interesting. What's more interesting is the way I work. I work for a very small other hosting company and no mind regs are. And the reason why it's interesting to you is because we are big enough such that we need proper metrics and logging systems in place to be able to even function. But on the other hand, we are small enough that we don't have a team who does that for us. We have to do it on the side. And it's just one part of our work. And I think that makes it kind of relatable to you. At least I don't think that Google sent their logging team over here to learn something from me. So to make it more convenient for you, I made a page with all the links, all the concepts, all everything I'm going to mention here. So just relax and listen. And the agenda is three things, basically. I'm going to talk about errors and how to get notified about them. I'm going to talk about metrics and how to know what the hell is going on on your service. I'm going to talk about logging and how to centralize it and whether you even need it. So one question I have, who is happy with their logging and metrics infrastructure? Liar. I'm not promising you happiness. It's computers after all. But maybe we can make it, maybe I can provide you with functional unhappiness, which is nice sometimes. So errors. I'm going to start with them right away because they happen. You have to deal with them. And they are the quickest wins to make. So I'm starting with them while everybody is still fully awake. And again, I have three expectations from my error log in, from my error notification system. And this is really timely notifies. I want to know right away when something happens. I want to be notified only once because this happens to people who use an exception to email logger. So I'm Hinnick and I had once 500 emails from such a thing. So I would also like to have some useful context of my errors because monitoring may tell you that something is broken, but this is not really helpful to have any idea what is broken, what's going on. So obviously there's a huge market to solutions. I'm going to talk about only one of them, which is Sentry. Sentry has a lot of things going on for it. Most importantly, it's owner and famous expert to DBA. David Cramer bought me a Burrito once. So consider it as my full disclosure. But it's also open source software. And it's written in Python using Django. So if you're deploying Python services, you may already know how to deploy it. And if you don't want to do that, there's a paid solution. The plans are pretty affordable, I think. And there's also a free trial and a free plan. So you can be up and running within seconds. So if you don't have any error notifications, you should try it out. So what do you get? You get instant useful notifications by email, but also by Slack, or whatever you want. There are plug-ins for that. Or it contains a trace back and some metadata. And the most interesting button, of course, is the view on Sentry one. A nice touch I find is that those emails have a reply to header set to your whole team. So maybe you're on a train and you see something that exploded. You can just hit reply and give them some hints how to fix it. So the web interface offers much, much more. And my favorite button is this button, because this button is telling me I've saved you 100 emails in your mailbox. So once you've fixed this exception, it gets marked as resolved. And if it happens again, it gets marked as regression and you get your notification again. So basically, it does exactly what you want it to do. As you can see, there's a lot more going on. There's a lot of metadata. It's much of it is collected automatically. So you can think of it like the Django stack trace view that many people still are serving to the customers, but it's just for you. So how do you get your data in there? So the short answer is JSON over HTTP. So you can use it with any language, any framework, even assembly if you want to go through scale. So there are nicer clients for various languages. They have usually the name Raven in it. And the Python one supports both multiple transports, which is how the others are delivered. So using GUM, the new I think I think IO, Twisted, request, and so on and so on. But also multiple integrations, which is basically how data is collected automatically without you doing it explicitly. So for example, logging, you install a logging handler and every exception that arrives there is forward to sentry. You don't basically have to change anything in your code if you're logging errors. For Django, there's great support. There's general whiskey support and nine more. Maybe it's even more since then, since I did the slide. So let's start simple. How do you do it when they love? You instantiate a client using a URL you get from sentry's web interface and then you capture it. You're done. This is how you capture errors and report them into a nice interface. What I personally like is this. For ad hoc tools, which you may or may not have a lot of in your operations, every exception is caught here. That happens in this function, get forward to sentry. So you don't even have to change your functions. You can just add a decorator to it and your errors are caught and forward it. Integrations make it even easier. I've already mentioned that it is built on top of Django. The author's thing or two about Django. The support is the best as far as I could tell. You add a single line, you get all of 500 reported and you can import a client from anywhere. That's that. We are already done. Deploy sentry or give David a few bucks so it can be via Miya and the burrito. Install Raven and add a few lines to your project. If you don't have error notifications, and I really have to stress that you are missing errors. Your customers are seeing those errors. You are not. You are losing customers. Get something done. To make it even easier, David was nice enough to issue a nice promo code, which is, I think, 100 bucks. I'm not getting anything out of it, but if you want to try it, there you go. There we go. There we go to metrics. What are metrics? Metrics are numbers in a database. That makes them a time series data because they are associated with a time stamp. They are basically the difference between guessing and knowing. If you want to make decisions, you need facts. I think it's an accepted wisdom. Otherwise, you spend weeks and months building something that's useless or even harmful. Metrics are those facts. We will give them a quick role. I would just think between system and application metrics. System metrics are something you observe on the server, like the load or how much traffic is going through. Very important should be collected using something like collecting, but not really part of my talk. What I'm talking about is app metrics, which is something you measure within your app. In the simplest metric, you can have our counters. Something happens and you increase an integer, which is pretty fast, even in Python. Then timers. Maybe you want to know how long your database queries take. Maybe you want to know how long your requests take on average. And finally, there's gauges, which I find undervalued because they are really useful if you want to debug something. They are just numbers, which you want to keep track of. So it can be the number of customers online or the number of connections in a connection pool. Think like that. I find them super helpful. There are much more, but these three are, in my opinion, the most important one. So what can you do with metrics? So we said there are time series of data so you can plot them. And such a plot gives you a lot of information that bare numbers don't. So for example, you see development over time. So you can tell that you are running at 99% capacity every day at 12 p.m. And if you don't do anything, it might fall over next week when you get one more customer. And you also see trends. So you can tell if you need to buy the, if you will have to scale out today, tomorrow, next week, or maybe never because you are using customers because you don't have proper error handling. So when you have graphs, you can correlate them so you can see, like, requests per second versus latency. How much requests per second can you handle? And since they are just numbers, you can do math on them. So, for example, if you have a graph of a counter, it's just a raising line. It's not really interesting. But you take the derivation of that and you have requests per second. If you have timers, taking the average is not very useful, but person types are very interesting. So for example, what is the average request time for the slowest 0.01% of your customers? Because what if every 1,000 requests takes one minute? You wouldn't know from the average because it gets smoothed out by the other 999. But this customer gets regularly, for some reason, one minute request, he may leave you to. Thing is, math is hard. The average human has one ovary and one testicle, which is true, but it's not very useful information. And you can do the same mistake with your system or your app metrics. So unless you know what exponentially decaying reservoirs are, use tools by people who do know what it is. So one I think you can also do monitoring on top of metrics, of course, because you can set a hard limit for acceptable latency. If the threshold is exceeded, ring the bell. Zero rates. If you have a busy application, you usually have always some kind of errors. If they go out of whack, something is going on. And it's actually true for any kind of anomaly. If, for example, benign errors like 404 or 401s go out of whack, there's something going on you should investigate. And there's actually a whole stack called Kail by Etsy that is just made for finding anomalies like that. So we've said they're living in a database but probably not SQLite. So what we're looking for are so-called time series databases, which have various features like special querying and everything. One of the most important ones is that you have a roll-up of your data, which means you have various resolutions of your data for the past, because you probably don't have enough storage to store a second resolution of all your metrics for the past year that might get expensive really fast, even if you're big hard disks. So you usually smooth it out somehow. So you have like, you want to know what the average load was a year ago per day, but you want to know it very precise for the past hour. So I'm going to introduce it to three. The first one is paid and hosted. And it's really, really nice. You can get started immediately by using curl and uptime and you have a curve of your system. I've done that. We started it like that too. The graphs are beautiful. There's a lot of goodies. It's a lot of fun to work with it. If you want to host them yourself, the current Inherfond gorilla is still graphite, which has been popularized by Etsy too, and it's written in Python. The front is in Django. The back end is in Twisted called Carbon. It's finally in trusty. You don't have to build it yourself. And you can say that it's a widely supported standard nowadays. So the network protocol of Carbon is supported by other applications too, just for compatibility. So the thing is, it is a little bit long on a tooth. So the storage configuration I just talked about, the roll-ups and limits, it's a bit finicky. And it might be not the most pretty interface you've seen today. It's XJS. If you don't have a pleasure to work with it yet, this is what happens when programmers build interfaces. I mean, it's open source of there, so I'm not complaining, but it's clearly kind of a problem. But that one is solved by Grafana, which is something that's really just to build pretty dashboards for Graphite. And once you install it, you will probably lose a few hours to it because it's so much fun to play with it, and it looks so good. And Grafana also supports InflexDB, which is the next generation time series database written in Go, because that's what you do nowadays. It has a company behind it that sells hosting, so let's hope they don't pull off FoundationDB. And it is used by Heroku, so it's not an obscure toy for nerds, but it is in production. It looks better. It's easier to manage its storage. You can tag values, which anyone appreciate would ever put the server names into their metrics names that you've seen in the slide before. Now you don't have to put any tag on a value and have clean names. It offers a SQL-like query language to those metrics, and a Graphite front-end, which means if you're running Graphite right now, you can point your tools just to InflexDB, and it should work, but it's computers, so I'm not sure. If you start on today, I would recommend to look into it first. If you run Graphite and are functionally unhappy only, then I would not abandon ship so quickly. It's not that big of a deal. So, collecting, how do we get the data into these databases? And there are basically two approaches. The one is that you aggregate externally. So something happens, and you send out a UDP packet to StatsD, or protocol buffers to RIMAN. StatsD is older, comes also from the Etsy ecosystem. Simple to use, simple to set up. RIMAN is by a super smart person, and it's configured in Clojure, so you probably have to be also super smart to use it. The good thing is it has no state. It's super simple to set up and to use. The bad thing is you have no direct introspection. So you need at least one more service to even see what metrics are coming out of your system. In the case of StatsD, you need even two, because StatsD does only aggregation, and then forwards it to graphite. With RIMAN, you get at least a kind of dashboard. So the second approach is that you aggregate your metrics within your application, and then deliver it to your metrics database. This approach has been polarized by CodaHale and his talk, metrics, metrics everywhere, which you totally should watch if you want to get into metrics. It's super interesting. It's super funny to watch. And this one gives you immediate insight into your application. You get some kind of dashboard out of your application, and this is useful both in development and in production just as well. Of course, you've got state. State is bad. State means bugs. But I personally prefer the second approach, because it's more practical. So the question is how will you do it in Python? So for StatsD, there's a gazillion Python clients, pick one. They work all the same. You instantiate a client with a URL, and you should packets around and don't look at return values, because they are not, it's UDP. Everything's going to be okay or not, because if your system is burning, UDP might not be the best way to message your state. So the only known working solution in AppMetrics to me is Scales. So it comes with a plethora of stats, but you have to set it up. These are the two I use most. The meter stat is for something that happens per second. So basically a derived counter, and a PMF stat is a timer. Nothing else. So how do you use it? For metering, you just call mark on it, and for timing, it is a context manager where you do something inside of it and you're done. Now, by doing this alone, you get a nice web dashboard out of the app. This is the metering. You've got already the average for the past minute, five minutes and 15 minutes. Even nicer is the thing you get out of your timing, because you get your person tiles for free, plus some more nice statistics. And all this data you also get as JSON, so you can collect it from, you can collect D or whatever. I personally use the graphite periodic pusher that comes with Scales. You just define the period, how often you should just send out the metrics, and you're done. We are done. You know how to collect metrics and how to store them. Now, we come to logging. In an ideal world, we wouldn't be logging, because you want to know about errors, which we now have error-centric, and you want to know the state of your system, which are metrics. So there are people like Armand Roenacker who just refuse to log anything. I personally cannot get away with that, simply because we needed some kind of bookkeeping, because the customer calls us, they always lied to us. They always stated they did not log into the server. They did not change that file. And we have, we need a way to double check what they are telling us. And that's usually not me. That's someone from support. And those people usually don't have the SSH keys to our servers. So this data should be somewhere searchable in a central place. So we are talking about centralized logging. And I can talk about centralized logging and not mention Splunk. And please, my, see there are more moneybags next to the name than on the other slides. It is for a reason, because this is enterprise software. And it's not just one web interface. It's a versatile platform. They have literally an app store. It works both on premise and in the cloud. It's great if you can afford it, but it is enterprise software. So the home page is full of PDF white papers. There's a lot of webinars for you to attend if you know that kind of things. So more down to earth, there's PaperTrail and Logly, which I have heard good and bad things about both. So it's a matter of taste. I'm sure you're going to be reasonably happy with any of them if you choose. And if you want to save your log files on foreign servers, which I personally don't. And that's why we are running Elk. Probably heard about it. It's currently the most popular stack. And it consists of Elasticsearch, LogStash, and Kibana. Let me just quickly show you how it works together. So we have servers that are generating log files. Those log files somehow get into LogStash, which parses them, adds meaning to it, and saves it into Elasticsearch, which is a database that is easily searchable and easily clusterable. And now the data is in there, you can view it using Kibana, which is a web interface to all these things. Yeah. And that's all. That's the Elk. There's a similar solution called GreyLog. It also uses Elasticsearch for storage and search. But Kibana is only a view on Elasticsearch, but GreyLog does more. Because, and I'm quoting here, Elasticsearch is not a log management system. So overall, it's a bit more integrated. They do more. But I'm personally not particularly fond of having a Mac vendor in my infrastructure. So you have to decide yourself. I haven't found a compelling reason to switch from Elk. But I'm sure there's someone. If you have any questions about Elk, Honza Kral is somewhere, probably in some pub here. And he works for Elastic, the company behind Elk, so he will be happy to answer all your questions. And he's also the maintainer of the client to Elasticsearch. So one more thing, Kibana is much more than just a web grab. They have a lot of nice things going on, like geo stuff and everything. So there's a lot of things to find. Now let's come to the finicky part. How do you get your data input? How do you produce it? So I'm going to say this should be the goal for you. A timestamp and something machine readable with as much useful context as possible. Because that makes configuration really simple. You literally tell LogStash there's a timestamp and JSON. LogStash will figure it out. Of course it's just one line, but I thought you might find it more readable in that size. So how do we get there? It's a matter of context and format. So you want to look out everything important and you want to format it in a machine readable way. And if you try to achieve that with the standard tools, you might find like I did, it's rather tedious. So I wrote something on my own called StratLock. Does anyone know StratLock? Okay, let's change this. So StratLock is not a logging system. It's not a replacement for LogBook. It's not a replacement for StratLock. Instead it gives you a bound logger that wraps your logger. So if you're going to ask me, does StratLock work with X, the answer is yes. Now it also gives you a context which you can bind key value pairs to it. And once you decide to log this event out, this context you saved before is combined with the new key value pairs to one event dictionary. And this event dictionary is run through a chain of processors which are just callables. A function that gets a dictionary in returns a dictionary. Nothing else. The last processor, the return value of it is passed into the original logger. So if you're using the standard logging from the standard library, you would return a string. For example, a JSON string or whatever format you want and return XML for all I care. StratLock comes with JSON and key value pairs for a matter. So the thing about the processor is it's really cool because it's really just callables. You can do whatever you want. You can plug data out of it. You can collect metrics from your log entries. You can report errors to sentry from them and enraged with the context you've collected. This is really nice. So this handles both context and format. And let me give you a few examples because it's a bit abstract. So simple case. You get a logger which is everything is pretty much configurable. And now you can log using key value pairs. You can stop writing pros. And if you're any like me, I hated writing pros before, but what's even worse is parsing pros. So this output is completely configurable. This is the default which is just key value pairs which is human readable in development. So I find this is already a huge progress over Stunner library, but you can do more. So this is incremental data binding. So again, you get your logger. And now you can just start binding key value pairs to the logger. And this lock object is a new object every single time. This is immutable data. And we have no mutable site at all. Ask your Haskell friends. It's a great property to have. So in the end, everything you bound to the logger gets logged along with the event. Again, output is configurable. And please notice that you don't care at all how the data is represented within your business code. That's something that you care about somewhere else in a processor or in your logging module, but not in your business code. You just bind key values and just log them out. So now maybe even more practical, how do you use it in practice? So this is a pyramid view, a very simple one, but it would probably work the same with any other. So at the beginning, you bind a request object to your logger, and then you lock something out. And how do you do something useful with that object? So you write a processor that extracts the data. So you try to remove the request from the event. If there's one, so you've removed it, but now you add some data from the request, like the IP address of your client or the user ID of the user. And you return the new dictionary. And this is what you get out of it. In case you have a JSON format installed. Again, you did not care about what you want to log out in your view. That's something that you decide elsewhere. That's all I'm going to say to StratLog. If you have any questions, just talk to me. I'm pretty proud of that one. Now to something slightly sadder. Let's talk about standard library slogging. I'm going to say this is all you should do and ignore all the rest. Just log to standard out and handle the logs outside. Because UNIX had over 40 years to develop solid logging tools and there's absolutely no need for us Python people to reinvent the wheel like date stamping or leuprotation. We are doing it worse. Stop doing it. Just go to standard out. Also, I've heard that it's not that much fun to use. But you be the judge. So now we have structured data on standard out. What do you do next? Send it into a file. Or send it to syslog or any other queue like Kafka. Pipe it into a logging agent like log-forwarder. You can do whatever you want. It's just a pipe. I'm personally a bit paranoid. Because I don't log a lot. But what I log is important to me. So I don't want to lose any log entry. No network in this world is as reliable as X4. I save everything in a file. This file is rotated for 48 hours and those entries are deleted. I ship it from there, from this file. So while I do not want to have to use grep, I still want to retain the reliability of greping through files on a file system. So let me put it all together. So I use struct log to bind data to log things out. Struct log makes it a JSON string which goes into logging and log in and sends it to standard out. Now, I use run it to run my processes. It doesn't really matter what you're using. But run it comes with a demand that takes standard out, add a timestamp to it and write it to a file. Now my log entry is safe. This file is watched by log-forwarder formerly known as lumberjack. And it sends to log-parsed. Log-parsed it, sends it into elastic search. Logging is solved. So, yeah, we are done here. Let's get some pinchos. We are not. We have three nice components. We forgot about the pragmatic part. How do you put those three things together without making it gross? Because this is gross. You can barely see the logic hidden in the jungle of reporting, measuring, counting and whatnot. So I want it to look like this, which is much nicer. Something happens, I tell the log system about it and I'm done. Of course, that's not always possible. But I would really try to hard. I would really like to try hard to get somewhere there. So with errors, it's pretty easy, I dare say. So either use some handler that comes with a sentry, either just logging or if you're using running Django, using the Django app there, using, or just track log. That's what I do when I use a pyramid. I just plug my errors out of the logging stream and I can drop entries if something is not interesting. And in that app, there's usually also a way to define error views. And this is really, really cool because, again, pyramid, you get the exception and the request object. Now, you serve back the error ID, which is served from sentry. So now, when your customer calls you complaining about errors, they can exactly tell you the error ID and you can look the error ID up and you have the exception that the customer saw. And this is so great that we've seen something that's even rarer than a white rhino, which is a happy armored runnaker. So although I have to say, since I made a slide, he joined sentry, so take it with a grain of salt, but still. So onto metrics. Most metrics can be observed from the outside and outside can mean outside of your views, outside of your app, outside of your server even. So maybe let's have a look at whiskey containers. The two major ones have both knobs that will help you with that. So G unicorn offers stat C integration right there. So you add one command line option and you have average request times in your stat C and in your graphite. You don't have to change your code at all. Micro whiskey, as usual, goes far, far, far further. They, of course, have stat C too. They have direct carbon, aka graphite support. They have a whole metric subsystem, including nightmare inducing things like SNMP. So you get your stuff done with that. And with this, you get a big picture of the state of your application without even touching your apps. So go for it. Then you can write middleware. Middleware is no dark magic. Again, pyramid. This is a tween, which is a very awkward contraction of between. And this is called on every request that comes in. So you have the request object. In this case, we just measure the time. But you can, of course, look at the data within the request object and start splitting up your data depending on the view or some argument that you're passing into your view. Probably don't have to because there are things like pyramid stat C that already do that for you, but you always have the possibility to do things from within your app but outside of your actual logic. Then, of course, you can extract data from logs. Because if you lock something out, you shouldn't have to also count it or measure it. So LogStash will do that for you. It supports all major metrics backends. The drawback is that you have to change the configuration of LogStash, which may or may not be a problem for you. It's not a really problem to me, but it adds friction, which I do not like, so I do not do that. I don't want to annoy those people who are responsible for that to fix it for me because I've added a new metric. Of course, you can do the extract log. It's what I do. You can just count events by their names and you already have something useful. Okay. Finally, you can also leverage monitoring, which is even for the outside. Any monitoring system has some support for metrics numbers. In the worst case, you just measure the time it takes to execute a check and save it. So you get a really external view of your data, external view of the behavior of your apps, which is not very precise, of course, but sometimes it's useful to see how your system feels from outside boundary and not from within your availability zone or your computing center. Okay. So what's left? What do you have to yourself? So if you want to measure code path, you probably have to add some code to your business logic. For example, database queries. Or if you have certain major use cases, like a view that sometimes uses only cache data and sometimes hits a database, it's not very useful to average those two numbers. Not to say it's completely useless. So you may want to split that up. And of course, gauges, if you want to expose numbers from within your application, you will probably have to touch your application in some way. And now we are really done. So what did you learn? Proper error logging is important. Sentry is awesome. Metrics are important. Influx TV is probably the future. Graphite is the present. Use whatever you want from those two. Centralized logging saves you a lot of pain. And maybe you need it. Elk will have your back. Strucklock will help you to get your data there. And now you know how to use all of them with Python without the gross code duplication. So I hope everyone learned something. So go forth and measure. Study the talk page. Follow me on Twitter and tell your German-speaking friends to get their domains from Vario Media. Thank you. And I'm sorry. I'm not taking any questions because whenever I did, I completely misunderstood the question and said something very stupid. So if you have any questions, I will be outside. I'm here through Sunday. I will be at the conference. I will be at lunch. Just check me out. I'm happy to answer any questions. Thank you.
 Why hello everybody. I'm from the Internet. You may or may not have heard of me. I do  quite a bit of writing and coding and whatnot. But I have little time and I'm not that interesting.  What's more interesting is the way I work. I work for a very small other hosting company  and no mind regs are. And the reason why it's interesting to you is because we are big enough  such that we need proper metrics and logging systems in place to be able to even function.  But on the other hand, we are small enough that we don't have a team who does that for  us. We have to do it on the side. And it's just one part of our work. And I think that  makes it kind of relatable to you. At least I don't think that Google sent their logging  team over here to learn something from me. So to make it more convenient for you, I made  a page with all the links, all the concepts, all everything I'm going to mention here.  So just relax and listen. And the agenda is three things, basically. I'm going to talk  about errors and how to get notified about them. I'm going to talk about metrics and  how to know what the hell is going on on your service. I'm going to talk about logging  and how to centralize it and whether you even need it. So one question I have, who is happy  with their logging and metrics infrastructure? Liar. I'm not promising you happiness. It's  computers after all. But maybe we can make it, maybe I can provide you with functional  unhappiness, which is nice sometimes. So errors. I'm going to start with them right  away because they happen. You have to deal with them. And they are the quickest wins  to make. So I'm starting with them while everybody is still fully awake. And again, I have three  expectations from my error log in, from my error notification system. And this is really  timely notifies. I want to know right away when something happens. I want to be notified  only once because this happens to people who use an exception to email logger. So I'm  Hinnick and I had once 500 emails from such a thing. So I would also like to have some  useful context of my errors because monitoring may tell you that something is broken, but  this is not really helpful to have any idea what is broken, what's going on. So obviously  there's a huge market to solutions. I'm going to talk about only one of them, which is Sentry.  Sentry has a lot of things going on for it. Most importantly, it's owner and famous expert  to DBA. David Cramer bought me a Burrito once. So consider it as my full disclosure. But  it's also open source software. And it's written in Python using Django. So if you're deploying  Python services, you may already know how to deploy it. And if you don't want to do that,  there's a paid solution. The plans are pretty affordable, I think. And there's also a free  trial and a free plan. So you can be up and running within seconds. So if you don't have  any error notifications, you should try it out. So what do you get? You get instant useful  notifications by email, but also by Slack, or whatever you want. There are plug-ins for  that. Or it contains a trace back and some metadata. And the most interesting button,  of course, is the view on Sentry one. A nice touch I find is that those emails have a reply  to header set to your whole team. So maybe you're on a train and you see something that  exploded. You can just hit reply and give them some hints how to fix it. So the web interface  offers much, much more. And my favorite button is this button, because this button is telling  me I've saved you 100 emails in your mailbox. So once you've fixed this exception, it gets  marked as resolved. And if it happens again, it gets marked as regression and you get your  notification again. So basically, it does exactly what you want it to do. As you can  see, there's a lot more going on. There's a lot of metadata. It's much of it is collected  automatically. So you can think of it like the Django stack trace view that many people  still are serving to the customers, but it's just for you. So how do you get your data  in there? So the short answer is JSON over HTTP. So you can use it with any language,  any framework, even assembly if you want to go through scale. So there are nicer clients  for various languages. They have usually the name Raven in it. And the Python one supports  both multiple transports, which is how the others are delivered. So using GUM, the new  I think I think IO, Twisted, request, and so on and so on. But also multiple integrations,  which is basically how data is collected automatically without you doing it explicitly. So for example,  logging, you install a logging handler and every exception that arrives there is forward  to sentry. You don't basically have to change anything in your code if you're logging errors.  For Django, there's great support. There's general whiskey support and nine more. Maybe  it's even more since then, since I did the slide. So let's start simple. How do you do  it when they love? You instantiate a client using a URL you get from sentry's web interface  and then you capture it. You're done. This is how you capture errors and report them  into a nice interface. What I personally like is this. For ad hoc tools, which you may or  may not have a lot of in your operations, every exception is caught here. That happens  in this function, get forward to sentry. So you don't even have to change your functions.  You can just add a decorator to it and your errors are caught and forward it. Integrations  make it even easier. I've already mentioned that it is built on top of Django. The author's  thing or two about Django. The support is the best as far as I could tell. You add a  single line, you get all of 500 reported and you can import a client from anywhere. That's  that. We are already done. Deploy sentry or give David a few bucks so it can be via Miya  and the burrito. Install Raven and add a few lines to your project. If you don't have  error notifications, and I really have to stress that you are missing errors. Your customers  are seeing those errors. You are not. You are losing customers. Get something done. To make  it even easier, David was nice enough to issue a nice promo code, which is, I think, 100  bucks. I'm not getting anything out of it, but if you want to try it, there you go. There  we go. There we go to metrics. What are metrics? Metrics are numbers in a database. That makes  them a time series data because they are associated with a time stamp. They are basically the  difference between guessing and knowing. If you want to make decisions, you need facts.  I think it's an accepted wisdom. Otherwise, you spend weeks and months building something  that's useless or even harmful. Metrics are those facts. We will give them a quick role.  I would just think between system and application metrics. System metrics are something you  observe on the server, like the load or how much traffic is going through. Very important  should be collected using something like collecting, but not really part of my talk. What I'm  talking about is app metrics, which is something you measure within your app. In the simplest  metric, you can have our counters. Something happens and you increase an integer, which  is pretty fast, even in Python. Then timers. Maybe you want to know how long your database  queries take. Maybe you want to know how long your requests take on average. And finally,  there's gauges, which I find undervalued because they are really useful if you want to debug  something. They are just numbers, which you want to keep track of. So it can be the number  of customers online or the number of connections in a connection pool. Think like that. I find  them super helpful. There are much more, but these three are, in my opinion, the most important  one. So what can you do with metrics? So we said there are time series of data so you  can plot them. And such a plot gives you a lot of information that bare numbers don't.  So for example, you see development over time. So you can tell that you are running at 99%  capacity every day at 12 p.m. And if you don't do anything, it might fall over next week  when you get one more customer. And you also see trends. So you can tell if you need to  buy the, if you will have to scale out today, tomorrow, next week, or maybe never because  you are using customers because you don't have proper error handling. So when you have  graphs, you can correlate them so you can see, like, requests per second versus latency.  How much requests per second can you handle? And since they are just numbers, you can do  math on them. So, for example, if you have a graph of a counter, it's just a raising  line. It's not really interesting. But you take the derivation of that and you have requests  per second. If you have timers, taking the average is not very useful, but person types  are very interesting. So for example, what is the average request time for the slowest  0.01% of your customers? Because what if every 1,000 requests takes one minute? You wouldn't  know from the average because it gets smoothed out by the other 999. But this customer gets  regularly, for some reason, one minute request, he may leave you to. Thing is, math is hard.  The average human has one ovary and one testicle, which is true, but it's not very useful information.  And you can do the same mistake with your system or your app metrics. So unless you know what  exponentially decaying reservoirs are, use tools by people who do know what it is. So  one I think you can also do monitoring on top of metrics, of course, because you can  set a hard limit for acceptable latency. If the threshold is exceeded, ring the bell.  Zero rates. If you have a busy application, you usually have always some kind of errors.  If they go out of whack, something is going on. And it's actually true for any kind of  anomaly. If, for example, benign errors like 404 or 401s go out of whack, there's something  going on you should investigate. And there's actually a whole stack called Kail by Etsy  that is just made for finding anomalies like that. So we've said they're living in a database  but probably not SQLite. So what we're looking for are so-called time series databases, which  have various features like special querying and everything. One of the most important  ones is that you have a roll-up of your data, which means you have various resolutions of  your data for the past, because you probably don't have enough storage to store a second  resolution of all your metrics for the past year that might get expensive really fast,  even if you're big hard disks. So you usually smooth it out somehow. So you have like, you  want to know what the average load was a year ago per day, but you want to know it very  precise for the past hour. So I'm going to introduce it to three. The first one is paid  and hosted. And it's really, really nice. You can get started immediately by using curl  and uptime and you have a curve of your system. I've done that. We started it like that too.  The graphs are beautiful. There's a lot of goodies. It's a lot of fun to work with it.  If you want to host them yourself, the current Inherfond gorilla is still graphite, which  has been popularized by Etsy too, and it's written in Python. The front is in Django.  The back end is in Twisted called Carbon. It's finally in trusty. You don't have to  build it yourself. And you can say that it's a widely supported standard nowadays. So the  network protocol of Carbon is supported by other applications too, just for compatibility.  So the thing is, it is a little bit long on a tooth. So the storage configuration I just  talked about, the roll-ups and limits, it's a bit finicky. And it might be not the most  pretty interface you've seen today. It's XJS. If you don't have a pleasure to work with  it yet, this is what happens when programmers build interfaces. I mean, it's open source  of there, so I'm not complaining, but it's clearly kind of a problem. But that one is  solved by Grafana, which is something that's really just to build pretty dashboards for  Graphite. And once you install it, you will probably lose a few hours to it because it's  so much fun to play with it, and it looks so good. And Grafana also supports InflexDB,  which is the next generation time series database written in Go, because that's what you do  nowadays. It has a company behind it that sells hosting, so let's hope they don't pull  off FoundationDB. And it is used by Heroku, so it's not an obscure toy for nerds, but  it is in production. It looks better. It's easier to manage its storage. You can tag  values, which anyone appreciate would ever put the server names into their metrics names  that you've seen in the slide before. Now you don't have to put any tag on a value and  have clean names. It offers a SQL-like query language to those metrics, and a Graphite  front-end, which means if you're running Graphite right now, you can point your tools  just to InflexDB, and it should work, but it's computers, so I'm not sure. If you start  on today, I would recommend to look into it first. If you run Graphite and are functionally  unhappy only, then I would not abandon ship so quickly. It's not that big of a deal. So,  collecting, how do we get the data into these databases? And there are basically two approaches.  The one is that you aggregate externally. So something happens, and you send out a UDP  packet to StatsD, or protocol buffers to RIMAN. StatsD is older, comes also from the Etsy  ecosystem. Simple to use, simple to set up. RIMAN is by a super smart person, and it's  configured in Clojure, so you probably have to be also super smart to use it. The good  thing is it has no state. It's super simple to set up and to use. The bad thing is you  have no direct introspection. So you need at least one more service to even see what  metrics are coming out of your system. In the case of StatsD, you need even two, because  StatsD does only aggregation, and then forwards it to graphite. With RIMAN, you get at least  a kind of dashboard. So the second approach is that you aggregate your metrics within  your application, and then deliver it to your metrics database. This approach has been  polarized by CodaHale and his talk, metrics, metrics everywhere, which you totally should  watch if you want to get into metrics. It's super interesting. It's super funny to watch.  And this one gives you immediate insight into your application. You get some kind of dashboard  out of your application, and this is useful both in development and in production just  as well. Of course, you've got state. State is bad. State means bugs. But I personally  prefer the second approach, because it's more practical. So the question is how will you  do it in Python? So for StatsD, there's a gazillion Python clients, pick one. They  work all the same. You instantiate a client with a URL, and you should packets around  and don't look at return values, because they are not, it's UDP. Everything's going to be  okay or not, because if your system is burning, UDP might not be the best way to message your  state. So the only known working solution in AppMetrics to me is Scales. So it comes  with a plethora of stats, but you have to set it up. These are the two I use most. The  meter stat is for something that happens per second. So basically a derived counter, and  a PMF stat is a timer. Nothing else. So how do you use it? For metering, you just call  mark on it, and for timing, it is a context manager where you do something inside of it  and you're done. Now, by doing this alone, you get a nice web dashboard out of the app.  This is the metering. You've got already the average for the past minute, five minutes  and 15 minutes. Even nicer is the thing you get out of your timing, because you get your  person tiles for free, plus some more nice statistics. And all this data you also get  as JSON, so you can collect it from, you can collect D or whatever. I personally use  the graphite periodic pusher that comes with Scales. You just define the period, how often  you should just send out the metrics, and you're done. We are done. You know how to  collect metrics and how to store them. Now, we come to logging. In an ideal world, we  wouldn't be logging, because you want to know about errors, which we now have error-centric,  and you want to know the state of your system, which are metrics. So there are people like  Armand Roenacker who just refuse to log anything. I personally cannot get away with that, simply  because we needed some kind of bookkeeping, because the customer calls us, they always  lied to us. They always stated they did not log into the server. They did not change that  file. And we have, we need a way to double check what they are telling us. And that's  usually not me. That's someone from support. And those people usually don't have the SSH  keys to our servers. So this data should be somewhere searchable in a central place. So  we are talking about centralized logging. And I can talk about centralized logging and  not mention Splunk. And please, my, see there are more moneybags next to the name than on  the other slides. It is for a reason, because this is enterprise software. And it's not  just one web interface. It's a versatile platform. They have literally an app store. It works  both on premise and in the cloud. It's great if you can afford it, but it is enterprise  software. So the home page is full of PDF white papers. There's a lot of webinars for  you to attend if you know that kind of things. So more down to earth, there's PaperTrail  and Logly, which I have heard good and bad things about both. So it's a matter of taste.  I'm sure you're going to be reasonably happy with any of them if you choose. And if you  want to save your log files on foreign servers, which I personally don't. And that's why we  are running Elk. Probably heard about it. It's currently the most popular stack. And  it consists of Elasticsearch, LogStash, and Kibana. Let me just quickly show you how it  works together. So we have servers that are generating log files. Those log files somehow  get into LogStash, which parses them, adds meaning to it, and saves it into Elasticsearch,  which is a database that is easily searchable and easily clusterable. And now the data is  in there, you can view it using Kibana, which is a web interface to all these things. Yeah.  And that's all. That's the Elk. There's a similar solution called GreyLog. It also uses  Elasticsearch for storage and search. But Kibana is only a view on Elasticsearch, but  GreyLog does more. Because, and I'm quoting here, Elasticsearch is not a log management  system. So overall, it's a bit more integrated. They do more. But I'm personally not particularly  fond of having a Mac vendor in my infrastructure. So you have to decide yourself. I haven't  found a compelling reason to switch from Elk. But I'm sure there's someone. If you have  any questions about Elk, Honza Kral is somewhere, probably in some pub here. And he works for  Elastic, the company behind Elk, so he will be happy to answer all your questions. And  he's also the maintainer of the client to Elasticsearch. So one more thing, Kibana is  much more than just a web grab. They have a lot of nice things going on, like geo stuff  and everything. So there's a lot of things to find. Now let's come to the finicky part.  How do you get your data input? How do you produce it? So I'm going to say this should  be the goal for you. A timestamp and something machine readable with as much useful context  as possible. Because that makes configuration really simple. You literally tell LogStash  there's a timestamp and JSON. LogStash will figure it out. Of course it's just one line,  but I thought you might find it more readable in that size. So how do we get there? It's  a matter of context and format. So you want to look out everything important and you want  to format it in a machine readable way. And if you try to achieve that with the standard  tools, you might find like I did, it's rather tedious. So I wrote something on my own called  StratLock. Does anyone know StratLock? Okay, let's change this. So StratLock is not a  logging system. It's not a replacement for LogBook. It's not a replacement for StratLock.  Instead it gives you a bound logger that wraps your logger. So if you're going to ask me,  does StratLock work with X, the answer is yes. Now it also gives you a context which  you can bind key value pairs to it. And once you decide to log this event out, this context  you saved before is combined with the new key value pairs to one event dictionary. And  this event dictionary is run through a chain of processors which are just callables. A  function that gets a dictionary in returns a dictionary. Nothing else. The last processor,  the return value of it is passed into the original logger. So if you're using the standard  logging from the standard library, you would return a string. For example, a JSON string  or whatever format you want and return XML for all I care. StratLock comes with JSON  and key value pairs for a matter. So the thing about the processor is it's really cool because  it's really just callables. You can do whatever you want. You can plug data out of it. You  can collect metrics from your log entries. You can report errors to sentry from them  and enraged with the context you've collected. This is really nice. So this handles both  context and format. And let me give you a few examples because it's a bit abstract. So  simple case. You get a logger which is everything is pretty much configurable. And now you  can log using key value pairs. You can stop writing pros. And if you're any like me, I  hated writing pros before, but what's even worse is parsing pros. So this output is completely  configurable. This is the default which is just key value pairs which is human readable  in development. So I find this is already a huge progress over Stunner library, but  you can do more. So this is incremental data binding. So again, you get your logger. And  now you can just start binding key value pairs to the logger. And this lock object is a new  object every single time. This is immutable data. And we have no mutable site at all.  Ask your Haskell friends. It's a great property to have. So in the end, everything you bound  to the logger gets logged along with the event. Again, output is configurable. And please  notice that you don't care at all how the data is represented within your business code.  That's something that you care about somewhere else in a processor or in your logging module,  but not in your business code. You just bind key values and just log them out. So now maybe  even more practical, how do you use it in practice? So this is a pyramid view, a very  simple one, but it would probably work the same with any other. So at the beginning,  you bind a request object to your logger, and then you lock something out. And how do  you do something useful with that object? So you write a processor that extracts the  data. So you try to remove the request from the event. If there's one, so you've removed  it, but now you add some data from the request, like the IP address of your client or the  user ID of the user. And you return the new dictionary. And this is what you get out of  it. In case you have a JSON format installed. Again, you did not care about what you want  to log out in your view. That's something that you decide elsewhere. That's all I'm  going to say to StratLog. If you have any questions, just talk to me. I'm pretty proud  of that one. Now to something slightly sadder. Let's talk about standard library slogging.  I'm going to say this is all you should do and ignore all the rest. Just log to standard  out and handle the logs outside. Because UNIX had over 40 years to develop solid logging  tools and there's absolutely no need for us Python people to reinvent the wheel like  date stamping or leuprotation. We are doing it worse. Stop doing it. Just go to standard  out. Also, I've heard that it's not that much fun to use. But you be the judge. So now  we have structured data on standard out. What do you do next? Send it into a file. Or send  it to syslog or any other queue like Kafka. Pipe it into a logging agent like log-forwarder.  You can do whatever you want. It's just a pipe. I'm personally a bit paranoid. Because  I don't log a lot. But what I log is important to me. So I don't want to lose any log entry.  No network in this world is as reliable as X4. I save everything in a file. This file  is rotated for 48 hours and those entries are deleted. I ship it from there, from this  file. So while I do not want to have to use grep, I still want to retain the reliability  of greping through files on a file system. So let me put it all together. So I use struct  log to bind data to log things out. Struct log makes it a JSON string which goes into  logging and log in and sends it to standard out. Now, I use run it to run my processes.  It doesn't really matter what you're using. But run it comes with a demand that takes  standard out, add a timestamp to it and write it to a file. Now my log entry is safe. This  file is watched by log-forwarder formerly known as lumberjack. And it sends to log-parsed.  Log-parsed it, sends it into elastic search. Logging is solved. So, yeah, we are done  here. Let's get some pinchos. We are not. We have three nice components. We forgot about  the pragmatic part. How do you put those three things together without making it gross? Because  this is gross. You can barely see the logic hidden in the jungle of reporting, measuring,  counting and whatnot. So I want it to look like this, which is much nicer. Something  happens, I tell the log system about it and I'm done. Of course, that's not always possible.  But I would really try to hard. I would really like to try hard to get somewhere there. So  with errors, it's pretty easy, I dare say. So either use some handler that comes with  a sentry, either just logging or if you're using running Django, using the Django app  there, using, or just track log. That's what I do when I use a pyramid. I just plug my  errors out of the logging stream and I can drop entries if something is not interesting.  And in that app, there's usually also a way to define error views. And this is really,  really cool because, again, pyramid, you get the exception and the request object. Now,  you serve back the error ID, which is served from sentry. So now, when your customer calls  you complaining about errors, they can exactly tell you the error ID and you can look the  error ID up and you have the exception that the customer saw. And this is so great that  we've seen something that's even rarer than a white rhino, which is a happy armored runnaker.  So although I have to say, since I made a slide, he joined sentry, so take it with a  grain of salt, but still. So onto metrics. Most metrics can be observed from the outside  and outside can mean outside of your views, outside of your app, outside of your server  even. So maybe let's have a look at whiskey containers. The two major ones have both knobs  that will help you with that. So G unicorn offers stat C integration right there. So  you add one command line option and you have average request times in your stat C and in  your graphite. You don't have to change your code at all. Micro whiskey, as usual, goes  far, far, far further. They, of course, have stat C too. They have direct carbon, aka graphite  support. They have a whole metric subsystem, including nightmare inducing things like SNMP.  So you get your stuff done with that. And with this, you get a big picture of the state  of your application without even touching your apps. So go for it. Then you can write  middleware. Middleware is no dark magic. Again, pyramid. This is a tween, which is a very  awkward contraction of between. And this is called on every request that comes in. So  you have the request object. In this case, we just measure the time. But you can, of  course, look at the data within the request object and start splitting up your data depending  on the view or some argument that you're passing into your view. Probably don't have to because  there are things like pyramid stat C that already do that for you, but you always have  the possibility to do things from within your app but outside of your actual logic. Then,  of course, you can extract data from logs. Because if you lock something out, you shouldn't  have to also count it or measure it. So LogStash will do that for you. It supports all major  metrics backends. The drawback is that you have to change the configuration of LogStash,  which may or may not be a problem for you. It's not a really problem to me, but it adds  friction, which I do not like, so I do not do that. I don't want to annoy those people  who are responsible for that to fix it for me because I've added a new metric. Of course,  you can do the extract log. It's what I do. You can just count events by their names and  you already have something useful. Okay. Finally, you can also leverage monitoring,  which is even for the outside. Any monitoring system has some support for metrics numbers.  In the worst case, you just measure the time it takes to execute a check and save it. So  you get a really external view of your data, external view of the behavior of your apps,  which is not very precise, of course, but sometimes it's useful to see how your system  feels from outside boundary and not from within your availability zone or your computing center.  Okay. So what's left? What do you have to yourself? So if you want to measure code path,  you probably have to add some code to your business logic. For example, database queries.  Or if you have certain major use cases, like a view that sometimes uses only cache data  and sometimes hits a database, it's not very useful to average those two numbers. Not to  say it's completely useless. So you may want to split that up. And of course, gauges, if  you want to expose numbers from within your application, you will probably have to touch  your application in some way. And now we are really done. So what did you learn? Proper  error logging is important. Sentry is awesome. Metrics are important. Influx TV is probably  the future. Graphite is the present. Use whatever you want from those two. Centralized logging  saves you a lot of pain. And maybe you need it. Elk will have your back. Strucklock will  help you to get your data there. And now you know how to use all of them with Python without  the gross code duplication. So I hope everyone learned something. So go forth and measure.  Study the talk page. Follow me on Twitter and tell your German-speaking friends to get  their domains from Vario Media. Thank you. And I'm sorry. I'm not taking any questions  because whenever I did, I completely misunderstood the question and said something very stupid.  So if you have any questions, I will be outside. I'm here through Sunday. I will be at the  conference. I will be at lunch. Just check me out. I'm happy to answer any questions.  Thank you.
Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_0_1115.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_1116_1247.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_1248_1319.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_1320_2639.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_2640_2915.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_2916_3263.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_3264_3539.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_3540_4475.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_4476_5075.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_5076_5147.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_5148_10823.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_10824_12767.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_12768_13031.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_13032_13235.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_13236_13643.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_13644_16283.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_16284_16475.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_16476_16571.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_16572_16775.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_16776_17327.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_17328_17747.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_17748_18179.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_18180_18899.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_18900_19979.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_19980_24947.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_24948_26879.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_26880_28031.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_28032_29195.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_29196_29267.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_29268_29735.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_29736_29999.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_30000_30431.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_30432_32075.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_32076_32351.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_32352_36119.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_36120_38423.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_38424_40115.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_40116_42431.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_42432_43391.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_43392_43475.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_43476_53087.png Hynek_Schlawack_-_Beyond_grep__Practical_Logging_and_Metrics_53088_53761.png