What does "Monitoring" mean?
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Brazil, Brian
Monitoring can mean very different things to different people, and this oftenleads to confusion and misunderstandings. There are many offerings both freesoftware and commercials, and it's not always clear where each fits in thebigger picture. This talk will look a bit at the history of monitoring, andthen into the general categories of Metrics, Logs, Profiling and Distributedtracing and how each of these is important in Cloud-based environment.
 So this talk, I just want to look at the word monitoring, because lots of people have different meanings for that word. So just so you know who I am, I am one of the core developers of Prometheus. This is not a Prometheus talk. Don't worry, there's plenty more on this track. I studied computer science and strategy, was in Google for many years, and have been contributing to open source as well for many years, and not just in the sense of Prometheus, of which you know I am trying to consult against support route to make money. So monitoring, monitoring, monitoring. When you see this word, is everyone here thinking the same ideas, the same notions? Because I've talked to a lot of people, and I find we are often talking past each other. Because obviously when I'm thinking monitoring, I'm thinking kind of Prometheus-y solution. Other people are thinking L-stack. Other people are thinking you're the guy who checks whether or not I'm accessing Facebook during work time. I've had that one a few times. So like lots of people have different meanings for this, and this leads to a situation where you know, we're talking past each other. We're using the same word, but talking about completely different systems, with completely different trade-offs, completely different constraints, so we can't actually have a proper meeting of minds, and everyone just leaves angry thinking yes, I'm right, dare wrong, everyone's used my solution. This is silly, okay? We should be all thinking about the same idea, and understand where the other person is coming from. And that's what I kind of want to address in this talk. So we kind of have at least a shared vocabulary, and realizing the other types of monitoring systems that there are. The other thing you notice today is a lot of the monitoring we're doing is actually based on tools and techniques from the 80s and the largely 90s. And we haven't, in the last few years certainly, there's been quite an upsurge in monitoring stuff, both commercial and open source. You know, last five years like New Relic, Gap Dynamics, you've got InfluxDB from 80s, a whole pile of stuff came out. But we're still kind of stuck in the past in a lot of ways. And it comes from an era where, you know, you had the server, or the server rack, singular, managed by the sysadmins, plural, and each of them got personal attention, very artisan, you know, loving care and attention. And special cases were the norm, everything's a snowflake, everything is a pesh, not cattle. So this impacts how we approach monitoring, because if you come from a world where every machine is special, then all the monitoring is also special. And we've moved beyond this in other places, so we should really do that for monitoring, too. Because right now, we are still, to quote CMU, we are still often feeding the machine with human blood. We are feeding it with fatigue, we are feeding it with engineers, burnout, all these things, and we don't need to. The system will work reasonably well without that. So to give an example, at a previous company I was at, there were, the on-call was getting, I think, 2,300 alerts a week of varying levels of utility. And I went in, and I basically deleted all the alerts, and added like five new ones to cover what we cared about. So the volume of alerting went from like 2,300 a week down to like, I think, five a week, with no loss of availability, customers never noticed. So thinking about things that way, like we were doing all this work causing not unnoticeable burnout from all this alerting from this old school way of doing monitoring for no real reason. And I was just kind of thinking about those sort of things. So let's start, though, by looking at the past and looking at some of the main tools you come across. So the oldest monitoring thing that's still in common use is basically ORD, which comes from the heritage of MRTG. It was started in 1994 by Tobias Ohteker, if that's correct, perhaps correctly. He created a Perl script, because, you know, that was the fashion at the time. And that became MRTG in 1995. And it basically worked with SNMP, and it worked with external programs. And it just constantly wrote a NASCY data file with all the data every time it's scraped. And I remember setting this stuff up 15 years ago on my home system, and it was amazing. You know, never seen a graph before of my load and my CPU usage. It was really amazing with one minute resolution. In 1997, basically, it moved some code to C to make it more efficient. And also, it was interesting to me where ORD, the round-robin database, was started to improve performance, and that came out in 1999. And ORD was the basis of many tools like Graphite, although that's used in whisper these days. But if you look at the heritage, that's where this is all coming from. It's all started from a tool, a Perl script, written in 1994. So when you're thinking Graphite or anything that's Graphite-like, it comes from 1994, which is what, 23 years ago? Nice modern software, lads. And this is an MRTG graph. You have all seen them. They're still very common in network world. But yeah, seeing that for the first time, isn't it awesome? Yeah? Nagios is the other kind of tool we have. So it started by Etten Galstad in 1996. It started actually as an NSDOS program to do Pings, right? Which is interesting. And, you know, the actual project itself started in 1998. It became NetSaint and was renamed for legal reasons in 2002 to Nagios, a name which we all know and love. And basically, it runs a script. It looks at the exit code. If it's non-zero, you get them woken up. And there's lots of things that are still following the Nagios model. So a Shinga, Senzu, Zedmon, which is basically a Python-based one, just trying to scale this all up in various ways, but still following the same basic idea of, I run a script, I get an alert if it fails. That's what it does. So here's the Nagios dashboard. So that's some of the history. And we see the stuff we're basing off is like, 94, 96, we're talking 20-year-old ideas, systems that came from a different era in terms of how we manage machines. You know, we're easily managing what? Tree orders, magnitude machines is not uncommon, going from, you know, 10 to 10,000, not unusual. But it left us with another legacy, which does, because we had kind of this already in graphite versus Nagios alerting split, we ended up with a world where graphing and alerting are separate. These are totally different concerns, which has some interesting effects as well. And it's also kind of a world where, you know, because everything's a pet, even the slightest deviance, the slightest increase in load must be jumped on by a knock, because obviously the most efficient aid to use engineers is to have them staring at a screen and noticing if a line moves. Good news, 2017 computers to do that. You know, but if we're talking about going to a cloud environment, or even an environment from 10 years ago where you've got hundreds, thousands of machines and applications that is not unusual, this does not scale. Unless you're going to throw a whole pile of engineers or juniors or operators or whatever to get burned out by managing, hey, the load is 4.1, you should do something about that. And so that's important. So I think we need to kind of step back and look at things from first principles, right, and answer the question of what is monetary. You know, we can't just have it as alerting, graphing, and then jumping on everything that looks slightly off. There's just too many things that are slightly off, because real world. Okay, and we probably want to look at other things as well, like logs. Logs are kind of handy for debugging and finding problems. Like how else are you going to find out your hard disk failed? Browser events, you know, the web happened, like since, well, what was it, web was 94. So we'd have to check the exact date, so potentially MRTG is older than web. You know, we need to look at these things. And look at what it is, the actual problem we're trying to solve is rather than what these tools from the 90s give us. So I see monitoring as basically one, as three things and one extra. One of them is knowing what goes wrong, being able to debug that thing that is wrong, looking at trending over time, and, well, when all you've got is a monitoring system, everything looks like a monitoring problem. So if you are talk to people and say, it's awesome monitoring, a lot of them will say just alerting. Like, I have met people who think that monitoring is only nagging us. All you ever need is nothing else involved. And it's like, yes, certainly alerting is important. But the question we need to ask is, OK, we want to detect what's wrong, because wrongness is bad, and we want to fix that. But what sort of wrongness? And you know, what level of wrongness? Like, if we have a blip that lasts half a second and causes a problem, that's not what we can summon up for. But if it's a latency issue affecting users or things are down, that's something you want to wake someone up about. So we need to think about the severity and where we best invest our time. Where's the best return for sending an alert? Rather than just saying, oh, yeah, load's higher. Because fundamentally, we're humans, and we are limited in what we can handle. And one analysis would say that realistically, an on-call engineer can handle two events per shift. Because, well, it's going to take you four hours to debug-ish, write the postmortem, you know, and go through all the steps if it's a real problem. If you alert on every single thing, you're going to get pager fatigue. If you ever get in a situation or you receive an alert, you go, meh, put it back in your pocket, you have a problem. Because it means if there's a real problem, you might do the exact same thing, not noticing, no, no, it's actually a true positive for once. So you need to care about your signal to noise ratio. And you know, if that's happening in the middle of the night, that's ruining your sleep, you're not an effective engineer. And so the thing is, though, you care about user-facing latency, for example, because that's important. That directly drives revenue in many systems. There are hundreds of things that could cause an increase in user latency. You know, you could have a deadlock, you could have the network being weird, you could have someone running a batch job in the same machine, someone could have, you know, hit the network cable with their backside, and now it's doing 10, maybe it's rather than 100. So many hundreds of things that could cause latency to go up. Trying to alert or even tink of every single one of those, it's never going to happen. This one? Cool. Apparently, I need that for a video. Okay? Okay. That's what you're getting? Yeah, okay, however, the thing we care about is not, you know, what the network interface is negotiated to. The thing we care about is user-facing latency. So hey, let's alert on user-facing latency. And what you'll see is if you have a system like, say, the Nagio-style system, so all you can do is run a script, you can alert on CPU usage, that's not too hard, but you can't alert on user latency because that requires too much math and analysis that you can't do. And even if you were to do that, you're still only doing it on one machine. Whereas these days, a service might be on 10, 100 machines. So you end up getting false positives because, hey, log rotate ran too long, you had too much CPU usage. And false negatives because, well, a deadlock is going to use no CPU and never trigger your CPU alert. The closest you could get to having a user latency alert was a high CPU alert, but it's going to miss stuff. And the result is spammy alerts waking up people in the middle of the night and missing real problems. So that kind of sucks. So the thing is, what I would say to you, what we're taking about alerting and what a monitoring system whatever it is is doing is alerts should require intelligent human action, right? If we're engineers and we can pay for it or we pay for our brains, not just to do what a shell script can do, right? We write the shell scripts, we are not the shell scripts. And alerts should actually relate to real end user problems, OK? Not some, you know, your users do not care if your machine has load four. Your users care if they cannot get to their cat videos. And that is the reality of the thing. Sure you might get the warm fuzzies from saying, yes, the load is now lower, that minor weirdness is solved. But there's so many minor weirdnesses, you know, you'll just end up fatigued and burned out if you try and do that. Once you have your alert, you need to investigate it because let's be honest, if you have an alert like disk full, it's pretty obvious what you need to go look at or high CPU. If you get an alert like the latency is high, yep, you now need to figure out what's going on from there. So you need some way with dashboards and what not to be able to drill down through your system and figure out what the problem is, OK? So if you think about your stack of services and how they're talking to us, it's a tree which is hopefully acyclic. So you can start at the top and say, right, there's high latency. Is this service overloaded? No. It's back ends. Have any of them increased in latency? Ah, that one has. Look at it. OK, this service, is it overloaded? No. Does any of its back ends have increased latency? Ah, this one. This service, is it overloaded? Yes. OK, why is it overloaded? And stepping through things logically rather than looking at, you know, a screen of 100 graphs and hoping you're going to get divine inspiration to remember, A, what the graphs mean, and B, find the right one. So we need to think about this logically and methodically because we need to scale out how we approach these problems. And you don't just use graphs. You might have to use logs. You might have to use source code. You might have to use some end user debugging, maybe an ETCP dump. Bring in more tools as needed. There is no one tool that will help you debug problems because you need a pile of tools. You're going to need some metrics. You're going to need some logs, profiling tools like GDB speed dump and the source code because you're coming across any non-trivial problem inside an application or whatnot. You're going to need all of these. So as an example, recently I was optimizing communities itself and I was using a mix of source code, profiling, and metrics. Logs were abuses in this particular case. But I couldn't just use one of those tools. I had to use them in concert. And without those, I would never have made some rather nice performance improvements. If you're thinking about problems, you need to not just lock yourself into one way of looking at data. You need as much as possible. The third use case for monitoring is training and reporting because alerting and debugging tends to be on the order of minutes to maybe days. Training and reporting is weeks to years, much longer term, much more strategic. Because you might want to know, hey, how is my cache hit rate changing? Is anyone using some obscure feature and will I need more machines? So more long term, being able to make engineering decisions, being able to make business decisions. And the final one is plumbing. Because sometimes people just want to use your monitoring system as a data bus. This is not monitoring, but it's done often so you should be aware of it because it's going to happen. So basically, what are the APIs in? What are the APIs out? And often it's for control systems of some form, whether it be business or technical. And to be honest, if someone needs to pass like one piece of data true once a minute, that's basically free. Why not? Assuming all the reliability stuff is OK. If they want to push through lots of data, there's this thing called Kafka or whatever your preferred message bus is. So now we're talking about the tree general goes on monitoring. How do we actually do it? What data should we collect? How should we process it? What sort of things do we need to decide on? Because let's be honest, money doesn't grow on trees. Resources aren't free. Time is not free. We can't just say everything all the time. That doesn't work. At the core of all monitoring are events. An event is something that happens. Like I come in here, I sit down, I turn on the machine. More commonly, we're talking about computers, it's going to be a HTTP request. It's a package, a library call, or a broadcast probe. And there's context. If I do that, it's that I did that inside this room at this time in the context of FOSDM. And all that context adds up. So a single piece of data, a single event, and all the call trace, Skolstack might have hundreds of pieces of context there where you can see everything. Like what browser was I using? And there can be millions of events per second if you're talking at that level, because we're thinking about all the function calls being made, all the context switches, all the ins and outs, which you could potentially want data for. And I see that there are basically four classes of monitoring very roughly. Profiling, metrics, logs, and distributed tracing. If you're talking to someone without monitoring, they normally mean one of the first three. Four wasn't the newer. And they're only thinking of that, not of the other ones. But you need all of them in a modern environment, because they're complementary. They're all of different trade-offs. So profiling is the one we all know and love. PCP, DOM, GDB, S trace, D trace. Basically Brendan and Greg, if he has ever talked about it, it is profiling. That's the easy way to think about it. Also read his stuff. It's awesome. And the thing is that it's awesome because it gives you really detailed information about individual events. I can see how many times this kernel function was called. And oh, yeah, that caused whatever weirdness that Brendan and Greg has found for the first time. But the thing is, because this data is so detailed, you can't turn it on all the time. You wouldn't have this space for it. It would hit your performance like no one's business. So any use of it needs to be highly targeted, and it needs to be temporary. Like if you're turning this stuff on for more than a few minutes, depending on what you're doing, you just can't because you're going to run out of this space or whatnot. So profiling is great. If you have an idea of what's wrong, it's like, ah, I think it's kernel latency, then grand, let's look at the kernel stuff. Or oh, I think it's a monitoring problem, let me get a TCP dump of those sides. But you're not going to get the kernel dump when you don't know what's going on in the first place because what are you looking for? So that's kind of handy. The second one is metrics. Because profiling is looking at all the events with all the context, but we can only do it for a very short period of time. Metrics, what we do is that we ignore the individual events, but we look at the context, and we aggregate based on that over time, like temporarily, maybe a minute, 10 seconds. So with metrics, if you have a HTTP request coming in, we wouldn't track the exact URL it looks at. We will tell you that there were 3,000 HTTP requests at the last minute. We will tell you that 14 of them failed, that 17 of them hit cache, that one of them hit some weird obscured code path, and we have data at that level so we can get a general overview of what's going on. We can't tell what's happening with individual events, although we can cross correlate to kind of figure it out, but we can get a nice big broad view of the system. And that's what metrics give you. Because you can track, practically speaking, tens of thousands of metrics about a single binary. Don't try too many more than that. But you can't break them out by 2D interventions, because every single one of those, all that cardinality, that's one more metric, one more time series. So if you broke out metrics, for example, by customer email address, and you have more than 10 customers, that's not going to end well, because that's what eats all your resources. So it's great for just figuring out generally what's going on, but it's not so great for tracking individual events, because you've made the trade-off, I'm having breadth not depth. On the other side, we have logs. Logs track individual events. So those hasty requests earlier, that was Mr. Fu visiting my endpoint. Yesterday at 7, he had this size response, this status code, these endpoints. But the thing is, though, because it's per event, you're limited bandwidth-wise, whether that be disk or network, to 100-ish call-ish fields in your logs. And the data volume is involved, typically means there's a propagation delay in processing all this stuff. So the Elk stack and gray log, that sort of solution is what you're looking for there, syslog, and something like that. And in logs, little fractally, there's also another thing that people don't realize, we're talking about different types of logs when they say logs, because there's once again a pile of trade-offs. So at one end, you've got your transactional logs, or the audit logs, where you basically have to have them. You cannot lose a single piece of data because you're basing your billing on it or something else key. Like, you can imagine if you were in a financial trading, everything's transactional, you can't use it, you have to keep it around for seven years for compliance. On the other hand, you have like the bug logs, you know, which is basically profiling, which you might keep around for a few hours if you have the disk space. And the trade-offs involved in these are completely different. One of them, you know, you basically need a CPC system, get everything absolutely perfect. The other one is like, yeah, there's data there, might be there in five minutes, might not, if we lose it, who cares? And those are different trade-offs. And immediately, we've got request logs and application logs. Like application logs need to be viewable by a human, by eye, without any tooling, because that happens sometimes. Entirely different trade-offs. And the final one is distributed tracing, and this is really a special case of logging. But in the more cloudy environments, like microservices, not distributed systems, you need this one, because trying to figure out some certain problems with just logs and metrics doesn't work. What it does is, each request as it comes into a system gets an ID, and as it causes, you know, a chain reaction of requests, those also, that ID is propagated and logged. And then they basically pulls back together all the requests, the result of the original request and said, right, here's the full tree and the timing tree. And what caused it? So when weird stuff's happening, you kind of need this. So open tracing and open zipkin for that. And you get this sort of result from it. But I can see, oh, yeah, that was the main request. It calls this, calls this, calls this, which calls these parallel requests, and they responded in this time, therefore such and such was slow. So it's an interesting use of logging. So if we go back then to the original question of what monitoring mean, monitoring isn't just black box alerts that are going to a knock. It's not just per host graphs. It's not just aggregated white box application. Oops, there is it. Yeah, graphs. It's not just logs. It's not just metrics. And it's not just the big TV screen on the wall. How many people have been in the place where it says, we need monitoring? So let's get tree ground, get a massive screen, push it up on the wall, and it's going to have the graphs monitoring solved. I'm seeing a lot of smiling faces who have been in companies where that's happened. Yeah. And there's also a question. This is a big question to monitor. What does word monitoring mean? And very generically, monitoring is the tools and techniques you have to keep an eye on things. And that includes the humans and the culture and the policy around them. As important as well, remember, as Jason Dix was saying, there is no silver bullet product. There's no product that's going to do everything for you. You can't grab all the data. You have to make a trade off, some trade off at the end of the day. So to summarize, the goal of monitoring, the thing we're always trying to do at a high level is knowing when things go wrong, it's being able to debug those things, and then more strategically, rather than tactically, being able to make business decisions based on trends. And moving data from A to B sometimes because someone's going to want to do it. And sort of monitoring systems you'll see is profiling, which is looking at lots of data in lots of detail, but you can only turn it on briefly in a very targeted way. Metrics which are great for breadth at the cost of depth. Logs which are great for depth at the cost of breadth. And the distributed tracing which you need to debug distributed systems. And I will say as well, just as we're looking at these things, what made sense 20 years ago when you had a handful of machines does not make sense when you have a handful of data centers, each with a handful of services, but a handful of replicas of, you know, thousands of machines. In the same way that say 10 years ago, we were starting to use tools like Chef and Puppet to stop doing deployments by hand and starting to automating all that stuff, we need to take the same mindset and apply it to monitoring. We need to stop caring about individual machines being slightly slow as long as the service and aggregate is doing okay. And making that sort of culture shift to it. So think about the same way, how do we scale up as we move into a situation where we're going from five machines to 10,000 containers which are recreated every hour because that is a reality that we have to look at. Incidentally, I hear Prometheus is good for metrics. So if there's any questions, I think we've got a few minutes. Four of us to stop. Three minutes. Any questions? Is one there? Fabian, do you want to get hooked in? Hello. I do not really understand the difference between profiling and logs. Okay. So the difference between profiling and logs is that profiling is so information dense you cannot turn it on all the time. So if you can imagine TCP dump, you know, announcing that's not extremely low traffic, if you turn that on, it's going to take you a hit on the network and a hit on disk space that you just can't sustain. Like if you ask Richie, a network person, I want to mirror all the traffic going through this switch onto this machine. It's like, wait, wait, wait. This machine has an aggregate bandwidth in the backbone of a terabit. Please explain to me which interface exactly you would like me to send this data over. Think of it in that way. Or if I wanted to track every single kernel call that's being made, you know, that's going to add maybe at least double, maybe triple the time it takes to do a kernel call. So you can do that for debugging and hopefully it's not going to affect debugging too much, but you can turn it on all the time because it's too much data and it's going to slow you down too much, too much for performance. Does that answer your question? One more question maybe we can squeeze in. No? Yeah, we have one here. So you said that a lot of things should be done based on user-visible aspects of that form. I don't really agree with that. I mean, this is useful, of course, but there are lots of problems that can arise from easily identifiable problems that are not user-visible. For example, this space going up at some point you want to get an alert just to say we are over 90 percent. Okay. So the question is about basically saturation alerts about running out of disk space. What are the situations for your running out of quota? If you know that running out of resource is going to imminently cause, and imminently in this context means about four hours because that's about how long it takes to deal with an issue like that, is going to imminently cause an outage, running out of disk space, running out of any form of quota, whether that be file descriptors or how much QPS you bought off your CEN provider, yes, that should be an alert because you're just preempting a user-visible alert that's going to happen in, let's say, four to 24 hours. But if it's just, yeah, that machine filled on disk, but that's it. We have another hundred of them which haven't filled up on disk. You can deal with that with business hours. So I was just thinking about, okay, is this a user impact right now or in the very near future such that it's worth me getting a human to look at this immediately from bed, or is it just a ticket that we can deal with in business hours? So that sort of change because it's silly to not prevent outages. But if we're just saying, yep, disk space just went to 70%, it will take two years to fill, but it hit 70% if we need to page someone, that's silly. That's where it is. So thank you very much. Thank you.
 So this talk, I just want to look at the word monitoring, because lots of people have different  meanings for that word.  So just so you know who I am, I am one of the core developers of Prometheus.  This is not a Prometheus talk.  Don't worry, there's plenty more on this track.  I studied computer science and strategy, was in Google for many years, and have been contributing  to open source as well for many years, and not just in the sense of Prometheus, of which  you know I am trying to consult against support route to make money.  So monitoring, monitoring, monitoring.  When you see this word, is everyone here thinking the same ideas, the same notions?  Because I've talked to a lot of people, and I find we are often talking past each other.  Because obviously when I'm thinking monitoring, I'm thinking kind of Prometheus-y solution.  Other people are thinking L-stack.  Other people are thinking you're the guy who checks whether or not I'm accessing Facebook  during work time.  I've had that one a few times.  So like lots of people have different meanings for this, and this leads to a situation where  you know, we're talking past each other.  We're using the same word, but talking about completely different systems, with completely  different trade-offs, completely different constraints, so we can't actually have a proper  meeting of minds, and everyone just leaves angry thinking yes, I'm right, dare wrong,  everyone's used my solution.  This is silly, okay?  We should be all thinking about the same idea, and understand where the other person is coming  from.  And that's what I kind of want to address in this talk.  So we kind of have at least a shared vocabulary, and realizing the other types of monitoring  systems that there are.  The other thing you notice today is a lot of the monitoring we're doing is actually based  on tools and techniques from the 80s and the largely 90s.  And we haven't, in the last few years certainly, there's been quite an upsurge in monitoring  stuff, both commercial and open source.  You know, last five years like New Relic, Gap Dynamics, you've got InfluxDB from 80s,  a whole pile of stuff came out.  But we're still kind of stuck in the past in a lot of ways.  And it comes from an era where, you know, you had the server, or the server rack, singular,  managed by the sysadmins, plural, and each of them got personal attention, very artisan,  you know, loving care and attention.  And special cases were the norm, everything's a snowflake, everything is a pesh, not cattle.  So this impacts how we approach monitoring, because if you come from a world where every  machine is special, then all the monitoring is also special.  And we've moved beyond this in other places, so we should really do that for monitoring,  too.  Because right now, we are still, to quote CMU, we are still often feeding the machine with  human blood.  We are feeding it with fatigue, we are feeding it with engineers, burnout, all these things,  and we don't need to.  The system will work reasonably well without that.  So to give an example, at a previous company I was at, there were, the on-call was getting,  I think, 2,300 alerts a week of varying levels of utility.  And I went in, and I basically deleted all the alerts, and added like five new ones to  cover what we cared about.  So the volume of alerting went from like 2,300 a week down to like, I think, five a week,  with no loss of availability, customers never noticed.  So thinking about things that way, like we were doing all this work causing not unnoticeable  burnout from all this alerting from this old school way of doing monitoring for no real  reason.  And I was just kind of thinking about those sort of things.  So let's start, though, by looking at the past and looking at some of the main tools  you come across.  So the oldest monitoring thing that's still in common use is basically ORD, which comes  from the heritage of MRTG.  It was started in 1994 by Tobias Ohteker, if that's correct, perhaps correctly.  He created a Perl script, because, you know, that was the fashion at the time.  And that became MRTG in 1995.  And it basically worked with SNMP, and it worked with external programs.  And it just constantly wrote a NASCY data file with all the data every time it's scraped.  And I remember setting this stuff up 15 years ago on my home system, and it was amazing.  You know, never seen a graph before of my load and my CPU usage.  It was really amazing with one minute resolution.  In 1997, basically, it moved some code to C to make it more efficient.  And also, it was interesting to me where ORD, the round-robin database, was started to improve  performance, and that came out in 1999.  And ORD was the basis of many tools like Graphite, although that's used in whisper these days.  But if you look at the heritage, that's where this is all coming from.  It's all started from a tool, a Perl script, written in 1994.  So when you're thinking Graphite or anything that's Graphite-like, it comes from 1994,  which is what, 23 years ago?  Nice modern software, lads.  And this is an MRTG graph.  You have all seen them.  They're still very common in network world.  But yeah, seeing that for the first time, isn't it awesome?  Yeah?  Nagios is the other kind of tool we have.  So it started by Etten Galstad in 1996.  It started actually as an NSDOS program to do Pings, right?  Which is interesting.  And, you know, the actual project itself started in 1998.  It became NetSaint and was renamed for legal reasons in 2002 to Nagios, a name which we  all know and love.  And basically, it runs a script.  It looks at the exit code.  If it's non-zero, you get them woken up.  And there's lots of things that are still following the Nagios model.  So a Shinga, Senzu, Zedmon, which is basically a Python-based one, just trying to scale this  all up in various ways, but still following the same basic idea of, I run a script, I  get an alert if it fails.  That's what it does.  So here's the Nagios dashboard.  So that's some of the history.  And we see the stuff we're basing off is like, 94, 96, we're talking 20-year-old ideas,  systems that came from a different era in terms of how we manage machines.  You know, we're easily managing what?  Tree orders, magnitude machines is not uncommon, going from, you know, 10 to 10,000, not unusual.  But it left us with another legacy, which does, because we had kind of this already  in graphite versus Nagios alerting split, we ended up with a world where graphing and  alerting are separate.  These are totally different concerns, which has some interesting effects as well.  And it's also kind of a world where, you know, because everything's a pet, even the slightest  deviance, the slightest increase in load must be jumped on by a knock, because obviously  the most efficient aid to use engineers is to have them staring at a screen and noticing  if a line moves.  Good news, 2017 computers to do that.  You know, but if we're talking about going to a cloud environment, or even an environment  from 10 years ago where you've got hundreds, thousands of machines and applications that  is not unusual, this does not scale.  Unless you're going to throw a whole pile of engineers or juniors or operators or whatever  to get burned out by managing, hey, the load is 4.1, you should do something about that.  And so that's important.  So I think we need to kind of step back and look at things from first principles, right,  and answer the question of what is monetary.  You know, we can't just have it as alerting, graphing, and then jumping on everything that  looks slightly off.  There's just too many things that are slightly off, because real world.  Okay, and we probably want to look at other things as well, like logs.  Logs are kind of handy for debugging and finding problems.  Like how else are you going to find out your hard disk failed?  Browser events, you know, the web happened, like since, well, what was it, web was 94.  So we'd have to check the exact date, so potentially MRTG is older than web.  You know, we need to look at these things.  And look at what it is, the actual problem we're trying to solve is rather than what  these tools from the 90s give us.  So I see monitoring as basically one, as three things and one extra.  One of them is knowing what goes wrong, being able to debug that thing that is wrong, looking  at trending over time, and, well, when all you've got is a monitoring system, everything  looks like a monitoring problem.  So if you are talk to people and say, it's awesome monitoring, a lot of them will say  just alerting.  Like, I have met people who think that monitoring is only nagging us.  All you ever need is nothing else involved.  And it's like, yes, certainly alerting is important.  But the question we need to ask is, OK, we want to detect what's wrong, because wrongness  is bad, and we want to fix that.  But what sort of wrongness?  And you know, what level of wrongness?  Like, if we have a blip that lasts half a second and causes a problem, that's not what  we can summon up for.  But if it's a latency issue affecting users or things are down, that's something you want  to wake someone up about.  So we need to think about the severity and where we best invest our time.  Where's the best return for sending an alert?  Rather than just saying, oh, yeah, load's higher.  Because fundamentally, we're humans, and we are limited in what we can handle.  And one analysis would say that realistically, an on-call engineer can handle two events  per shift.  Because, well, it's going to take you four hours to debug-ish, write the postmortem,  you know, and go through all the steps if it's a real problem.  If you alert on every single thing, you're going to get pager fatigue.  If you ever get in a situation or you receive an alert, you go, meh, put it back in your  pocket, you have a problem.  Because it means if there's a real problem, you might do the exact same thing, not noticing,  no, no, it's actually a true positive for once.  So you need to care about your signal to noise ratio.  And you know, if that's happening in the middle of the night, that's ruining your sleep, you're  not an effective engineer.  And so the thing is, though, you care about user-facing latency, for example, because  that's important.  That directly drives revenue in many systems.  There are hundreds of things that could cause an increase in user latency.  You know, you could have a deadlock, you could have the network being weird, you could have  someone running a batch job in the same machine, someone could have, you know, hit the network  cable with their backside, and now it's doing 10, maybe it's rather than 100.  So many hundreds of things that could cause latency to go up.  Trying to alert or even tink of every single one of those, it's never going to happen.  This one?  Cool.  Apparently, I need that for a video.  Okay?  Okay.  That's what you're getting?  Yeah, okay, however, the thing we care about is not, you know, what the network interface  is negotiated to.  The thing we care about is user-facing latency.  So hey, let's alert on user-facing latency.  And what you'll see is if you have a system like, say, the Nagio-style system, so all  you can do is run a script, you can alert on CPU usage, that's not too hard, but you  can't alert on user latency because that requires too much math and analysis that you  can't do.  And even if you were to do that, you're still only doing it on one machine.  Whereas these days, a service might be on 10, 100 machines.  So you end up getting false positives because, hey, log rotate ran too long, you had too  much CPU usage.  And false negatives because, well, a deadlock is going to use no CPU and never trigger your  CPU alert.  The closest you could get to having a user latency alert was a high CPU alert, but it's  going to miss stuff.  And the result is spammy alerts waking up people in the middle of the night and missing  real problems.  So that kind of sucks.  So the thing is, what I would say to you, what we're taking about alerting and what  a monitoring system whatever it is is doing is alerts should require intelligent human  action, right?  If we're engineers and we can pay for it or we pay for our brains, not just to do what  a shell script can do, right?  We write the shell scripts, we are not the shell scripts.  And alerts should actually relate to real end user problems, OK?  Not some, you know, your users do not care if your machine has load four.  Your users care if they cannot get to their cat videos.  And that is the reality of the thing.  Sure you might get the warm fuzzies from saying, yes, the load is now lower, that minor weirdness  is solved.  But there's so many minor weirdnesses, you know, you'll just end up fatigued and burned  out if you try and do that.  Once you have your alert, you need to investigate it because let's be honest, if you have an  alert like disk full, it's pretty obvious what you need to go look at or high CPU.  If you get an alert like the latency is high, yep, you now need to figure out what's going  on from there.  So you need some way with dashboards and what not to be able to drill down through your  system and figure out what the problem is, OK?  So if you think about your stack of services and how they're talking to us, it's a tree  which is hopefully acyclic.  So you can start at the top and say, right, there's high latency.  Is this service overloaded?  No.  It's back ends.  Have any of them increased in latency?  Ah, that one has.  Look at it.  OK, this service, is it overloaded?  No.  Does any of its back ends have increased latency?  Ah, this one.  This service, is it overloaded?  Yes.  OK, why is it overloaded?  And stepping through things logically rather than looking at, you know, a screen of 100  graphs and hoping you're going to get divine inspiration to remember, A, what the graphs  mean, and B, find the right one.  So we need to think about this logically and methodically because we need to scale out  how we approach these problems.  And you don't just use graphs.  You might have to use logs.  You might have to use source code.  You might have to use some end user debugging, maybe an ETCP dump.  Bring in more tools as needed.  There is no one tool that will help you debug problems because you need a pile of tools.  You're going to need some metrics.  You're going to need some logs, profiling tools like GDB speed dump and the source code  because you're coming across any non-trivial problem inside an application or whatnot.  You're going to need all of these.  So as an example, recently I was optimizing communities itself and I was using a mix of  source code, profiling, and metrics.  Logs were abuses in this particular case.  But I couldn't just use one of those tools.  I had to use them in concert.  And without those, I would never have made some rather nice performance improvements.  If you're thinking about problems, you need to not just lock yourself into one way of  looking at data.  You need as much as possible.  The third use case for monitoring is training and reporting because alerting and debugging  tends to be on the order of minutes to maybe days.  Training and reporting is weeks to years, much longer term, much more strategic.  Because you might want to know, hey, how is my cache hit rate changing?  Is anyone using some obscure feature and will I need more machines?  So more long term, being able to make engineering decisions, being able to make business decisions.  And the final one is plumbing.  Because sometimes people just want to use your monitoring system as a data bus.  This is not monitoring, but it's done often so you should be aware of it because it's  going to happen.  So basically, what are the APIs in?  What are the APIs out?  And often it's for control systems of some form, whether it be business or technical.  And to be honest, if someone needs to pass like one piece of data true once a minute,  that's basically free.  Why not?  Assuming all the reliability stuff is OK.  If they want to push through lots of data, there's this thing called Kafka or whatever  your preferred message bus is.  So now we're talking about the tree general goes on monitoring.  How do we actually do it?  What data should we collect?  How should we process it?  What sort of things do we need to decide on?  Because let's be honest, money doesn't grow on trees.  Resources aren't free.  Time is not free.  We can't just say everything all the time.  That doesn't work.  At the core of all monitoring are events.  An event is something that happens.  Like I come in here, I sit down, I turn on the machine.  More commonly, we're talking about computers, it's going to be a HTTP request.  It's a package, a library call, or a broadcast probe.  And there's context.  If I do that, it's that I did that inside this room at this time in the context of FOSDM.  And all that context adds up.  So a single piece of data, a single event, and all the call trace, Skolstack might have  hundreds of pieces of context there where you can see everything.  Like what browser was I using?  And there can be millions of events per second if you're talking at that level, because we're  thinking about all the function calls being made, all the context switches, all the ins  and outs, which you could potentially want data for.  And I see that there are basically four classes of monitoring very roughly.  Profiling, metrics, logs, and distributed tracing.  If you're talking to someone without monitoring, they normally mean one of the first three.  Four wasn't the newer.  And they're only thinking of that, not of the other ones.  But you need all of them in a modern environment, because they're complementary.  They're all of different trade-offs.  So profiling is the one we all know and love.  PCP, DOM, GDB, S trace, D trace.  Basically Brendan and Greg, if he has ever talked about it, it is profiling.  That's the easy way to think about it.  Also read his stuff.  It's awesome.  And the thing is that it's awesome because it gives you really detailed information about  individual events.  I can see how many times this kernel function was called.  And oh, yeah, that caused whatever weirdness that Brendan and Greg has found for the first  time.  But the thing is, because this data is so detailed, you can't turn it on all the time.  You wouldn't have this space for it.  It would hit your performance like no one's business.  So any use of it needs to be highly targeted, and it needs to be temporary.  Like if you're turning this stuff on for more than a few minutes, depending on what you're  doing, you just can't because you're going to run out of this space or whatnot.  So profiling is great.  If you have an idea of what's wrong, it's like, ah, I think it's kernel latency, then  grand, let's look at the kernel stuff.  Or oh, I think it's a monitoring problem, let me get a TCP dump of those sides.  But you're not going to get the kernel dump when you don't know what's going on in the  first place because what are you looking for?  So that's kind of handy.  The second one is metrics.  Because profiling is looking at all the events with all the context, but we can only do it  for a very short period of time.  Metrics, what we do is that we ignore the individual events, but we look at the context,  and we aggregate based on that over time, like temporarily, maybe a minute, 10 seconds.  So with metrics, if you have a HTTP request coming in, we wouldn't track the exact URL  it looks at.  We will tell you that there were 3,000 HTTP requests at the last minute.  We will tell you that 14 of them failed, that 17 of them hit cache, that one of them hit  some weird obscured code path, and we have data at that level so we can get a general  overview of what's going on.  We can't tell what's happening with individual events, although we can cross correlate to  kind of figure it out, but we can get a nice big broad view of the system.  And that's what metrics give you.  Because you can track, practically speaking, tens of thousands of metrics about a single  binary.  Don't try too many more than that.  But you can't break them out by 2D interventions, because every single one of those, all that  cardinality, that's one more metric, one more time series.  So if you broke out metrics, for example, by customer email address, and you have more  than 10 customers, that's not going to end well, because that's what eats all your resources.  So it's great for just figuring out generally what's going on, but it's not so great for  tracking individual events, because you've made the trade-off, I'm having breadth not  depth.  On the other side, we have logs.  Logs track individual events.  So those hasty requests earlier, that was Mr. Fu visiting my endpoint.  Yesterday at 7, he had this size response, this status code, these endpoints.  But the thing is, though, because it's per event, you're limited bandwidth-wise, whether  that be disk or network, to 100-ish call-ish fields in your logs.  And the data volume is involved, typically means there's a propagation delay in processing  all this stuff.  So the Elk stack and gray log, that sort of solution is what you're looking for there,  syslog, and something like that.  And in logs, little fractally, there's also another thing that people don't realize, we're  talking about different types of logs when they say logs, because there's once again  a pile of trade-offs.  So at one end, you've got your transactional logs, or the audit logs, where you basically  have to have them.  You cannot lose a single piece of data because you're basing your billing on it or something  else key.  Like, you can imagine if you were in a financial trading, everything's transactional, you can't  use it, you have to keep it around for seven years for compliance.  On the other hand, you have like the bug logs, you know, which is basically profiling, which  you might keep around for a few hours if you have the disk space.  And the trade-offs involved in these are completely different.  One of them, you know, you basically need a CPC system, get everything absolutely perfect.  The other one is like, yeah, there's data there, might be there in five minutes, might  not, if we lose it, who cares?  And those are different trade-offs.  And immediately, we've got request logs and application logs.  Like application logs need to be viewable by a human, by eye, without any tooling, because  that happens sometimes.  Entirely different trade-offs.  And the final one is distributed tracing, and this is really a special case of logging.  But in the more cloudy environments, like microservices, not distributed systems, you  need this one, because trying to figure out some certain problems with just logs and metrics  doesn't work.  What it does is, each request as it comes into a system gets an ID, and as it causes,  you know, a chain reaction of requests, those also, that ID is propagated and logged.  And then they basically pulls back together all the requests, the result of the original  request and said, right, here's the full tree and the timing tree.  And what caused it?  So when weird stuff's happening, you kind of need this.  So open tracing and open zipkin for that.  And you get this sort of result from it.  But I can see, oh, yeah, that was the main request.  It calls this, calls this, calls this, which calls these parallel requests, and they responded  in this time, therefore such and such was slow.  So it's an interesting use of logging.  So if we go back then to the original question of what monitoring mean, monitoring isn't just  black box alerts that are going to a knock.  It's not just per host graphs.  It's not just aggregated white box application.  Oops, there is it.  Yeah, graphs.  It's not just logs.  It's not just metrics.  And it's not just the big TV screen on the wall.  How many people have been in the place where it says, we need monitoring?  So let's get tree ground, get a massive screen, push it up on the wall, and it's going to  have the graphs monitoring solved.  I'm seeing a lot of smiling faces who have been in companies where that's happened.  Yeah.  And there's also a question.  This is a big question to monitor.  What does word monitoring mean?  And very generically, monitoring is the tools and techniques you have to keep an eye on  things.  And that includes the humans and the culture and the policy around them.  As important as well, remember, as Jason Dix was saying, there is no silver bullet product.  There's no product that's going to do everything for you.  You can't grab all the data.  You have to make a trade off, some trade off at the end of the day.  So to summarize, the goal of monitoring, the thing we're always trying to do at a high  level is knowing when things go wrong, it's being able to debug those things, and then  more strategically, rather than tactically, being able to make business decisions based  on trends.  And moving data from A to B sometimes because someone's going to want to do it.  And sort of monitoring systems you'll see is profiling, which is looking at lots of data  in lots of detail, but you can only turn it on briefly in a very targeted way.  Metrics which are great for breadth at the cost of depth.  Logs which are great for depth at the cost of breadth.  And the distributed tracing which you need to debug distributed systems.  And I will say as well, just as we're looking at these things, what made sense 20 years  ago when you had a handful of machines does not make sense when you have a handful of  data centers, each with a handful of services, but a handful of replicas of, you know, thousands  of machines.  In the same way that say 10 years ago, we were starting to use tools like Chef and Puppet  to stop doing deployments by hand and starting to automating all that stuff, we need to take  the same mindset and apply it to monitoring.  We need to stop caring about individual machines being slightly slow as long as the service  and aggregate is doing okay.  And making that sort of culture shift to it.  So think about the same way, how do we scale up as we move into a situation where we're  going from five machines to 10,000 containers which are recreated every hour because that  is a reality that we have to look at.  Incidentally, I hear Prometheus is good for metrics.  So if there's any questions, I think we've got a few minutes.  Four of us to stop.  Three minutes.  Any questions?  Is one there?  Fabian, do you want to get hooked in?  Hello.  I do not really understand the difference between profiling and logs.  Okay.  So the difference between profiling and logs is that profiling is so information dense  you cannot turn it on all the time.  So if you can imagine TCP dump, you know, announcing that's not extremely low traffic,  if you turn that on, it's going to take you a hit on the network and a hit on disk space  that you just can't sustain.  Like if you ask Richie, a network person, I want to mirror all the traffic going through  this switch onto this machine.  It's like, wait, wait, wait.  This machine has an aggregate bandwidth in the backbone of a terabit.  Please explain to me which interface exactly you would like me to send this data over.  Think of it in that way.  Or if I wanted to track every single kernel call that's being made, you know, that's  going to add maybe at least double, maybe triple the time it takes to do a kernel call.  So you can do that for debugging and hopefully it's not going to affect debugging too much,  but you can turn it on all the time because it's too much data and it's going to slow  you down too much, too much for performance.  Does that answer your question?  One more question maybe we can squeeze in.  No?  Yeah, we have one here.  So you said that a lot of things should be done based on user-visible aspects of that  form.  I don't really agree with that.  I mean, this is useful, of course, but there are lots of problems that can arise from easily  identifiable problems that are not user-visible.  For example, this space going up at some point you want to get an alert just to say we are  over 90 percent.  Okay.  So the question is about basically saturation alerts about running out of disk space.  What are the situations for your running out of quota?  If you know that running out of resource is going to imminently cause, and imminently  in this context means about four hours because that's about how long it takes to deal with  an issue like that, is going to imminently cause an outage, running out of disk space,  running out of any form of quota, whether that be file descriptors or how much QPS you  bought off your CEN provider, yes, that should be an alert because you're just preempting  a user-visible alert that's going to happen in, let's say, four to 24 hours.  But if it's just, yeah, that machine filled on disk, but that's it.  We have another hundred of them which haven't filled up on disk.  You can deal with that with business hours.  So I was just thinking about, okay, is this a user impact right now or in the very near  future such that it's worth me getting a human to look at this immediately from bed, or is  it just a ticket that we can deal with in business hours?  So that sort of change because it's silly to not prevent outages.  But if we're just saying, yep, disk space just went to 70%, it will take two years to  fill, but it hit 70% if we need to page someone, that's silly.  That's where it is.  So thank you very much.  Thank you.
What_does_Monitoring_mean_0_47.png What_does_Monitoring_mean_48_131.png What_does_Monitoring_mean_132_179.png What_does_Monitoring_mean_180_311.png What_does_Monitoring_mean_312_419.png What_does_Monitoring_mean_420_899.png What_does_Monitoring_mean_900_1655.png What_does_Monitoring_mean_1656_1787.png What_does_Monitoring_mean_1788_5987.png What_does_Monitoring_mean_5988_6119.png What_does_Monitoring_mean_6120_8291.png What_does_Monitoring_mean_8292_10187.png What_does_Monitoring_mean_10188_15851.png What_does_Monitoring_mean_15852_18695.png What_does_Monitoring_mean_18696_19859.png What_does_Monitoring_mean_19860_21875.png What_does_Monitoring_mean_21876_22835.png What_does_Monitoring_mean_22836_23543.png What_does_Monitoring_mean_23544_24551.png What_does_Monitoring_mean_24552_25163.png What_does_Monitoring_mean_25164_26399.png What_does_Monitoring_mean_26400_27119.png What_does_Monitoring_mean_27120_28943.png What_does_Monitoring_mean_28944_28991.png What_does_Monitoring_mean_28992_30263.png What_does_Monitoring_mean_30264_31151.png What_does_Monitoring_mean_31152_32051.png What_does_Monitoring_mean_32052_33563.png What_does_Monitoring_mean_33564_34583.png What_does_Monitoring_mean_34584_34895.png What_does_Monitoring_mean_34896_35195.png What_does_Monitoring_mean_35196_35255.png What_does_Monitoring_mean_35256_35327.png What_does_Monitoring_mean_35328_35855.png What_does_Monitoring_mean_35856_36551.png What_does_Monitoring_mean_36552_37163.png What_does_Monitoring_mean_37164_37619.png What_does_Monitoring_mean_37620_38999.png What_does_Monitoring_mean_39000_44027.png What_does_Monitoring_mean_44028_44145.png