Performance Analysis in the Cloud
CC Attribution 4.0 International:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Chawla, Mohit
This talk situates the art and science of performance analysis (in the cloud), in a framework of reflective practice as outlined in Donald A. Schön's book "The Reflective Practitioner: How Professionals Think in Action" (1983). Placing performance analysis in such a framework can help operators transcend an overwhelming spectrum of challenges and focus on work that matters. For eg., its important to see through the common expectations that modern enterprises might have of cloud providers - a plug-and-play mindset, where plugging-in is easy, but where play can sometimes cost millions of dollars in operating cost, raising questions about the cloud's effectiveness as a platform. With such a mindset, performance engineering might be considered unimportant, where it might actually be most useful. But that's only the tip of the iceberg. Once an operator has gathered the grit and resources to continue with performance analysis - the assumptions of fellow engineers, developers and leaders must also be addressed, and a process of edification and implementation should be carried out - sometimes this can be a draining process for a single operator - as it involves answering misinformed skepticism from different directions. Last but not the least - meaningful performance analysis in the cloud might even be a pipe dream for most - as its not easy to get the right information (if at all) from cloud vendor engineers and architects, for deeper questions and investigations around available monitoring data, without which useful performance modeling is impossible. Nonetheless, the insights gained with such practice can be helpful overall, in defining performance and capacity planning strategies. In this talk specifically, many anecdotes about various software tools and enterprise application platforms will be discussed to illustrate the points mentioned above, which can be used by operators to make improvements in their environment. - for eg., * How to model disk usage for a popular big data software - with a queuing model, or linear regression? If using a queueing model, does your cloud vendor provide all the metrics? Are those metrics really what they appear? Does your operating system agree? * Metrics, metrics everywhere, but oops! there's a performance issue with the central metrics system. Metrics, metrics nowhere? * Spot is cheap, but show me the scaling metric. * Concurrency in serverless apps. * etc.
 Hi, everyone. Thank you for coming. Today I'm going to speak about performance analysis in the cloud, a reflective practitioner's perspective. Before I start, I assume that you guys know about what the cloud is and what performance generally means. But how many of you know about reflective practice or the ideas of reflection in action? That's fine. To introduce the idea of reflection in action, I want to first talk about an article written by the Danish astronomer Peter Nauer. He wrote an article called Programming as Theory Building. In the article, he emphasizes that programming should not be viewed as an activity whose sole purpose is the production of a program text. Instead, programming should be viewed as a process where a programmer or a team of programmers build an insight or a theory about an external reality which they convert into symbols of the choice of their programming language. One of the merits of such an approach to programming is that when it comes to modifying the program, during that time, the changes that are happening in the external reality, the programmer can see the similarities with those changes with the theory of the symbols that he's using, and so he can accommodate those changes easily. Now, I want to use this idea as it is and think about performance analysis in the same way. Thankfully, there's a framework of reflective practice which Donald Schoen wrote about in his book called The Reflective Practitioner, How Professionals Think in Action. In this book, Donald Schoen took ideas from many diverse fields like architecture, town planning, science-based professions, and engineering, and he tried to come up with what really does it mean to have a professional practice. So, I was, of course, very excited about making this proposal to Frostconn and thanks to them for accepting this, but I was very nervous as well when I was writing the proposal description, but I recently read another book called The New ABCs of Research by Ben Schneiderman. How many of you know who Ben Schneiderman is? Cool. Okay, for those of you who do not know, Ben Schneiderman, okay, how many of you know what a tree map is? Cool. And what a clickable hyperlink is? So, Ben Schneiderman created the clickable hyperlink. He's very influential in the fields of design and human-computer interaction as well as information visualization. So, he writes in his book The New ABCs of Research that the design community celebrates Donald Schoen's book for its illuminating discussion about creative problem solving by experienced professionals. So, this gave me quite a boost to think about my talk more. And with that, I'm going to describe reflection in action. This will be a more textual description, what's going to follow in the form of tables. I hope to convey you the ideas throughout as the talk progresses. So, reflection in action was a view of professional practice. And Donald Schoen compared reflection in action with a conventional school of thought called technical rationality, which followed the positivist view of professional practice. In the conventional view, means and ends were considered completely separate, and problem solving was quite simply the use of a procedure to achieve a predefined objective. In reflection in action, means and ends are framed interdependently, and the focus or the central idea is to set a problem or frame a problem on a situation in which the practitioner thinks about. In the conventional view of thought, research and practice have very clear boundaries, and practice becomes the application of research-based theories which are highly objective and are based on controlled experiment. Whereas in reflection in action, practice becomes a kind of research, and the practitioner conducts a reflective conversation with the situation, which also involves experiments which are rigorous, similarly rigorous as the research-based or the technical rationality view of research, but they are on the spot experiments within the frame that the practitioner imposes on a situation. Lastly, knowing and doing are inseparable in reflection in action. Once again, in the conventional view of thought, action just becomes implementation or a test of technical decision. Now, earlier I had a couple of examples from the book, but I had to remove those so that we can put all of this directly in the framework of performance analysis. So with that, I want to mention something that Donald Schoen writes in the chapter on engineering sciences. He writes about an article which was written by Harvey Brooks in 1967. Harvey Brooks was at Harvard during that time, and Harvey Brooks described the difficult situation of a practicing engineer who has to or who is expected to bridge the rapidly changing body of knowledge of whatever his science or chosen engineering field was with the expectations of society. Keep in mind that this was written in 1967 as well as consider the use of the word bridge in this slide because I'm going to come back to this. I believe that we are still, we are exactly in this situation for people who are working in this field and especially in a cloud-based environment. Let's consider what roles exist today in a modern organization. We have developers or system engineers or data scientists, and then we have team leads of the teams of such developers or data scientists or system engineers. Then you have managers of such teams, then you have managers of managers, and you have many other roles in between. And then there's the cloud. What unites all of these when it comes to their thoughts about the cloud? And before I carry on, how many of you guys actually work in IT? Okay, cool. So what unites all of them or what's common to all of them is what they speculate about the cloud. And on average, that speculation tends to be along the lines of that everything needs to be fully functional out of the box, and it needs to be more economical compared to the alternatives. The speculation is fine. I think all of us do that to an extent. But when those speculations start manifesting as expectations, that's when things can happen. And now I want to go back to the earlier slide where I have underlined the word bridge. I believe that the metaphor of bridge no longer holds in such an environment because I believe that some of these bridges are burning faster than they are being built. And so that's where a framework of reflection in action can help or benefit quite a bit to navigate such an environment. I was lucky or fortunate to be able to cross some of these bridges, and that's why I'm here today to share some of the things that I learned and hopefully help you in your process of discovery and experiment. The very first thing that we need to consider is performance modeling. This is the frame that a performance analyst brings to an environment, and this is the frame in which or this is the frame which enables any kind of experimentation or problem solving in a cloud environment. So problem solving is part of reflection in action, but it becomes the part of the overall experiment which is carried out with a particular frame. Now in the morning, I was a bit sad because I noticed that this time around the t-shirts at the conference are not being given early in the morning. Instead, you'll get those around noon. However, that was really good because my example about some ideas of queuing theory are going to be based on the t-shirt counter at Frostconn. I actually prefer the cold brew counter because I don't get this cold brew anywhere, so this is pretty cool. So yeah, that's the context of an example I'm going to give. In reflection in action, the frame is informed by certain guiding principles or overarching theory or the appreciate a system of the practitioner. And in case of performance analysis and performance modeling specifically, those guiding principles are queuing theory and statistics and data analysis. Just keep in mind that what I'm going to present now, it's more as a technician of queues and not a mathematician. This is in equal parts and admittance of a weakness, but also an expression of wonder. So keep that in mind and let's have some fun. Oh, well, not with those, but so let's see what a queue actually is or what does a queuing center mean. And keep in mind that now we are thinking about the t-shirt counter at Frostconn. So at the t-shirt counter, there's a helper or a volunteer who is servicing participants in the conference by giving out t-shirts. So now that the t-shirts are only being given starting at noon, we can expect a queue of participants at the counter. So participants come and maybe they have to wait because there might be other participants waiting to get their t-shirts. And of course, all of them would leave the counter at some point, hopefully with a t-shirt. So this forms the queue. The queue actually includes the participant who is being served right now, as well as the ones who are waiting. When queuing theory as a discipline was maturing, a notation was introduced for understanding different kinds of queues. We're going to take one example of the MM1 queue. This is considered as to be one of the more simpler models of queues. And in the notation, the first letter represents the arrival distribution. In our case, that would be the arrivals of the incoming participants to the t-shirt counter. The second one is the service distribution. And lastly, the number of servers or in our case, the FrostCon helpers who are giving out the t-shirts. There are a few statistical assumptions in this, but the one which is really important to keep in mind throughout is that we are talking about average quantities. With that, let's just take a look at some queuing metrics. One thing I want to mention at this point is that in reflection in action, there always is a medium in which the practitioner thinks about whatever he's doing. So in architecture, that could be a sketch pad. Well, today it might be software, but in a sketch pad, the practitioner can experiment with the designs that a building should have, where its location should be relative to other parts of the overall architecture and things like that. For queuing for performance analysis, a medium, a very nice medium to think about such things is a software called PDQ, and we'll learn more about this. Some of the metrics which are very important, actually, these are the most important metrics are the arrival rate, the service time. Service time is how long a helper takes to actually give out a t-shirt. So they have to find the t-shirt, they have to collect the money, maybe give some change back. Doing this for a customer on average is what the service time of the helper is, giving out a t-shirt. Then there are a couple of other ones. The response time is a very well-known metric. We'll see more about this. Then there's utilization. How you take, well, in case of computer systems utilization, a very simple example is CPU, for example, how you utilize the CPU while serving certain requests. And queue length, of course, it's one of the primary metrics that we think about. Once again, keep in mind we're talking about average quantities. This is a very important assumption whenever working with this, whenever working with queuing models generally. So a very quick example. Think about the inputs for a queue. We're going to use mathematical formulae to do this by hand first. And then I'll show you how PDQ approaches this. Consider the arrival rate of the participants at the t-shirt counter as being 0.75. This is simplified so that the calculation becomes easier. Consider also that on average, the helper takes about a minute to serve a participant. What we can expect as the outputs of just these two inputs to a queuing model are three metrics. The first one is utilization. And the calculated value can be obtained by arrival rate multiplied by the service time. And in this case, that happens to be 0.75. The response time, after using that formula, it comes to be four. So four minutes on average is the response time when there's a queue. And that's that. And I think that might increase when they actually open the, when they start giving out the t-shirts. And finally, the queue length is arrival rate times response time, and that's three. So this is just to give you an idea that you can calculate this by hand for a very simple queue like the MM1 queue that I just showed you. Let's see how PDQ does this. So how many of you are aware of the R language? Cool. So the example is based, or it uses the R library, the PDQ library in R. So let's take a look at how it looks like. Yeah. Cool. At the back. Cool. So let's do a very quick walkthrough through this. The very first thing we do is we load the library PDQ. We initialize the arrival rate of the participants and the service time of the helper. Now we actually start initializing the model parameters. So to initialize, we call our queuing center as the Frostcon t-shirt counter. So we initialize that. We can set an optional comment that this is the model of an MM1 queue at the Frostcon t-shirt counter. We define our queuing center in terms of who is going to serve the participants in our case. That's a Frostcon helper or a volunteer. And then we define our workload. We call the workload get a t-shirt, and the workload is characterized by the arrival rate of the participants, which was 0.75. Optionally, we can set the work unit. So in this case, the work unit is participants. If this was an application that you were developing, that would be how many requests it can handle, so requests per second or something like that. Time unit is minutes. Then we characterize the service demand of the helper himself or herself in terms of the workload, which is get a t-shirt, and how long it takes to serve on average one participant. With just these two inputs, we can solve the model just like we did, but let's see how PDQ does. In PDQ, the method to use is solve, and then we get a report. So let's see what kind of report PDQ generates. So let's see how the PDQ report actually looks like. So we are developing a model for the Frostcon t-shirt counter, and for our own reference, we know that we are doing a model of the MM1Q at the t-shirt counter. Then we define our inputs. Our workload parameters is basically what kind of arrival rate is there, what's the workload we are characterizing, and what's the service demand. So that's in this section of the PDQ model. And then finally, when we actually solve the, or invoke the solve function, we get the outputs. So we can see the system performance and the resource performance. The system performance includes things like the number and system. This is the queue length that I mentioned briefly. So this says on average, there are about three participants in the queue who want a t-shirt or are already getting a t-shirt. The throughput of the helper is about 0.75. This is, as you might have noticed, similar, same as the arrival rate. In this case, the average response time is about four minutes. Now let's take a look at the resource performance. So this is about the helper. And for the workload get a t-shirt, the helper is about 75% utilized. It doesn't make sense, but think about if this was an application. In that case, it would be a server utilization that we'll be talking about. But these are some of the parameters which define the resource performance. So that's a very brief demo of what PDQ can do. Now you might think, well, if we can already do this by hand, what's the need for things like PDQ? Now imagine that Frostcon is actually a flow of participants, organizers, speakers who are coming in and out of the halls and the counters at the dev rooms. If we were to characterize the queuing nature of all the counters or maybe even the rooms, that would be almost impossible to solve by hand. And that's where PDQ comes in for doing or taking a takes, PDQ takes an analytical approach towards this. And in fact, PDQ stands for pretty damn quick. So it does that computation quite fast. And that's, so think about Frostcon as a network of queues. That's where analytical tools like PDQ are quite important and in fact, indispensable for doing any meaningful performance analysis. So that's a bit of theory, a bit of theory and example. We are now going to go into how this looks like in a production environment which is based on the cloud in AWS. But before I carry on any questions so far. All right, cool. So consider a Java application running on multiple EC2 instances which receive requests via a load balancer. The EC2 instances are part of an auto scaling group which dynamically scales the instances based on their average CPU utilization. If we take a closer look at the application, a request comes in and the Apache web server does some pre-processing of the request, sends it to the Tomcat application server where the application code is also deployed and the application sends out further requests to some external services. The external services reply back with a few responses. The application does some actual business processing, business logic processing and sends out a few responses out or sends the response out to the user. Let's pause here for a second and consider the amount of data you have if you were to think about performance of this particular application. So you have not less than 50 to 100 metrics just within the scope of the architecture, very simplified architecture that I have showed you. At the load balancer level, there are at least about 10 metrics that you can use. There are metrics at the auto scaling group level. Then there are metrics at EC2 instance level. Of course, at the Apache web server, Tomcat and finally the application instrumentation itself which might be within itself about 100 or more metrics. Let's not forget the operating system as well. So with that amount of data and that amount of metrics, where do you actually start looking at if you want to come up with a capacity model for your application? Well, a lot of you might already have time series dashboards like these, perhaps even more complicated than these. However, this is not useful to be able to analyze the performance of a system. Let's consider some of the, or let's consider the most important performance metrics to characterize the performance of an application. Some of the, two of the well-known or popular metrics are throughput and response time. Throughput means how many requests the application can handle in a given period of time. Response time is on average how long it takes to serve or serve those requests. Equally important, in fact, I consider these to be the more important derived performance metrics which are given by a very important law called the Little's Law. This gives us the number of concurrent requests which is the product of the throughput and response time. And the second law, and the second version of the law is it relates utilization as the product of the throughput and the service time. However, we are going to consider the service time from this. So service time can be described as utilization over throughput. Why are these derived performance metrics more important? Because these actually express the relationships between the metrics and it's those relationships which determine how you should plan for your capacity. So from a speculation about that amount of data or metrics available to understand the performance of your application, which by the way might be running globally in multiple AWS regions and data centers, out of all of that, we come down to just about a handful of metrics. These ones are the measured throughput, the measured concurrency, the estimated service time, the measured response time, and the measured utilization. To get to these metrics, I have as a reference, there's a slide deck at the end of the presentation. You can refer to that because that tells you the internals of Tomcat how you can get some of these metrics. So that was the data collection or data gathering part. Now, how do you actually model that to understand the performance or the capacity of your system? First of all, we need to take a time independent view here. We need to consider the steady state of the application. So we cannot have variations in the application while we are collecting all this data. That's why Time Series doesn't work for performance analysis. We need time independent view. Now, within the frame of queuing theory, we can expect that a throughput curve look like this, having a concave shape compared to the user load. And the response time to look like a hockey stick handle where it increases linearly with the user load. One thing I forgot to mention is that now we are discussing a completely different model than what I showed before. So that was the example, very basic example of an MM1 queue, but these models are completely different. So we had some data to start with. This is a scatterplot of data that I collected in July 2016. On the y-axis, you'd see the throughput. On the y-axis, on the x-axis is the number of concurrent users. After we actually did the modeling, the blue line here represents outputs from the PDQ model. And we can see that, visually at least, that the model seems to do a good job of explaining the performance characteristics of the application. When we actually started evaluating the model, of course, it looked good visually, but there's a technique called dummy queues in PDQ, and we used about 350 of those to come up with the right values of certain parameters. So the question at that point became, well, what do these dummy queues represent? Is there some sort of parallelism in the application, or does this describe some kind of polling behavior? A few months passed, and I collected some more data, and what happened was the new data actually broke the model completely. So we had to do something here. After we adjusted the model, some key differences that we had to make to the model were that the service time initially was based on the CPU utilization or the CPU time, and that was about a millisecond or so. In the new model, instead of modeling the CPU as the resource or the server, or characterizing the server, if you remember the diagram before, instead of that, we were now modeling Tomcat threads themselves. And so the service time was now characterizing the service time of, on average, of a Tomcat thread which serves an HTTP request. Of course, we had to validate this data, this model as well. So we used the data, and we can see that from the adjusted PDQ model, the data does conform to the model, and so this validated the use of that model. Now once again, talking about reflection in action, right? This was an introduction to the frame of queuing theory that you can use when doing performance analysis in the cloud or otherwise, but problem solving, as I mentioned, is still part of the overall experiment or the overall frame, and these experiments are not less rigorous than some of the controlled experiments that happen in research. So to be able to do that, we need some on-the-spot experiments, and in this case, I'm talking about the spot offering from AWS as well. How many of you know about Spot? So Spot is an offering, very nice offering from AWS. It has a few nice properties that you can save up to 90% compared to the on-demand price that AWS has. However, to be able to save that kind of money, you need to improve your chances of getting an instance in the first place, and you can do that by diversifying your kind of instances across availability zones, their types and their sizes. Just as an example, you might have tested your application on an M4 10x large instance with many number of CPU cores, but the market might just only have M4 2x large instances during a specific time, so what do you do during that time? So you need to be able to reconfigure your application, consider all of these when you want to roll this thing out. The other nice thing about AWS Spot is that you can use auto-scaling directly with Spot, and it's now very seamless. However, the usual metrics that are the more common metrics of CPU utilization latency throughput, they no longer work, because if you have an instance which has about 40 cores and you have an instance which just has four cores, the average CPU utilization is completely bad for such an heterogeneous environment. And this time, what you can do is you can use the number of concurrent requests. As I mentioned before, little slow or concurrency, this gives us the relationship between the performance metrics, and that's why you already know what kind of behavior to expect if you use N or the number of concurrent requests as the metric. The CPU latency and throughput metrics are more reactive metrics. So this was a very good example of a rigorous on-the-sword experiment which was informed by some guiding principles from Kewing Theory and the enterprise benefited by saving cost, their operating cost of a very huge infrastructure. Of course, we had to still test our model with the new data after we made these changes, and it did conform to the model, so that was validated as well. And this is from the recent data about March this year, so the new model that we came up with worked for all the data. So one thing we can appreciate here is that, or you have to keep in mind, is that the application was going through many architectural changes, release cycles, all sorts of changes are happening, and that's what makes it important that a performance analyst has a frame in which he can think or accommodate these changes, and that's where I think reflection in action can be very indispensable to be able to navigate such an enterprise. Let's take another example, or before this, any questions so far? Cool. Let's take another example of disk utilization. Now, everybody says that storage is cheap, or everybody knows that storage is cheap and it's fast, but managing disks is still far from being easy. Let's take examples of two very popular software, Kafka and Graphite. These anecdotes are slightly different from what I have been talking about so far, and we'll see how they are different. The first point of difference which I can mention already is that this involved interaction with AWS quite a lot, so this had feedback from AWS team on possible reasons for some of the behavior that we were seeing. So in a large infrastructure, if you're using a central metric system or hosting your own central metric system, it becomes very important that you do proper capacity planning for that itself because of the number of metrics that you might have to deal with today in a modern enterprise with microservices, with hybrid environments, with the cloud, and perhaps on premise, or all sorts of variations on this. It becomes important to be able to plan for the capacity. In this case, we are going to talk about the disks because all of that time-seized data is lying on many disks. What was interesting in this example was that, in this case, was that the performance issue was discovered during the process of data collection itself. So before we could even think about modeling, within we discovered certain things during the data collection itself. What I observed was that the EBS volumes, just so you know, EBS is the storage offering from AWS, these get attached to your instance. What I observed was that these EBS volumes were getting less IOPS, which is a measure of throughput compared to what they were capable of. So the question was, where is all the throughput going while we're not getting the promised performance? There were many investigations carried out by AWS and myself. I shared dashboards that I had for our infrastructure from their side. Also, they tried to share something that the view that they had or that particular engineer had. What we found out was, after many of these conversations, that the volume or the activity was actually being throttled. However, this was not transparent to us at all or not easily because this required the engineer to point out and he also had to investigate on the AWS side to figure out what was really happening. And the reason for throttling was on the operating system level, we saw that IOPS that was showing that the writes per second is very high. What we did then or what was suggested by AWS was that we should configure the application to reduce that activity. So I did that and it did stabilize the behavior of Graphite, but this seemed like a compromise because we're still not getting the performance that the volume was capable of. So this pointed out questions about the architecture of Graphite itself and how EBS works. And that takes me to another point in reflection in action, which is, as I said, it's not always about problem solving, but once you have pursued a line of thought in your frame, it should open up a further line of inquiry. It is not supposed to end a particular investigation or so. It should always open a further line of inquiry and that's what happened in this case that we figured out, well, you know, we need to do this for Graphite. It's not ideal, but at least Graphite provided that those knobs, maybe if this was some other application, we might not have been that lucky. So this sort of signifies that we should ask the right questions to your cloud vendor to get the right information. The other example was about Kafka. Once again, now, I was considering queuing theory as the guiding principle and what I set out to do was to collect metrics which would help predicting the disk usage because one thing that modeling is done for is actually to predict how your application would behave with changes in the workload that it handles. So that's very central to the idea of modeling and based on that guiding principle, I thought of getting some data for predicting disk usage of Kafka. However, when I did collect the data, it did not make much sense. On the one hand, there were metrics from EBS volumes like queue length, throughput and latency, which to me sounded so encouraging because I thought this is just going to work out out of the box and we could predict disk usage quite easily. But comparing the metrics from EBS side with the operating system level metrics and then to figure out one of the very important metrics to characterize a queue is the service time. If we cannot find the service time of an application, we cannot do queuing models at all. Now, in case of Linux, in the main page for IOS tab, we see that for the service time field that warning do not trust this field anymore. This will be removed in a future version. With that kind of ambiguity about the metric and the lack of congruence between the operating system metrics and the EBS metrics, it was next, I mean, for me at least, it was becoming quite hard to be able to do this in a meaningful manner. I started talking with the AWS engineers and this was a very different suggestion. Instead of trying to validate the modeling techniques that I was trying to use, one of the suggestions that AWS team had was that instead of doing that, use statistical methods like linear regression. So that, once again, is sort of a new line of inquiry that triggered us on and from there we tried to predict the disk usage of Kafka. The next thing we're going to talk about is serverless applications and we're going to focus on the concurrency part of serverless applications. This is, once again, a different example from the previous ones, but the idea of concurrency is central to pretty much all of them. So how many of you have actually used serverless architecture or Lambda functions in AWS? Nice. In Lambda or serverless offering that AWS provides, what does concurrency even mean? So the invocation of a Lambda function is a basic unit of concurrency that you think about the concurrency in terms of the number of invocations of the function. Lambda requires you to specify the memory and AWS figures out the appropriate CPU accordingly. In some sense, this does sound to me like the Lambda function acts like a server in the queuing theory sense, but that's just in some sense, I haven't pursued that line of thought much further. So in Lambda, concurrency can be represented as events per second times the function duration and this is exactly the form of Little's law. So Little's law is not native to queuing theory. In fact, it's actually very generally applicable to a host of fields. Very prominent field is the field of operations research where Little's law is quite well known. However, to me, I jumped off my feet when I saw that this is exactly the form of Little's law and this informed my ideas about how best to program a particular task that I was doing which involved copying of a large number of files from one S3 bucket to another bucket over the network. So during that time, I had to figure out how to measure the concurrency. There are no out of the box ways to do this as far as when at least when I did this. So you have to instrument your code to calculate the correct events rate. Events are basically what trigger your Lambda function. So you have to figure out what's the rate of these events. As there's a reference at the end of the presentation, you can use to calculate that. By default, it's limited to about a thousand concurrent executions. After that, throttling kicks in and throttling could also be in some sense considered to be a queuing mechanism because in a way, based on the incoming events, your Lambda function is not being invoked for all of them. So AWS throttles you after about a thousand concurrent executions by default. So in such a scenario, how do you come up with an optimal concurrency? The approach. What I ended up doing was to use a mix of threads and Lambda's in different environments. So from our on-premise instance, I had a piece of code which was based on threads. We were using Python at the time. And on the Lambda side, earlier, there was just threads and Lambda. So the threads used to call Lambda functions and that was it. But then what I found out was even within a Lambda, you can make the processing faster by having threads inside the Lambda. So it was overall a mix of approaches. And there's a reference for this as well if you want more details. The next one is slightly different. But this is also one of the elements of the frame because monitoring, even though essential for doing operations, we need a new idea of monitoring when it comes to the cloud. There was a media theorist, American media theorist by the name of Marshall McLuhan. He came up with the phrase, the medium is the message. Actually, when the book came out, that phrase was accidentally changed to the medium is the massage. And we kind of see this in the operations teams these days because once you see that some of your monitors don't work as expected, there's a tendency to just let it be and repeat the same things that you have been doing, follow a run book or something like that, remove some files from the temporary folder so that there's more disk space, things like that. However, in the cloud, that doesn't really work for long. What is required is to think about the medium with more focus. As a very simple example of the kind of differences that the cloud has compared to other environments, let's consider the use of notebooks versus dashboards. How many of you know the general idea of notebooks? Well, maybe I can show you very quickly. If we use R as an example, we have the option to create an R notebook. It allows you to have pieces of code with certain documentation or other things that can insert the notebook. It becomes kind of a, for the lack of a better word, narrative. You can think about what you're doing, run it by other, sort of discuss around these ideas. And in this case, for example, we could, so that's one, we could add another section to this and say just, let's see, so I'm not sure if this is going to load the, okay. So let's say we are a team of, well, in this case, status issues and I want to share a scatter plot of data with you. And I mean, if you're at my desk and we start talking about this, after that, I just want to print a string. Let's say, yeah, so it prints a string, you can go on like this. In a dashboard, you would have, you would already have a plot, so to say, or many plots. And to make changes, you will have to sort of take it offline in the sense when you're editing it, you can no longer discuss the ideas that you're trying to do with the person who is at your desk. Now, in case of monitoring in the cloud, this becomes very important because due to the dynamic nature of the cloud, you cannot call meetings for all the things that might be breaking all the time. You need, if a group of software engineers come to your desk and you want to quickly discuss or brainstorm a few ideas on why their application might be behaving in a certain way, notebooks are much better for that kind of thing. So that, for me, kind of reflects the change of thinking that's required when considering monitoring in the cloud. I have more examples and these are some things which I learned, some of these, the hardware, and this is about we need better metrics to start with and also better programming language support for coming up with good monitoring and to be able to calculate or find all the performance metrics. The very first thing I noticed is that when it comes to manipulating metrics or instrumenting your code, I was very surprised that Python did not have as strong a library as Java has in terms of JMX, like Java has JMX. And it's so comprehensive that I did not have to spend too long to use that interface in the Tomcat application I showed you before, whereas in Python, I couldn't find a solid stable library which had similar interfaces and that was, so that's why I had to manually instrument a bit of Python code and luckily that worked, but I wasn't sure if I could do that again for other things. The second example is of monitoring Tomcat, a very common way of thinking or way of approaching Tomcat's monitoring or performances in terms of number of busy threads. In reality, that's a lot less significant than the number of threads which are actually serving requests. These are the threads which are in the service stage and if you remember the model that I showed you before, they are the ones which are actually servicing requests and that number is not the same as the number of busy threads and this becomes very important while doing the modeling and this is not even easy to obtain. So there's no direct metric for this that Tomcat exposes, you have to write your own script or a program to fetch this metric. Lastly, as I mentioned that anything in reflection in action, any hypothesis or experiment that you're doing, it should lead you to a further line of inquiry and in my case that was quite literally to find out where are all the cues forming in the whole environment in the operating system, in the cloud, components in between. One example I want to share is that we were, for our data pipeline, we were seeing a lot of delays in the processing and transfer of data and that was because we saw at the operating system level that the send queue values in that stat were so high for a long period of time and that's why the transfers were not even taking place for a particular period of time. This one required even more conversations and discussions with the AWS team. I think this extended over months and by the end of it, not only we had to change the application code but also the S3 SDK code with of course a lot of rules of thumb and best practices and general ideas being shared across both sides but this sort of tells you that if you take this into account in your application design and application architecture, what I'm saying is performance analysis, if you make that as part of a first class citizen of your application design, some of these things can be considered from the very beginning. This is more of a quiz time when it comes to data visualization but before that, are there any questions? Okay. So the quiz is like this. How many data stores do you know? MySQL, Postgres, Kafka, so between three and five, between one and three? Okay. Between three and five or one and three? Good. And keep in mind how many of you actually use in your work as well so if you have production used, that's more important. So that's just the data stores. How many web servers do you know of? Apache, Nginx, many little web servers, application specific web servers, so I guess still around three or five, between that, window three or five. How many programming languages do you know? More than 10, 20? It's a huge number, right? And how many cloud vendors do you know? Or IT infrastructure hosting providers do you know? What I'm trying to get at is think about the one developer or the one system engineer who is supposed or is very likely there is a system engineer's job. Think about the system engineer who has to make sure that all of these systems are always up and running and performing well. What I'm trying to get at is all of these, let's consider the data stores, for example, the data stores, most of them actually come up with some kind of management interfaces like they expose some monitoring data with their own dashboards. And similarly, some of the web servers, they also expose their own metrics, might have dashboards of their own. What I found out is at least that not one of, let me ask this as a question, how many of those do you think follow strong visualization principles when it comes to conveying that information which they think is important to run their application, but they're not thinking from what I think, they're not thinking about that one guy, one system engineer who is looking at all of that. Of course, it's a team of those guys, but they are still, you guys know about the idea of on-call, right? We rotate and we have to keep the systems always up and running. I believe that for me at least this is a much more important line, it's an equally important line of inquiry for all of us that we need to have instead, well, DevOps, SREs, that's all good. But I think the next phase for that should be visualization informed operations or development. And right now, if I can't show the data stores, but what I remember is two very important or very popular enterprise ready business high performance, all of that, two data stores, one of them showed data just for the last 10 minutes, which is fine, but the other data store shows it for a longer time, but doesn't show you the metrics which make sense from performance perspective. So what I'm trying to say is pretty much all of them want you to interpret their metrics in a terminate logic that they have invented. And that's what makes it a lot harder for a performance analyst to be able to correlate all that, even if, as I said, some of these bridges are burning faster than they are being built. So even if you can, it's not always possible to make sense of all the data out there. And yeah, I think that was the last note. Some references. So a lot of the performance modeling stuff I learned from Dr. Neil Gunther, he runs his own company. I can highly recommend his classes. Then the couple of links I shared before, they are part of the slide deck which I'll upload on Frostcon portal. And so we can have some Q&A right now. How many of you are actually running production workloads in the cloud? Cool. All right. I guess then that's it. Thanks a lot for your time. And hopefully see you next year.
 Hi, everyone. Thank you for coming. Today I'm going to speak about performance analysis  in the cloud, a reflective practitioner's perspective. Before I start, I assume that  you guys know about what the cloud is and what performance generally means. But how many  of you know about reflective practice or the ideas of reflection in action? That's fine.  To introduce the idea of reflection in action, I want to first talk about an article written  by the Danish astronomer Peter Nauer. He wrote an article called Programming as Theory Building.  In the article, he emphasizes that programming should not be viewed as an activity whose  sole purpose is the production of a program text. Instead, programming should be viewed  as a process where a programmer or a team of programmers build an insight or a theory  about an external reality which they convert into symbols of the choice of their programming  language. One of the merits of such an approach to programming is that when it comes to modifying  the program, during that time, the changes that are happening in the external reality,  the programmer can see the similarities with those changes with the theory of the symbols  that he's using, and so he can accommodate those changes easily. Now, I want to use this  idea as it is and think about performance analysis in the same way. Thankfully, there's  a framework of reflective practice which Donald Schoen wrote about in his book called The  Reflective Practitioner, How Professionals Think in Action. In this book, Donald Schoen  took ideas from many diverse fields like architecture, town planning, science-based professions,  and engineering, and he tried to come up with what really does it mean to have a professional  practice. So, I was, of course, very excited about making this proposal to Frostconn and  thanks to them for accepting this, but I was very nervous as well when I was writing the  proposal description, but I recently read another book called The New ABCs of Research  by Ben Schneiderman. How many of you know who Ben Schneiderman is? Cool. Okay, for those  of you who do not know, Ben Schneiderman, okay, how many of you know what a tree map  is? Cool. And what a clickable hyperlink is? So, Ben Schneiderman created the clickable  hyperlink. He's very influential in the fields of design and human-computer interaction as  well as information visualization. So, he writes in his book The New ABCs of Research that  the design community celebrates Donald Schoen's book for its illuminating discussion about  creative problem solving by experienced professionals. So, this gave me quite a boost to think about  my talk more. And with that, I'm going to describe reflection in action. This will be  a more textual description, what's going to follow in the form of tables. I hope to convey  you the ideas throughout as the talk progresses. So, reflection in action was a view of professional  practice. And Donald Schoen compared reflection in action with a conventional school of thought  called technical rationality, which followed the positivist view of professional practice.  In the conventional view, means and ends were considered completely separate, and problem  solving was quite simply the use of a procedure to achieve a predefined objective. In reflection  in action, means and ends are framed interdependently, and the focus or the central idea is to set  a problem or frame a problem on a situation in which the practitioner thinks about. In  the conventional view of thought, research and practice have very clear boundaries, and  practice becomes the application of research-based theories which are highly objective and are  based on controlled experiment. Whereas in reflection in action, practice becomes a kind  of research, and the practitioner conducts a reflective conversation with the situation,  which also involves experiments which are rigorous, similarly rigorous as the research-based  or the technical rationality view of research, but they are on the spot experiments within  the frame that the practitioner imposes on a situation. Lastly, knowing and doing are  inseparable in reflection in action. Once again, in the conventional view of thought, action  just becomes implementation or a test of technical decision. Now, earlier I had a couple of examples  from the book, but I had to remove those so that we can put all of this directly in the  framework of performance analysis. So with that, I want to mention something that Donald  Schoen writes in the chapter on engineering sciences. He writes about an article which  was written by Harvey Brooks in 1967. Harvey Brooks was at Harvard during that time, and  Harvey Brooks described the difficult situation of a practicing engineer who has to or who  is expected to bridge the rapidly changing body of knowledge of whatever his science  or chosen engineering field was with the expectations of society. Keep in mind that this was written  in 1967 as well as consider the use of the word bridge in this slide because I'm going  to come back to this. I believe that we are still, we are exactly in this situation for  people who are working in this field and especially in a cloud-based environment. Let's consider  what roles exist today in a modern organization. We have developers or system engineers or  data scientists, and then we have team leads of the teams of such developers or data scientists  or system engineers. Then you have managers of such teams, then you have managers of managers,  and you have many other roles in between. And then there's the cloud. What unites all  of these when it comes to their thoughts about the cloud? And before I carry on, how many  of you guys actually work in IT? Okay, cool. So what unites all of them or what's common  to all of them is what they speculate about the cloud. And on average, that speculation  tends to be along the lines of that everything needs to be fully functional out of the box,  and it needs to be more economical compared to the alternatives. The speculation is fine.  I think all of us do that to an extent. But when those speculations start manifesting  as expectations, that's when things can happen. And now I want to go back to the earlier slide  where I have underlined the word bridge. I believe that the metaphor of bridge no longer  holds in such an environment because I believe that some of these bridges are burning faster  than they are being built. And so that's where a framework of reflection in action can help  or benefit quite a bit to navigate such an environment. I was lucky or fortunate to be  able to cross some of these bridges, and that's why I'm here today to share some of the things  that I learned and hopefully help you in your process of discovery and experiment.  The very first thing that we need to consider is performance modeling. This is the frame  that a performance analyst brings to an environment, and this is the frame in which or this is  the frame which enables any kind of experimentation or problem solving in a cloud environment.  So problem solving is part of reflection in action, but it becomes the part of the overall  experiment which is carried out with a particular frame.  Now in the morning, I was a bit sad because I noticed that this time around the t-shirts  at the conference are not being given early in the morning. Instead, you'll get those  around noon. However, that was really good because my example about some ideas of queuing  theory are going to be based on the t-shirt counter at Frostconn. I actually prefer the  cold brew counter because I don't get this cold brew anywhere, so this is pretty cool.  So yeah, that's the context of an example I'm going to give. In reflection in action,  the frame is informed by certain guiding principles or overarching theory or the appreciate a system  of the practitioner. And in case of performance analysis and performance modeling specifically,  those guiding principles are queuing theory and statistics and data analysis. Just keep  in mind that what I'm going to present now, it's more as a technician of queues and not  a mathematician. This is in equal parts and admittance of a weakness, but also an expression  of wonder. So keep that in mind and let's have some fun. Oh, well, not with those, but  so let's see what a queue actually is or what does a queuing center mean. And keep in mind  that now we are thinking about the t-shirt counter at Frostconn. So at the t-shirt counter,  there's a helper or a volunteer who is servicing participants in the conference by giving out  t-shirts. So now that the t-shirts are only being given starting at noon, we can expect  a queue of participants at the counter. So participants come and maybe they have to wait  because there might be other participants waiting to get their t-shirts. And of course,  all of them would leave the counter at some point, hopefully with a t-shirt. So this forms  the queue. The queue actually includes the participant who is being served right now,  as well as the ones who are waiting. When queuing theory as a discipline was maturing,  a notation was introduced for understanding different kinds of queues. We're going to  take one example of the MM1 queue. This is considered as to be one of the more simpler  models of queues. And in the notation, the first letter represents the arrival distribution.  In our case, that would be the arrivals of the incoming participants to the t-shirt counter.  The second one is the service distribution. And lastly, the number of servers or in our  case, the FrostCon helpers who are giving out the t-shirts. There are a few statistical  assumptions in this, but the one which is really important to keep in mind throughout  is that we are talking about average quantities. With that, let's just take a look at some  queuing metrics. One thing I want to mention at this point is that in reflection in action,  there always is a medium in which the practitioner thinks about whatever he's doing. So in architecture,  that could be a sketch pad. Well, today it might be software, but in a sketch pad, the  practitioner can experiment with the designs that a building should have, where its location  should be relative to other parts of the overall architecture and things like that. For queuing  for performance analysis, a medium, a very nice medium to think about such things is  a software called PDQ, and we'll learn more about this. Some of the metrics which are  very important, actually, these are the most important metrics are the arrival rate, the  service time. Service time is how long a helper takes to actually give out a t-shirt. So they  have to find the t-shirt, they have to collect the money, maybe give some change back. Doing  this for a customer on average is what the service time of the helper is, giving out  a t-shirt. Then there are a couple of other ones. The response time is a very well-known  metric. We'll see more about this. Then there's utilization. How you take, well, in case of  computer systems utilization, a very simple example is CPU, for example, how you utilize  the CPU while serving certain requests. And queue length, of course, it's one of the primary  metrics that we think about. Once again, keep in mind we're talking about average quantities.  This is a very important assumption whenever working with this, whenever working with queuing  models generally. So a very quick example. Think about the inputs for a queue. We're  going to use mathematical formulae to do this by hand first. And then I'll show you how  PDQ approaches this. Consider the arrival rate of the participants at the t-shirt counter  as being 0.75. This is simplified so that the calculation becomes easier. Consider also  that on average, the helper takes about a minute to serve a participant. What we can  expect as the outputs of just these two inputs to a queuing model are three metrics. The  first one is utilization. And the calculated value can be obtained by arrival rate multiplied  by the service time. And in this case, that happens to be 0.75. The response time, after  using that formula, it comes to be four. So four minutes on average is the response time  when there's a queue. And that's that. And I think that might increase when they actually  open the, when they start giving out the t-shirts. And finally, the queue length is arrival rate  times response time, and that's three. So this is just to give you an idea that you can calculate  this by hand for a very simple queue like the MM1 queue that I just showed you. Let's  see how PDQ does this. So how many of you are aware of the R language? Cool. So the example  is based, or it uses the R library, the PDQ library in R. So let's take a look at how  it looks like. Yeah. Cool. At the back. Cool. So let's do a very quick walkthrough through  this. The very first thing we do is we load the library PDQ. We initialize the arrival  rate of the participants and the service time of the helper. Now we actually start initializing  the model parameters. So to initialize, we call our queuing center as the Frostcon t-shirt  counter. So we initialize that. We can set an optional comment that this is the model  of an MM1 queue at the Frostcon t-shirt counter. We define our queuing center in terms of who  is going to serve the participants in our case. That's a Frostcon helper or a volunteer.  And then we define our workload. We call the workload get a t-shirt, and the workload is  characterized by the arrival rate of the participants, which was 0.75. Optionally, we can set the  work unit. So in this case, the work unit is participants. If this was an application  that you were developing, that would be how many requests it can handle, so requests per  second or something like that. Time unit is minutes. Then we characterize the service  demand of the helper himself or herself in terms of the workload, which is get a t-shirt,  and how long it takes to serve on average one participant. With just these two inputs,  we can solve the model just like we did, but let's see how PDQ does. In PDQ, the method  to use is solve, and then we get a report. So let's see what kind of report PDQ generates.  So let's see how the PDQ report actually looks like. So we are developing a model for the  Frostcon t-shirt counter, and for our own reference, we know that we are doing a model  of the MM1Q at the t-shirt counter. Then we define our inputs. Our workload parameters  is basically what kind of arrival rate is there, what's the workload we are characterizing,  and what's the service demand. So that's in this section of the PDQ model. And then finally,  when we actually solve the, or invoke the solve function, we get the outputs. So we can see  the system performance and the resource performance. The system performance includes things like  the number and system. This is the queue length that I mentioned briefly. So this says on  average, there are about three participants in the queue who want a t-shirt or are already  getting a t-shirt. The throughput of the helper is about 0.75. This is, as you might have  noticed, similar, same as the arrival rate. In this case, the average response time is  about four minutes. Now let's take a look at the resource performance. So this is about  the helper. And for the workload get a t-shirt, the helper is about 75% utilized. It doesn't  make sense, but think about if this was an application. In that case, it would be a server  utilization that we'll be talking about. But these are some of the parameters which define  the resource performance. So that's a very brief demo of what PDQ can do. Now you might  think, well, if we can already do this by hand, what's the need for things like PDQ?  Now imagine that Frostcon is actually a flow of participants, organizers, speakers who  are coming in and out of the halls and the counters at the dev rooms. If we were to characterize  the queuing nature of all the counters or maybe even the rooms, that would be almost  impossible to solve by hand. And that's where PDQ comes in for doing or taking a takes, PDQ  takes an analytical approach towards this. And in fact, PDQ stands for pretty damn quick.  So it does that computation quite fast. And that's, so think about Frostcon as a network  of queues. That's where analytical tools like PDQ are quite important and in fact, indispensable  for doing any meaningful performance analysis. So that's a bit of theory, a bit of theory  and example. We are now going to go into how this looks like in a production environment  which is based on the cloud in AWS. But before I carry on any questions so far. All right,  cool. So consider a Java application running on multiple EC2 instances which receive requests  via a load balancer. The EC2 instances are part of an auto scaling group which dynamically  scales the instances based on their average CPU utilization. If we take a closer look  at the application, a request comes in and the Apache web server does some pre-processing  of the request, sends it to the Tomcat application server where the application code is also  deployed and the application sends out further requests to some external services. The external  services reply back with a few responses. The application does some actual business  processing, business logic processing and sends out a few responses out or sends the  response out to the user. Let's pause here for a second and consider the amount of data  you have if you were to think about performance of this particular application. So you have  not less than 50 to 100 metrics just within the scope of the architecture, very simplified  architecture that I have showed you. At the load balancer level, there are at least about  10 metrics that you can use. There are metrics at the auto scaling group level. Then there  are metrics at EC2 instance level. Of course, at the Apache web server, Tomcat and finally  the application instrumentation itself which might be within itself about 100 or more metrics.  Let's not forget the operating system as well. So with that amount of data and that amount  of metrics, where do you actually start looking at if you want to come up with a capacity  model for your application? Well, a lot of you might already have time series dashboards  like these, perhaps even more complicated than these. However, this is not useful to  be able to analyze the performance of a system. Let's consider some of the, or let's consider  the most important performance metrics to characterize the performance of an application.  Some of the, two of the well-known or popular metrics are throughput and response time.  Throughput means how many requests the application can handle in a given period of time. Response  time is on average how long it takes to serve or serve those requests. Equally important,  in fact, I consider these to be the more important derived performance metrics which are given  by a very important law called the Little's Law. This gives us the number of concurrent  requests which is the product of the throughput and response time. And the second law, and  the second version of the law is it relates utilization as the product of the throughput  and the service time. However, we are going to consider the service time from this. So  service time can be described as utilization over throughput. Why are these derived performance  metrics more important? Because these actually express the relationships between the metrics  and it's those relationships which determine how you should plan for your capacity. So  from a speculation about that amount of data or metrics available to understand the performance  of your application, which by the way might be running globally in multiple AWS regions  and data centers, out of all of that, we come down to just about a handful of metrics. These  ones are the measured throughput, the measured concurrency, the estimated service time, the  measured response time, and the measured utilization. To get to these metrics, I have as a reference,  there's a slide deck at the end of the presentation. You can refer to that because that tells you  the internals of Tomcat how you can get some of these metrics. So that was the data collection  or data gathering part. Now, how do you actually model that to understand the performance or  the capacity of your system? First of all, we need to take a time independent view here.  We need to consider the steady state of the application. So we cannot have variations  in the application while we are collecting all this data. That's why Time Series doesn't  work for performance analysis. We need time independent view. Now, within the frame of  queuing theory, we can expect that a throughput curve look like this, having a concave shape  compared to the user load. And the response time to look like a hockey stick handle where  it increases linearly with the user load. One thing I forgot to mention is that now  we are discussing a completely different model than what I showed before. So that was the  example, very basic example of an MM1 queue, but these models are completely different.  So we had some data to start with. This is a scatterplot of data that I collected in  July 2016. On the y-axis, you'd see the throughput. On the y-axis, on the x-axis is the number  of concurrent users. After we actually did the modeling, the blue line here represents  outputs from the PDQ model. And we can see that, visually at least, that the model seems  to do a good job of explaining the performance characteristics of the application. When we  actually started evaluating the model, of course, it looked good visually, but there's  a technique called dummy queues in PDQ, and we used about 350 of those to come up with  the right values of certain parameters. So the question at that point became, well, what  do these dummy queues represent? Is there some sort of parallelism in the application,  or does this describe some kind of polling behavior? A few months passed, and I collected  some more data, and what happened was the new data actually broke the model completely.  So we had to do something here. After we adjusted the model, some key differences that we had  to make to the model were that the service time initially was based on the CPU utilization  or the CPU time, and that was about a millisecond or so. In the new model, instead of modeling  the CPU as the resource or the server, or characterizing the server, if you remember  the diagram before, instead of that, we were now modeling Tomcat threads themselves. And  so the service time was now characterizing the service time of, on average, of a Tomcat  thread which serves an HTTP request. Of course, we had to validate this data, this  model as well. So we used the data, and we can see that from the adjusted PDQ model,  the data does conform to the model, and so this validated the use of that model.  Now once again, talking about reflection in action, right? This was an introduction to  the frame of queuing theory that you can use when doing performance analysis in the cloud  or otherwise, but problem solving, as I mentioned, is still part of the overall experiment or  the overall frame, and these experiments are not less rigorous than some of the controlled  experiments that happen in research. So to be able to do that, we need some on-the-spot  experiments, and in this case, I'm talking about the spot offering from AWS as well.  How many of you know about Spot? So Spot is an offering, very nice offering from AWS.  It has a few nice properties that you can save up to 90% compared to the on-demand price  that AWS has. However, to be able to save that kind of money, you need to improve your  chances of getting an instance in the first place, and you can do that by diversifying  your kind of instances across availability zones, their types and their sizes. Just as  an example, you might have tested your application on an M4 10x large instance with many number  of CPU cores, but the market might just only have M4 2x large instances during a specific  time, so what do you do during that time? So you need to be able to reconfigure your  application, consider all of these when you want to roll this thing out.  The other nice thing about AWS Spot is that you can use auto-scaling directly with Spot,  and it's now very seamless. However, the usual metrics that are the more common metrics of  CPU utilization latency throughput, they no longer work, because if you have an instance  which has about 40 cores and you have an instance which just has four cores, the average CPU  utilization is completely bad for such an heterogeneous environment.  And this time, what you can do is you can use the number of concurrent requests. As  I mentioned before, little slow or concurrency, this gives us the relationship between the  performance metrics, and that's why you already know what kind of behavior to expect if you  use N or the number of concurrent requests as the metric. The CPU latency and throughput  metrics are more reactive metrics. So this was a very good example of a rigorous  on-the-sword experiment which was informed by some guiding principles from Kewing Theory  and the enterprise benefited by saving cost, their operating cost of a very huge infrastructure.  Of course, we had to still test our model with the new data after we made these changes,  and it did conform to the model, so that was validated as well. And this is from the recent  data about March this year, so the new model that we came up with worked for all the data.  So one thing we can appreciate here is that, or you have to keep in mind, is that the application  was going through many architectural changes, release cycles, all sorts of changes are happening,  and that's what makes it important that a performance analyst has a frame in which he  can think or accommodate these changes, and that's where I think reflection in action  can be very indispensable to be able to navigate such an enterprise.  Let's take another example, or before this, any questions so far? Cool. Let's take another  example of disk utilization. Now, everybody says that storage is cheap, or everybody knows  that storage is cheap and it's fast, but managing disks is still far from being easy.  Let's take examples of two very popular software, Kafka and Graphite. These anecdotes are slightly  different from what I have been talking about so far, and we'll see how they are different.  The first point of difference which I can mention already is that this involved interaction  with AWS quite a lot, so this had feedback from AWS team on possible reasons for some  of the behavior that we were seeing. So in a large infrastructure, if you're using  a central metric system or hosting your own central metric system, it becomes very important  that you do proper capacity planning for that itself because of the number of metrics that  you might have to deal with today in a modern enterprise with microservices, with hybrid  environments, with the cloud, and perhaps on premise, or all sorts of variations on this.  It becomes important to be able to plan for the capacity. In this case, we are going to  talk about the disks because all of that time-seized data is lying on many disks. What was interesting  in this example was that, in this case, was that the performance issue was discovered  during the process of data collection itself. So before we could even think about modeling,  within we discovered certain things during the data collection itself. What I observed  was that the EBS volumes, just so you know, EBS is the storage offering from AWS, these  get attached to your instance. What I observed was that these EBS volumes were getting less  IOPS, which is a measure of throughput compared to what they were capable of. So the question  was, where is all the throughput going while we're not getting the promised performance?  There were many investigations carried out by AWS and myself. I shared dashboards that  I had for our infrastructure from their side. Also, they tried to share something that the  view that they had or that particular engineer had. What we found out was, after many of  these conversations, that the volume or the activity was actually being throttled. However,  this was not transparent to us at all or not easily because this required the engineer to  point out and he also had to investigate on the AWS side to figure out what was really  happening. And the reason for throttling was on the operating system level, we saw that  IOPS that was showing that the writes per second is very high. What we did then or what  was suggested by AWS was that we should configure the application to reduce that activity. So  I did that and it did stabilize the behavior of Graphite, but this seemed like a compromise  because we're still not getting the performance that the volume was capable of. So this pointed  out questions about the architecture of Graphite itself and how EBS works. And that takes me  to another point in reflection in action, which is, as I said, it's not always about  problem solving, but once you have pursued a line of thought in your frame, it should  open up a further line of inquiry. It is not supposed to end a particular investigation  or so. It should always open a further line of inquiry and that's what happened in this  case that we figured out, well, you know, we need to do this for Graphite. It's not ideal,  but at least Graphite provided that those knobs, maybe if this was some other application,  we might not have been that lucky. So this sort of signifies that we should ask the right  questions to your cloud vendor to get the right information.  The other example was about Kafka. Once again, now, I was considering queuing theory as the  guiding principle and what I set out to do was to collect metrics which would help predicting  the disk usage because one thing that modeling is done for is actually to predict how your  application would behave with changes in the workload that it handles. So that's very central  to the idea of modeling and based on that guiding principle, I thought of getting some  data for predicting disk usage of Kafka. However, when I did collect the data, it did not make  much sense. On the one hand, there were metrics from EBS volumes like queue length, throughput  and latency, which to me sounded so encouraging because I thought this is just going to work  out out of the box and we could predict disk usage quite easily. But comparing the metrics  from EBS side with the operating system level metrics and then to figure out one of the  very important metrics to characterize a queue is the service time. If we cannot find the  service time of an application, we cannot do queuing models at all. Now, in case of  Linux, in the main page for IOS tab, we see that for the service time field that warning  do not trust this field anymore. This will be removed in a future version. With that  kind of ambiguity about the metric and the lack of congruence between the operating system  metrics and the EBS metrics, it was next, I mean, for me at least, it was becoming quite  hard to be able to do this in a meaningful manner. I started talking with the AWS engineers  and this was a very different suggestion. Instead of trying to validate the modeling  techniques that I was trying to use, one of the suggestions that AWS team had was that  instead of doing that, use statistical methods like linear regression. So that, once again,  is sort of a new line of inquiry that triggered us on and from there we tried to predict the  disk usage of Kafka.  The next thing we're going to talk about is serverless applications and we're going  to focus on the concurrency part of serverless applications. This is, once again, a different  example from the previous ones, but the idea of concurrency is central to pretty much all  of them. So how many of you have actually used serverless architecture or Lambda functions  in AWS? Nice. In Lambda or serverless offering that AWS provides, what does concurrency even  mean? So the invocation of a Lambda function is a basic unit of concurrency that you think  about the concurrency in terms of the number of invocations of the function. Lambda requires  you to specify the memory and AWS figures out the appropriate CPU accordingly. In some  sense, this does sound to me like the Lambda function acts like a server in the queuing  theory sense, but that's just in some sense, I haven't pursued that line of thought much  further. So in Lambda, concurrency can be represented as events per second times the  function duration and this is exactly the form of Little's law. So Little's law is not  native to queuing theory. In fact, it's actually very generally applicable to a host of fields.  Very prominent field is the field of operations research where Little's law is quite well  known. However, to me, I jumped off my feet when I saw that this is exactly the form of  Little's law and this informed my ideas about how best to program a particular task that  I was doing which involved copying of a large number of files from one S3 bucket to another  bucket over the network. So during that time, I had to figure out how to measure the concurrency.  There are no out of the box ways to do this as far as when at least when I did this. So  you have to instrument your code to calculate the correct events rate. Events are basically  what trigger your Lambda function. So you have to figure out what's the rate of these  events. As there's a reference at the end of the presentation, you can use to calculate  that. By default, it's limited to about a thousand concurrent executions. After that,  throttling kicks in and throttling could also be in some sense considered to be a queuing  mechanism because in a way, based on the incoming events, your Lambda function is not being invoked  for all of them. So AWS throttles you after about a thousand concurrent executions by  default. So in such a scenario, how do you come up with an optimal concurrency?  The approach. What I ended up doing was to use a mix of threads and Lambda's in different  environments. So from our on-premise instance, I had a piece of code which was based on threads.  We were using Python at the time. And on the Lambda side, earlier, there was just threads  and Lambda. So the threads used to call Lambda functions and that was it. But then what I  found out was even within a Lambda, you can make the processing faster by having threads  inside the Lambda. So it was overall a mix of approaches. And there's a reference for  this as well if you want more details. The next one is slightly different. But this is  also one of the elements of the frame because monitoring, even though essential for doing  operations, we need a new idea of monitoring when it comes to the cloud. There was a media  theorist, American media theorist by the name of Marshall McLuhan. He came up with the phrase,  the medium is the message. Actually, when the book came out, that phrase was accidentally  changed to the medium is the massage. And we kind of see this in the operations teams  these days because once you see that some of your monitors don't work as expected, there's  a tendency to just let it be and repeat the same things that you have been doing, follow  a run book or something like that, remove some files from the temporary folder so that  there's more disk space, things like that. However, in the cloud, that doesn't really  work for long. What is required is to think about the medium with more focus. As a very  simple example of the kind of differences that the cloud has compared to other environments,  let's consider the use of notebooks versus dashboards. How many of you know the general  idea of notebooks? Well, maybe I can show you very quickly. If we use R as an example,  we have the option to create an R notebook. It allows you to have pieces of code with  certain documentation or other things that can insert the notebook. It becomes kind of  a, for the lack of a better word, narrative. You can think about what you're doing, run  it by other, sort of discuss around these ideas. And in this case, for example, we could,  so that's one, we could add another section to this and say just, let's see, so I'm not  sure if this is going to load the, okay. So let's say we are a team of, well, in this  case, status issues and I want to share a scatter plot of data with you. And I mean,  if you're at my desk and we start talking about this, after that, I just want to print  a string. Let's say, yeah, so it prints a string, you can go on like this. In a dashboard,  you would have, you would already have a plot, so to say, or many plots. And to make changes,  you will have to sort of take it offline in the sense when you're editing it, you can  no longer discuss the ideas that you're trying to do with the person who is at your desk.  Now, in case of monitoring in the cloud, this becomes very important because due to the  dynamic nature of the cloud, you cannot call meetings for all the things that might be  breaking all the time. You need, if a group of software engineers come to your desk and  you want to quickly discuss or brainstorm a few ideas on why their application might  be behaving in a certain way, notebooks are much better for that kind of thing. So that,  for me, kind of reflects the change of thinking that's required when considering monitoring  in the cloud. I have more examples and these are some things which I learned, some of these,  the hardware, and this is about we need better metrics to start with and also better programming  language support for coming up with good monitoring and to be able to calculate or find all the  performance metrics. The very first thing I noticed is that when it comes to manipulating  metrics or instrumenting your code, I was very surprised that Python did not have as  strong a library as Java has in terms of JMX, like Java has JMX. And it's so comprehensive  that I did not have to spend too long to use that interface in the Tomcat application I  showed you before, whereas in Python, I couldn't find a solid stable library which had similar  interfaces and that was, so that's why I had to manually instrument a bit of Python code  and luckily that worked, but I wasn't sure if I could do that again for other things.  The second example is of monitoring Tomcat, a very common way of thinking or way of approaching  Tomcat's monitoring or performances in terms of number of busy threads. In reality, that's  a lot less significant than the number of threads which are actually serving requests.  These are the threads which are in the service stage and if you remember the model that I  showed you before, they are the ones which are actually servicing requests and that number  is not the same as the number of busy threads and this becomes very important while doing  the modeling and this is not even easy to obtain. So there's no direct metric for this  that Tomcat exposes, you have to write your own script or a program to fetch this metric.  Lastly, as I mentioned that anything in reflection in action, any hypothesis or experiment that  you're doing, it should lead you to a further line of inquiry and in my case that was quite  literally to find out where are all the cues forming in the whole environment in the operating  system, in the cloud, components in between. One example I want to share is that we were,  for our data pipeline, we were seeing a lot of delays in the processing and transfer of  data and that was because we saw at the operating system level that the send queue values in  that stat were so high for a long period of time and that's why the transfers were not  even taking place for a particular period of time. This one required even more conversations  and discussions with the AWS team. I think this extended over months and by the end of  it, not only we had to change the application code but also the S3 SDK code with of course  a lot of rules of thumb and best practices and general ideas being shared across both  sides but this sort of tells you that if you take this into account in your application  design and application architecture, what I'm saying is performance analysis, if you  make that as part of a first class citizen of your application design, some of these  things can be considered from the very beginning. This is more of a quiz time when it comes  to data visualization but before that, are there any questions? Okay. So the quiz is like  this. How many data stores do you know? MySQL, Postgres, Kafka, so between three and five,  between one and three? Okay. Between three and five or one and three? Good. And keep  in mind how many of you actually use in your work as well so if you have production used,  that's more important. So that's just the data stores. How many web servers do you know  of? Apache, Nginx, many little web servers, application specific web servers, so I guess  still around three or five, between that, window three or five. How many programming  languages do you know? More than 10, 20? It's a huge number, right? And how many cloud vendors  do you know? Or IT infrastructure hosting providers do you know? What I'm trying to  get at is think about the one developer or the one system engineer who is supposed or  is very likely there is a system engineer's job. Think about the system engineer who has  to make sure that all of these systems are always up and running and performing well.  What I'm trying to get at is all of these, let's consider the data stores, for example,  the data stores, most of them actually come up with some kind of management interfaces  like they expose some monitoring data with their own dashboards. And similarly, some  of the web servers, they also expose their own metrics, might have dashboards of their  own. What I found out is at least that not one of, let me ask this as a question, how  many of those do you think follow strong visualization principles when it comes to conveying that  information which they think is important to run their application, but they're not  thinking from what I think, they're not thinking about that one guy, one system engineer who  is looking at all of that. Of course, it's a team of those guys, but they are still,  you guys know about the idea of on-call, right? We rotate and we have to keep the systems  always up and running. I believe that for me at least this is a much more important line,  it's an equally important line of inquiry for all of us that we need to have instead,  well, DevOps, SREs, that's all good. But I think the next phase for that should be visualization  informed operations or development. And right now, if I can't show the data stores, but  what I remember is two very important or very popular enterprise ready business high performance,  all of that, two data stores, one of them showed data just for the last 10 minutes,  which is fine, but the other data store shows it for a longer time, but doesn't show you  the metrics which make sense from performance perspective. So what I'm trying to say is  pretty much all of them want you to interpret their metrics in a terminate logic that they  have invented. And that's what makes it a lot harder for a performance analyst to be  able to correlate all that, even if, as I said, some of these bridges are burning faster  than they are being built. So even if you can, it's not always possible to make sense  of all the data out there. And yeah, I think that was the last note. Some references. So  a lot of the performance modeling stuff I learned from Dr. Neil Gunther, he runs his  own company. I can highly recommend his classes. Then the couple of links I shared before,  they are part of the slide deck which I'll upload on Frostcon portal. And so we can have  some Q&A right now. How many of you are actually running production workloads in the cloud?  Cool. All right. I guess then that's it. Thanks a lot for your time. And hopefully see you  next year.
2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_0_5051.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_5052_5507.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_5508_6035.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_6036_7691.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_7692_9011.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_9012_9923.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_9924_10379.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_10380_10679.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_10680_11915.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_11916_12347.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_12348_12599.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_12600_13523.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_13524_13583.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_13584_13775.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_13776_13883.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_13884_13955.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_13956_14051.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_14052_14639.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_14640_15671.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_15672_16931.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_16932_20687.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_20688_22199.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_22200_22703.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_22704_22847.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_22848_23135.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_23136_23711.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_23712_27599.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_27600_27851.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_27852_28175.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_28176_28235.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_28236_30359.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_30360_33395.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_33396_33647.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_33648_34655.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_34656_38063.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_38064_38639.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_38640_38939.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_38940_40247.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_40248_41303.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_41304_41483.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_41484_42983.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_42984_43691.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_43692_43919.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_43920_45875.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_45876_49199.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_49200_49403.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_49404_49487.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_49488_49775.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_49776_50123.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_50124_51203.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_51204_51455.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_51456_52367.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_52368_52859.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_52860_53123.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_53124_53279.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_53280_55679.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_55680_55943.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_55944_57947.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_57948_58151.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_58152_58283.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_58284_59135.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_59136_59639.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_59640_59843.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_59844_60851.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_60852_61247.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_61248_61727.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_61728_62303.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_62304_62759.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_62760_67223.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_67224_67379.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_67380_67655.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_67656_70211.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_70212_70715.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_70716_70799.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_70800_72635.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_72636_74711.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_74712_76271.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_76272_77303.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_77304_78047.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_78048_78095.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_78096_81395.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_81396_82271.png 2018_-_Performance_Analysis_in_the_Cloud_T8kiNPrgf4M_82272_83895.png