Tracking Performance of a Big Application from Dev to Ops
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Waroquiers, Philippe
This talk describes how performance aspects of a big Air Traffic Flow Management mission critical application are tracked from development to operations. Tracking performance is needed when new functionality is added, to balance the additional services versus the resource increase needed. Measuring and tracking performance is also critical to ensure a new release can cope with the current or expected load. We will discuss various aspects such as which tools and techniques are used for performance tracking and measurements, what are the traps and pitfalls encountered for these activities. The application in question is using Ada, but most of the items discussed are not particularly Ada related.
 Okay, our next speaker is Philippe Oroquieu from Belgium and he's working for your control where he manages about 2.5 million lines of Ida code. 2.3. 2.3 girls. Yes. Welcome to the 2.3. We will reduce some of your... Okay, so today I will speak about the Ida code. I will talk about tracking performance of a big application from development to operations. In fact, what I will discuss is not too much related to Ida. In fact, yesterday there was even zero slide in my presentation containing a line of Ida code. And so I was a little bit afraid that the dev room organizer would kill me after the presentation. So I added one slide. In fact, I was joking when I said that they would kill me after the presentation because Jean-Pierre, one of the dev room organizer, in fact, is working for me because I'm paying him to support the Ida control tool that we use to check the coding rules of our code and the second dev room organizer is working in my team so they have reasons not to kill me except maybe of course, Derk might have reason to kill me because he's working for me. So let's switch to the next slide. What are the objective of performance tracking? So the objective of performance tracking is to evaluate major resource needed by new functionalities. That's one objective. Another objective is to verify the estimated resource budget, CPU, memory of what you develop. Sorry. So we want also to ensure that a new release will cope with a current or expected new load and we want to avoid performance degradation during development. For example, imagine that we have a team of 20 developers working six months on a new release. That's about the size of the team working on this application and let's imagine that each developer integrate X changes per month and if one change on X degrades a performance by one person, then optimistically after six months we have a new release which is 2.2 times slower. We start from performance 100% six months, 20% one person. That's an optimistic view. The pessimistic view is that the new release is 3.3 times slower. We start from 100% times 1.01 to the power of 120. So clearly we have to do something and we cannot wait the end of the release to check the performance and see where the degradation is coming from. And so the objective is to do a daily track of the performance during the development. So we have a development performance tracking objective which is relatively precise because we want to reliably detect performance difference of one person or less. So a little bit of explanation about Euro control. So a little bit of explanation about where I'm working and what we are doing. So Euro control is the European organization for the safety of air navigation. So it's an international organization with 41 member states who would like to draw your attention on the fact that it is 41 and not 40 since last midnight. We have several sites, directorates, etc. The activities we have operations, handling management of the traffic in Europe. We do concept development. We do some European wide project implementation and support, etc. More info on our website. Where I'm working is the directorate of network management. It has nothing to do with network routers, cable and so on. It manages the network which is used by the flights to fly. So we have routes, we have points, we have air routes. This is a network and that's what we manage. We develop and operate the air traffic management network. So we have different operational phase. We have strategical. We prepare the network, the restructuration of the network and so on. Months or years in advance. Pre-tactical, a few days in advance. Tactical, today, tomorrow. Post-operation, we have to analyze what has happened and maybe improve for the future. So we are doing airspace route data, flight plan processing for complete Europe. Flow and capacity management. So if you have an airspace control center and you have too many flights that are planned to fly through this airspace control center, the controller might be overload, which is not good for your safety. And then the idea is that all systems will detect this load and then provide mechanism to balance the capacity and the load. So network management. We have two mission safety critical systems. So these are IFS flight plan processing and ETF mess flow and capacity management. And my team is developing these two applications, which are quite a lot of code in Ada. Ada is by far the main language use. We have something like the core of the system is 99% Ada. So some more words about IFS and ETF. So it's big application. The core is 2.3 million lines of code. And then around we have shells, scripts for the building, infrastructure for the monitoring. We have a lot of test code around, et cetera. So in terms of capacity, let's speak about a peak day of ETF MS, the flow management system. So in Europe, we have about in peak, more than 37,000 flights to Wendell. And we have around 8.6 million radar position. And these are planned to increase to 18 million in one year because we will add new sources of position, namely the ADSB data provided by new systems. So we have from external users, so the aircraft operator, Air France, British Airways and so on, the airports, the air traffic control center, they all query our systems in order to have data. And this results in about 3.3 million queries per day in peak. And we are publishing changes on the flight data through public subscribe mechanisms such as AMQP. And we have more than 3.5 million messages per day to publish. So what hardware do we run on? So the online processing is done on a Linux server 28 core. So it is not a small server, but it's by far not a huge, huge setup. We have some workstations that are running a graphical user interface for our internal flow controllers. And on these workstations, because they are quite powerful, we are also doing some batch processing and some background jobs. So we have many heavy queries. We have complex algorithms that are called a lot. For example, we have queries such as count or give me a flight list for all the flights that are traversing France between 10 o'clock and 20 o'clock. We have some algorithms like lateral route prediction or route proposal optimization. We have vertical trajectory calculation. I'll show after some drawing to show a little bit the complexity. And a lot of other things you can imagine in 2.3 million lines of code. We do a lot. So this is a graph of a flight departing somewhere in Turkey arriving somewhere outside of the European Union. And you see that when we have to calculate a trajectory. So for example, these orange stuff that you see here are radar position. And each time the system receives a radar position, it has to detect if it is fine or not from the planned trajectory. And if it deviates a lot, it has to recompute the trajectory. We also, so this is just computing one or several routes if the aircraft operator changes the flight plan for a flight. But what we also provide as a service is a route optimization service. So they can call our system and say, please find me a good route. And then we have to start to search maybe this way, maybe this way, maybe this way. Of course, basically, it's a shortest path algorithm day extra. But we can't use purely a day extra. It's a lot more complex than that because there are constraints which cannot be modeled in a simple graph. And so even searching, let's say the end shortest route is a lot more complex than typically than a typical similar problem. This is the same flight that we have seen, but on a vertical view. So we see departs here and we land there. And we see that we have plenty of constraints like forbidden airspace. We have some levels that have to be respected. We have blocks of airspace that can be traversed or not, depending on the direction and other condition. So you see that our algorithms that we have to make are quite complex. For this algorithm, for example, the algorithm that search where the flight can fly, we have per day several hundred of millions of calculations to do per day, which are done either on the central server or in batch or background jobs on workstations. So performance needs and ethnomestical ability. So as we have indicated, we have a lot of users. We have a lot of queries. And so we need horizontal scalability for this or operational configuration of our main server because we have a lot of instance of this software because we have, for example, an instance of the same system, which is doing the prediction for the next week. For example, we have standalone systems that are used to study what happens in several months. Here I'm speaking about the operational system. So on our 28 core system, we have 10 high priority server process that are handling the critical input. We have the flight plan, the radar position, the external user's queries. We have nine lower priority server process having each four threads, which are handling lower priority queries such as find me a better route for flight Air France 1 to 3. We have about up to 20 process running on workstation, which are executing this batch job or background queries. For example, every year search a better route for all flights of aircraft operator British Airways departing in the next three hours. And that, of course, is quite a high load. We also need vertical scalability. So this is a little bit different of systems where you can have a lot of users asking a lot of queries and you can distribute it. And that's what we do. But we also have some functionalities like simulation where one single user needs a lot of power because we have our internal flow controllers. We have sometimes to take heavy actions like if an airport calls on an Airspace Control Center calls and say, we have a problem, we have a technical problem, and we have to take maybe 1,000 flights. And we have to find how to best handle these 1,000 flights. All these actions will be done on the complete set of flights in the system. And so we must provide to our flow controller a functionality which goes fast for a lot of changes. So we need vertical scalability, so something very fast. For example, to evaluate heavy actions such as we close an Airspace, we close a country, and we have to spread the route or delay the whole traffic impacted. So to give an example, starting a simulation, it implies to clone the whole traffic from the server to the workstation. It's a very fat client. We need to recreate the in-memory indexes which are needed to execute all these algorithms and so on. So it's about 20 million in-memory indexes that we have to recreate. And in the release, we are busy developing. We have spent quite some time to optimize a start simulation. We are now starting a simulation in less than four seconds, including bringing the data from the server. And this is using multi-treadings. So we have one task that decodes the flight data received from a stream from the server. One task creates a flight data structure once it is decoded. And we have six tasks which are recreating the indexes. OK. So what we have seen now is that we have high performance requirement, and we can't degrade. And so we have to track performance during development. And one thing we can use for that is performance unit tests. So performance unit tests, these are useful to measure things such as basic data structure, hash table, binary trees, and so on. For example, here you see that we have a performance unit test which is checking the speed of an insert in a balanced binary tree. And then we can with this double check that, for example, the N log N behavior that you expect is effective respected by your implementation. So we can use the performance unit test to check the performance of low level primitives such as p-treads, multi-treads, a-da-protected objects, et cetera. So this is performance unit test which is verifying various things. So it checks the low level p-treads calls that we are available on Linux. We can compare with a protected object and higher level a-da concept for these kind of things. We have also some timing here in this unit, performance unit test system called clock get time clock monotonic, 40 nanoseconds for this one. Clock get time, but for the thread CPU time, about 400 nanoseconds. Anything to remember these figures because I will speak about it a little bit later and the difference between these two. So we can also use performance unit test to evaluate, measure, and be sure that we have the required performance for low level library, for example, maloc. In a-da, we don't use maloc directly. We use the a-da language allocation, the allocation, but at least with Gnat, it is based on calls to the underlying maloc library. So performance unit test, they have a lot of advantage. They are usually small. They are usually fast. And they are usually reproducible and precise. Remember one-person objective. If there is a degradation, we want to detect it and we want to detect with a one-person objective. No, we have some pitfalls with performance unit test. And I'll describe a real-life example with maloc. So maloc performance unit test, we have developed a performance unit test to compare the JLPS maloc with a TC maloc and JEMaloc. So seven years ago, we switched from the JLPS maloc to TC maloc because we had less fragmentation and it was faster with TC maloc. All fine, all good. But when we parallelize the start simulation, where we had, for example, to recreate this 20 million in memory indexes, we saw some not understandable 25-person variation. So sometimes start simulation was taking four seconds and sometimes it was taking five seconds. And this, we saw that the performance difference was varying depending on linking a little bit more or less code in the executable, but this code was not called. And so the size, minimal change to the size of the executable were causing difference. And so we said, let's analyze where this is coming from. And we started to analyze this val-green called green to really see in detail the instruction executed and we saw no difference. We used the Linux Perf tool to analyze then what was the behavior not under the val-green called green simulator, but the real stuff. And Perf shows effectively that the TC maloc slow pass was called a lot more when we had maybe 10 bytes more or 100 bytes less of executable code not called. We couldn't understand this mystery. We saw it was more often in the slow pass, but we couldn't determine why. So we said, that's easy. Now we will re-measure the maloc library. So we do a maloc performance unit test and tasks, simulating or indexer tasks, doing a million maloc and then a million free. What have we seen? We saw that JLIP say was slower, but had consistent performance with this unit test. JE maloc with this performance unit test was significantly faster than TC maloc. We were really happy. But when we use JE maloc with the real code, the real start simul with the complete system was slower with JE maloc. So what was the conclusion? More work needed on the unit test so that it better simulates what we have to do. So we continue to work on the unit test after improving the unit test to better reflect the start simulation work. What have we seen? TC maloc was slower with many threads, but became faster when the unit test was doing L loops of simulating start-top simul. So you start the unit test TC maloc once, you start the unit test start simul once with JE maloc. TC maloc slower, but when you say to TC maloc, start, stop, start, stop, start, stop, then it was faster. So our unit test was still not OK. With JE maloc, we have observed that then doing the M million free in the main task was slower. And that is, in fact, when we stop simul is a main task that is doing the free. The unit test, it does not evaluate fragmentation. And I have not put all the mysteries that we have seen with this unit test. But still, based on what we have measured, we obtain a very clear conclusion with this unit test and what to do about maloc. The very clear conclusion is that we cannot conclude from the maloc performance unit test. So currently, we have decided to keep TC maloc and we will reevaluate with a newer JLPSA in red at 8. We are currently on red at 7 because on red at 7, the JLPSA is quite old. OK. So pitfall of performance unit test. As we have seen, it's difficult to have a performance unit test which is representative of the reload. Maloc, we obtain no conclusion. The pitfall takes timing that we have seen. Well, it was a very simple measure. In fact, we have measured without contention. But what should we do? We should maybe also measure with contention and what type of contention and so on. And if we have to have a unit test representative of the reload, what kind of contention do you have in the reload? And if you measure and you see where you have contention and you simulate this in your performance unit test that might be for your current release, but if you change your code, the pattern of contention might change. And so performance unit tests are nice, small, fast, but it's difficult to have them representative. Even for hash table, binary trees and so on, the real behavior depends on the key types of the hash function, of the compare function, of the distribution of the key values, et cetera. So if it is already difficult to have performance unit tests for such low level algorithms, what about performance unit tests for more complex algorithms? For example, how to have a representative trajectory calculation performance unit test? You remember the picture at the beginning? How do you do a performance unit test for that? With which data? How many airports? Root air spaces? With what flights? A lot of short all, a lot of long all, flying where? You might have a lot of variation in the data. So conclusion on performance unit tests, these are useful, somewhat useful, but they are largely insufficient. And so for this, the solution is to complement, in fact, to do most of the performance tracking, not with performance unit tests, but to measure and track performance with a full system and the real data. So we want to replay one day of operational data. So replay operational data. The operational system, ETFMS, it records all its external input. So it records the messages that are modifying the state of the system, the flight plan, the radar position, et cetera. It records the query messages, you know, the flying in front end systems, don't really ETFMS recording it. For example, queries such as flight list entering France between 10 and 12, which might be asked by France, for example, by the France control center, this is recorded. And so we have a replay tool, ETFMS has a replay tool which can replay the input data. So of course, it means that the new release that we are preparing, we must be able to replay the somewhat recent old input format. So this brings a little bit of constraint on the development. And with this, we have some difficulties. We need several days of input to replay one day because you can have flights that are filed several days in advance. So if you have a flight today, you see the flight plan might have been filed two days in advance, and so we have to replay more days. Now, what is the elapsed time that we need to replay several days of operational data? This is a problem, of course. What is the hardware needed to replay the full operational data? Well, we have seen that we have, let's say, a medium-sized server and workstation. If a developer wants to evaluate the impact of a performance or we want to track daily, well, we have to ask them quite a lot of hardware. So that's a problem to replay the full operational data. Now also, we want, as you remember, our objective of one person or better, how to have a sufficiently deterministic replay in a multiprocess system, multi-trader system. This is quite a challenge, and we will describe it a little bit after. Remember one person. So volume of data to replay. So replaying the full operational input is too heavy, and so the compromise is to replay the full data that changes state of the system, flight plan, radar data, et cetera, but to replay only a subset of the query load. So we replay only one hour of the query load of the real system. And even in this, we replay a subset of the background and the bad jobs. We also have a problem that replaying in real time is too slow, as I have said. If we have to replay several days to have the result of one day, if we replay in real time, it means several days. If you want to do daily tracking of the performance, you have a lot of replays that will be in parallel a little bit everywhere. And so we have to try to reduce the time needed to replay. But we can't just take all the input and replay it instantaneously as fast as possible without doing something, because an input must be replayed at the time it was received on OPS. If you have a flight plan at 9 o'clock and a radar data at 9.30 and another one at 9.35, you can't replay the radar data together with the arrival of the flight plan. You have to wait till the radar data is a correct time to process it. So many actions also happen on timer events. And so what we need is an accelerated fast time replay mode. So what is it? The replay tool controls the clock value and the clock value jumps over the time periods with no input and no event. And so we are processing the data at the correct simulated clock. But when there is nothing to do, the clock jumps instead of doing nothing. So the fast time mode with all these limitations on the data and with this time control, we can no replaying one day and the data needed for one day in about 13 hours on a fast Linux workstation. We don't need anymore a big server and we can use a workstation which is not so huge. Still, we have to take attention to the source of non-deterministic results. So one source of non-deterministic result is a network, DNFS and so on. If the database files or whatever are on the network, we are not then the replay is not the only user of the network and that can introduce quite a lot of variation, much bigger than the one person we want to detect. So the solution for this is very easy. We replay on isolated workstations. They have their local file system, their local database and so on. Now another source of non-deterministic result is a system administrator. You might say, but how can a system administrator give non-deterministic result on a replay? Well, they are running jobs to audit to see what's happening, whatever and these jobs, if they suddenly intervene during a replay, it changes the performance. And so the solution is to discuss with a system administrator so that we can disable their job on the replay workstation and the solution was not too difficult. Another source of non-deterministic is the security officer. They absolutely want to be sure that there is no virus, no strange thing, no route executable, not allowed. And so they also want to run audit jobs and scan jobs and so on. And here also the solution is to discuss with them, but it was a little bit more difficult. We also see that non-deterministic results are obtained because of the input or put past the story. You start to replay from scratch, but because you use files, a file system and a database that was already used previously, even if you removed all the files, if you clear the database, you still do not obtain exactly the same and this was annoying us with a one-person objective. So removing a file clearing the database was not good enough and so before each replay, we completely recreate the file system and the database for each replay. Even with that, it was not okay because the operating system uses the story itself. If you do twice, two things, one after each other on Linux, the second time it might have been, I don't know, the memory in the kernel, whatever might have been changed. And so what we do is we recreate the file system, the database, and we reboot the workstation before each replay. With this, we still have some remaining source of non-deterministic results. For example, the time control tool, we serialize most of the input processing, most but not all. For example, because if we would serialize everything, it would slow down a lot the replay. For example, the radar position that they've arrived at the same second, we are not replaying them one by one. We have several processes and the several processes will process the radar data that arrived at the same second in parallel and that can introduce some non-determinism. The replays are done on identical workstations, same hardware, as I've said, file system recreate, database recreate, reboot. Still, despite same hardware, same operating system, restart from scratch and so on, we are observing difference between workstations. So all workstations are not born equal. That's our conclusion. Small difference in the clock of the CPU or whatever, but we see some impact. With all these caveats and the limitation and all what we did, we have finally achieved a reasonably deterministic replay performance with three levels of results. We have a global tracking where we track the elapsed user and system CPU for a replay in the complete system. We do a per-process tracking, user and system CPU and some per-stat recording. And we have a detailed tracking. We run one or of replays under Valgrind called Grind. This we run on the side because this takes quite a lot of time. It's very slow. It takes 26 hours, but it is very precise. So this is a drawing. These are all the baselines that we are building. So we do continuous integration. Every day, developers are integrating. We are building and replaying. And this is the release. This is a drawing showing the performance of the tracking of the global performance of the release we are developing, where we have done this optimization for the start CPU. In Grind, you see the total user CPU of all the processes that are replaying. In blue, it is a total system CPU for all the processes. And in red, it is the elapsed type. So what can you see is that we have seen a very gradual improvement of the performance during the development. And this gradual improvement, we have managed to track, to see that the optimizations that we are doing were effectively optimizing or we were able to see performance degradation on a global level using this. By the way, this, this, and this, and that is not that suddenly we got a quantum computer that did the replay in zero points on thing second. It's because we had a problem during the replay, of course. I'll discuss more in details later on this part of the graph. Remember a little bit the pattern. So that's a global tracking. We also have a per process tracking where we record the user system CPU, the EAP status, how much was used, free TC maloc details because we are using TC maloc and so on. So that's the kind of thing that we will record for each process so that we can see what's happening here. We see four, four process which are processing flight. The counter there is the one which is processing flight list and cons and so on. And for each of these process we are recording data that allows us to understand which process if we see a difference on the big global graph which process has increased. And then the third level is when we have to analyze what has happened inside the process. Then we have a one hour of replay under Valgrind call green and then we use the excellent K-cache green tool and the excellent call green tool to record the cold stack who has spent what we can see the functions which have consumed the top most and we can see the code which jumps we have, which condition was often true or false and we can go up to the assembly language level. So this is the main tool we are using when we want to optimize some specific algorithms or when we see degradation. By the way I am also the organizer of the debugging tool deaf room and tomorrow there is a talk about, at least one talk about Valgrind. So if you are interested that was the advertisement. So another interesting thing to discuss is that what we measure is we want to avoid performance degradation but we also want to see that if we believe we are doing an optimization is that really an optimization. And as a real life example of what we believe was a mist optimization, something that we could optimize, that we try to optimize and then it was becoming a failed optimization. So this is the slide with a little bit of data code I promised to Jean-Pierre and we would have some lines of data code here they are. So here what we see, we see a little bit of code of a task, a data task and it has two rendezvous, it accepts, so this task is maintaining the automatic loading of data like when the airports are changing and whatever, it's synchronizing the access and the load of this data. So while someone is accessing this data it must be locked and when it is not locked the task will load new data in the memory. And so this task among others has to maintain the number of locks and so it accepts, so it has a rendezvous which is called unlock where when a client task says I don't need a lock anymore, it's calling unlock and the task decrements the number of locks and there is also a rendezvous called getlock count which returns a current lock to the client. What is it used for? This is used for because when a process is activated to handle something like a flight plan it will take some locks but when it is finished to process the flight plan it's not supposed to have a lock anymore and so at the top level of the processing we are checking that there is no lock by calling this, this getlock count. Okay, a rendezvous, a rendezvous is a data task is something which is relatively costly because it is a task calling between quotes another task so there is a task which there is some system called to synchronize and so it's relatively expensive. And so the optimization idea was to decrease the number of rendezvous by using lower level synchronization based on volatile. That was the optimization idea. The idea was to take locks not as a task maintain variable but as a global volatile variable. We have a function that returns the value and inside the task we know accept unlock and then we are doing the decrement in the body. Why do we have to do it in the body? While above the unlock the decrement was done outside of the body because imagine that you have the task which is processing my flight plan. It has finished. It releases the last lock and just after says no I will check that there is zero lock. Here when with this solution we must absolutely be sure that when the task that says unlock gets the lock on that it must have been done and so we must do it in the body because otherwise possibly we could have a race condition in our check. So this should be faster because we will have the same number of unlock rendezvous but we will have much faster get lock calls accessing a volatile variable is much faster than a rendezvous and so this is supposed to be much much faster. That was the idea. In reality we detected with the performance tracking that this was a pessimization this thing. The compiler the compiler is quite efficient and helps us to build high level synchronization algorithms an efficient way. And so for example if you have a no body rendezvous then the compiler will optimize this and it will be a lot less costly to have a no body rendezvous than to have a body in the unlock calls and so because of this becoming more heavy in fact it was a pessimization and this was detected this is the extract of the big of the big drawing at the beginning. So this was without the optimization here we had the little bit of problem and here was with the optimization and you see that the system time slightly increased. So here we start to say what have we done during several release and after we have said we better roll back and so here we have rolled back the optimization so that we are we are back at the original system time performance. So what can we conclude about that is that you need to track performance you need to track performance because otherwise you can have problems you can try you have to track performance of your optimization because otherwise it might it might become a pessimization and a third thing to remember is that you can believe you are maybe smarter than the compiler but it's difficult difficult but not impossible because here you see we did the optimization correctly not using volatile but using atomic and doing a part of the decrement the decrement operation was done outside of the task and so it all went okay. So we detect here that we can optimize something we optimize but it is a pessimization we ask ourselves why are we so stupid we understand it and then after we become more intelligent so that's quite positive as an evolution. So performance tracking a summary we have good depth performance tracking using a mix of performance unit tests replay operational data as deterministic as possible the replay day we have to change it because the pattern of usage might change we might have new companies appearing airspace being restructured new route old route and so on so we change the reference day that we compare relatively frequently so that's too much new usage pattern we use various tools Valgrin call green kash green perth top and so on and but we have to take care about blind spot of your tools for example Valgrin call green kash green is very easy to use it's really a main tool we are using to optimize but it is very slow and it serializes multi-traded application and so it measures something for contention for example you never have any contention when you run something under Valgrin call green and it had a limited system call measurement it was not measuring the number of system call and the elapsed time in system call but not the system CPU and we saw here in our case a system CPU had to measure as I also happen to be a valgrin developer we I change I change a valgrin so that now it also measure the system CPU spending system calls so we need to have global indicators we have to zoom on the details where needed and as I have said improvement in the pipeline the next version of our green call green will measure system CPU and we are also working on developing a cold cream cold green diff to help visualizing the difference because currently comparing the kash green graphs is a little bit difficult so it looks wonderful we have a nice tracking system nice graph we can measure from the top from the global system to the details of the process and can we be happy with that is that good enough to go operational what about you are on call or me I'm on call because I'm on call from time to time and you are waking up Saturday 4 o'clock and because the users are complaining that the system is slow well I can't reply I will replay tomorrow Sunday the day and on Monday I'll explain you why we had big problems on Saturday this is not acceptable we have other questions like is the reference day that we replay is it representative of what happens on ops we need some indication for that what about evolution of the ops workload and capacity planning for example if we say we believe that our users will do a lot more queries to optimize their roots because we have improved because we have new users that will say we want to use our service will the system cope or do we have to change the hardware setup do we have to upgrade the hardware put more hardware for this we need something else than the replay for example what additional capacity hardware capacities need to support each time more queries of that specific type the solution for this is to have permanently activate response time monitoring and statistics so this I'm speaking about the tactical response time tactical because it's mostly useful during tactical operation but we of course also use this during the replay or the development the idea of this is that the application contains measurement code at critical points such as every remote procedure calling vocation begin and so one process is invoking something in another process it will measure how long it took at the at the process side that execute the remote procedure call we will also measure how much it took to process and send back the reply we are measuring the database access time begin and of the database and see if you can algorithm begin and such as calculate a vertical trajectory and so on and so on so the measurements are typically nested for example inside an RPC execution begin and we will order begin and for sub things that are being used by the thing so this tactical response time package maintains a circular a circular buffer with the last m measurement so the begin and measurement and for each begin and measurement it records the elapsed time the thread CPU time optionally the full process CPU time you remember the beginning I said clock get time monotonic and clock get time thread CPU this is measured with clock get time monotonic and this is measured with clock get time thread CPU this is relatively heavy costly this is in fact a virtual system called implemented for those I know in the VDSO of Linux and this is really switching to kernel to get the data so if they are kernel developers in the room if you could improve the the clock get time thread CPU that would be really nice so we the dispatch also maintain statistics how many measurements of what kind we're done an histogram of elapsed thread CPU and details about the n-worth case this is giving a reasonable overhead about one point seven percent of the CPU is spent in measuring what the application is doing rather than doing real work and for this we largely prefer to have it always activated on ops and on on our test and replay and so on because this is critical for us to understand what's happening in our system so this is an example of we have an online access to this data structure so this is a screen which allows to look interactively if you are called for example during the night what's happening and you see here a kind of tree a kind of tree of actions where we we have received a flight plan we had a flight deviation then we did several derivation phase over for the calculation of the trajectory we read the flight in the database we calculate another concept finally we have distributed some data and we have we have committed at the end the data in the database and so we can track the details of what's what's happening in the last m measurements this is the statistics that are maintained so we maintain how many measurement we had sorry the total time spent elapsed he'll try to CPU the average and a distribution an histogram of the distribution and the n worst case in case something is really abnormal it will appear in this in this data structure so this tax response time as I've indicated is used from development to operation on development it helps to understand how the system works see message exchange between process the algorithms executed the statistics are used to analyze the performance replay we can compare using this tax response time the profile of the replay day with the ops profile we can measure the resource consumption for new functionalities because we are recording things that are need to be recorded and some on ops online investigation of performance problem bug investigation in our system in our code the policies that exceptions are used for bugs not for normal behavior and so if we have an exception we take a core dump we can take a core dump without stopping the process we take a core dump we drop the input that has created the problem and we process the next message and the core dump contains the full status of what the thing was doing including in the tax response time the last n measurement of what the process did because possibly the bug might be created by something that was done a little bit before as always this we can see what what what the process was doing and recently did we also use this for post ops analysis and trade analysis and input for our capacity planning so performance tracking of a big application a summary so we have a reasonably deterministic per performance tracking during development we it allows us to detect performance regression on a daily basis we can verify that the optimization that we believe our optimization have the desired effect it allows us to plan capacity upgrade for demand growth and new functionality etc we are using a mix of various techniques and tools such as performance unit test replay real data application self measurement we have to take care to avoid blind spots by using various tools Perf val green cold green top s trace all we learn you something about your application the tooling we are also using for other purpose for example the replay tool is also a automatic testing tool because if we say we can replay 50,000 flight plans we can also say inject to fly plan and verify that this one is this characteristic and that is it has received this delay for example so the replay tool is also the test tool it is also used by your users to analyze optimize operational actions and procedure if they did something and they say maybe we should have done something else they can replay of course on offline systems and then they can decide to try other actions that what they did during operations so as I have indicated your operational system needs to have performance tracking and statistics is not only for development but also that finishes my presentation yes typically okay so the question is when we implement new features do we inject some data in the replay in order to see how it behaves yes the idea is that during the development when we implement a new feature we have we are of course developing unit test for this new feature which are done with the replay tool and whenever we have relevant data that we can use we have we might have to discuss with our users or we might have to create them ourselves we will measure how the code that we have developed behaves now it's very difficult to know what to do exactly because the real usage depends a lot of the pattern that the real users will do and sometimes users are a big source of non-determinism yes most of the load yes okay so the question is I've said that it runs on one server and so how is this highly reliable yes I said when it is working it's enough to have one server but of course we have several servers we have cluster of server and if one server has an hardware problem for example we move the system to another hardware server so operationally it's good enough to have well let's say medium-sized Linux server but we can move the application to another place yes yes no we are just taking so first another second measure so the question was how do we ensure precise measurement in the multiprocess so first the measurement that I showed where performance unit test as all this we just have a single typically a single executable which is the performance unit test in the real system we just say this process as consume this this threat of a this threat sorry this task as consume this CPU to do this action in that elapsed time no what we want to measure is what is really happening let's imagine for example that we have somewhere a kind of critical sections and two process are accessing some data and are looking this data you might have that one process has to wait a long time so that we will see because it says I see that I have spent let's say 0.1 second waiting and consume only one millisecond either there was not enough CPU on the system it was overloaded or there is wait time and so we just measure what the kernel gives back via the clock get time system calls other questions no thank you all speakers.
 Okay, our next speaker is Philippe Oroquieu from Belgium and he's working for your control  where he manages about 2.5 million lines of Ida code.  2.3.  2.3 girls.  Yes.  Welcome to the 2.3.  We will reduce some of your...  Okay, so today I will speak about the Ida code.  I will talk about tracking performance of a big application from development to operations.  In fact, what I will discuss is not too much related to Ida.  In fact, yesterday there was even zero slide in my presentation containing a line of Ida  code.  And so I was a little bit afraid that the dev room organizer would kill me after the presentation.  So I added one slide.  In fact, I was joking when I said that they would kill me after the presentation because  Jean-Pierre, one of the dev room organizer, in fact, is working for me because I'm paying  him to support the Ida control tool that we use to check the coding rules of our code  and the second dev room organizer is working in my team so they have reasons not to kill  me except maybe of course, Derk might have reason to kill me because he's working for  me.  So let's switch to the next slide.  What are the objective of performance tracking?  So the objective of performance tracking is to evaluate major resource needed by new functionalities.  That's one objective.  Another objective is to verify the estimated resource budget, CPU, memory of what you develop . Sorry.  So we want also to ensure that a new release will cope with a current or expected new load  and we want to avoid performance degradation during development.  For example, imagine that we have a team of 20 developers working six months on a new  release.  That's about the size of the team working on this application and let's imagine that  each developer integrate X changes per month and if one change on X degrades a performance  by one person, then optimistically after six months we have a new release which is 2.2  times slower.  We start from performance 100% six months, 20% one person.  That's an optimistic view.  The pessimistic view is that the new release is 3.3 times slower.  We start from 100% times 1.01 to the power of 120.  So clearly we have to do something and we cannot wait the end of the release to check  the performance and see where the degradation is coming from.  And so the objective is to do a daily track of the performance during the development.  So we have a development performance tracking objective which is relatively precise because  we want to reliably detect performance difference of one person or less.  So a little bit of explanation about Euro control.  So a little bit of explanation about where I'm working and what we are doing.  So Euro control is the European organization for the safety of air navigation.  So it's an international organization with 41 member states who would like to draw your  attention on the fact that it is 41 and not 40 since last midnight.  We have several sites, directorates, etc.  The activities we have operations, handling management of the traffic in Europe.  We do concept development.  We do some European wide project implementation and support, etc.  More info on our website.  Where I'm working is the directorate of network management.  It has nothing to do with network routers, cable and so on.  It manages the network which is used by the flights to fly.  So we have routes, we have points, we have air routes.  This is a network and that's what we manage.  We develop and operate the air traffic management network.  So we have different operational phase.  We have strategical.  We prepare the network, the restructuration of the network and so on.  Months or years in advance.  Pre-tactical, a few days in advance.  Tactical, today, tomorrow.  Post-operation, we have to analyze what has happened and maybe improve for the future.  So we are doing airspace route data, flight plan processing for complete Europe.  Flow and capacity management.  So if you have an airspace control center and you have too many flights that are planned  to fly through this airspace control center, the controller might be overload, which is  not good for your safety.  And then the idea is that all systems will detect this load and then provide mechanism  to balance the capacity and the load.  So network management.  We have two mission safety critical systems.  So these are IFS flight plan processing and ETF mess flow and capacity management.  And my team is developing these two applications, which are quite a lot of code in Ada.  Ada is by far the main language use.  We have something like the core of the system is 99% Ada.  So some more words about IFS and ETF.  So it's big application.  The core is 2.3 million lines of code.  And then around we have shells, scripts for the building, infrastructure for the monitoring.  We have a lot of test code around, et cetera.  So in terms of capacity, let's speak about a peak day of ETF MS, the flow management  system.  So in Europe, we have about in peak, more than 37,000 flights to Wendell.  And we have around 8.6 million radar position.  And these are planned to increase to 18 million in one year because we will add new sources  of position, namely the ADSB data provided by new systems.  So we have from external users, so the aircraft operator, Air France, British Airways and so  on, the airports, the air traffic control center, they all query our systems in order  to have data.  And this results in about 3.3 million queries per day in peak.  And we are publishing changes on the flight data through public subscribe mechanisms such  as AMQP.  And we have more than 3.5 million messages per day to publish.  So what hardware do we run on?  So the online processing is done on a Linux server 28 core.  So it is not a small server, but it's by far not a huge, huge setup.  We have some workstations that are running a graphical user interface for our internal  flow controllers.  And on these workstations, because they are quite powerful, we are also doing some batch  processing and some background jobs.  So we have many heavy queries.  We have complex algorithms that are called a lot.  For example, we have queries such as count or give me a flight list for all the flights  that are traversing France between 10 o'clock and 20 o'clock.  We have some algorithms like lateral route prediction or route proposal optimization.  We have vertical trajectory calculation.  I'll show after some drawing to show a little bit the complexity.  And a lot of other things you can imagine in 2.3 million lines of code.  We do a lot.  So this is a graph of a flight departing somewhere in Turkey arriving somewhere outside  of the European Union.  And you see that when we have to calculate a trajectory.  So for example, these orange stuff that you see here are radar position.  And each time the system receives a radar position, it has to detect if it is fine  or not from the planned trajectory.  And if it deviates a lot, it has to recompute the trajectory.  We also, so this is just computing one or several routes if the aircraft operator changes  the flight plan for a flight.  But what we also provide as a service is a route optimization service.  So they can call our system and say, please find me a good route.  And then we have to start to search maybe this way, maybe this way, maybe this way.  Of course, basically, it's a shortest path algorithm day extra.  But we can't use purely a day extra.  It's a lot more complex than that because there are constraints which cannot be modeled  in a simple graph.  And so even searching, let's say the end shortest route is a lot more complex than typically  than a typical similar problem.  This is the same flight that we have seen, but on a vertical view.  So we see departs here and we land there.  And we see that we have plenty of constraints like forbidden airspace.  We have some levels that have to be respected.  We have blocks of airspace that can be traversed or not, depending on the direction and other  condition.  So you see that our algorithms that we have to make are quite complex.  For this algorithm, for example, the algorithm that search where the flight can fly, we have  per day several hundred of millions of calculations to do per day, which are done either on the  central server or in batch or background jobs on workstations.  So performance needs and ethnomestical ability.  So as we have indicated, we have a lot of users.  We have a lot of queries.  And so we need horizontal scalability for this or operational configuration of our main  server because we have a lot of instance of this software because we have, for example,  an instance of the same system, which is doing the prediction for the next week.  For example, we have standalone systems that are used to study what happens in several  months.  Here I'm speaking about the operational system.  So on our 28 core system, we have 10 high priority server process that are handling the critical  input.  We have the flight plan, the radar position, the external user's queries.  We have nine lower priority server process having each four threads, which are handling  lower priority queries such as find me a better route for flight Air France 1 to 3.  We have about up to 20 process running on workstation, which are executing this batch  job or background queries.  For example, every year search a better route for all flights of aircraft operator British  Airways departing in the next three hours.  And that, of course, is quite a high load.  We also need vertical scalability.  So this is a little bit different of systems where you can have a lot of users asking a  lot of queries and you can distribute it.  And that's what we do.  But we also have some functionalities like simulation where one single user needs a lot  of power because we have our internal flow controllers.  We have sometimes to take heavy actions like if an airport calls on an Airspace Control  Center calls and say, we have a problem, we have a technical problem, and we have to  take maybe 1,000 flights.  And we have to find how to best handle these 1,000 flights.  All these actions will be done on the complete set of flights in the system.  And so we must provide to our flow controller a functionality which goes fast for a lot  of changes.  So we need vertical scalability, so something very fast.  For example, to evaluate heavy actions such as we close an Airspace, we close a country,  and we have to spread the route or delay the whole traffic impacted.  So to give an example, starting a simulation, it implies to clone the whole traffic from  the server to the workstation.  It's a very fat client.  We need to recreate the in-memory indexes which are needed to execute all these algorithms  and so on.  So it's about 20 million in-memory indexes that we have to recreate.  And in the release, we are busy developing.  We have spent quite some time to optimize a start simulation.  We are now starting a simulation in less than four seconds, including bringing the data from  the server.  And this is using multi-treadings.  So we have one task that decodes the flight data received from a stream from the server.  One task creates a flight data structure once it is decoded.  And we have six tasks which are recreating the indexes.  OK.  So what we have seen now is that we have high performance requirement, and we can't degrade.  And so we have to track performance during development.  And one thing we can use for that is performance unit tests.  So performance unit tests, these are useful to measure things such as basic data structure,  hash table, binary trees, and so on.  For example, here you see that we have a performance unit test which is checking the  speed of an insert in a balanced binary tree.  And then we can with this double check that, for example, the N log N behavior that you  expect is effective respected by your implementation.  So we can use the performance unit test to check the performance of low level primitives  such as p-treads, multi-treads, a-da-protected objects, et cetera.  So this is performance unit test which is verifying various things.  So it checks the low level p-treads calls that we are available on Linux.  We can compare with a protected object and higher level a-da concept for these kind of  things.  We have also some timing here in this unit, performance unit test system called clock  get time clock monotonic, 40 nanoseconds for this one.  Clock get time, but for the thread CPU time, about 400 nanoseconds.  Anything to remember these figures because I will speak about it a little bit later and  the difference between these two.  So we can also use performance unit test to evaluate, measure, and be sure that we have  the required performance for low level library, for example, maloc.  In a-da, we don't use maloc directly.  We use the a-da language allocation, the allocation, but at least with Gnat, it is based on calls  to the underlying maloc library.  So performance unit test, they have a lot of advantage.  They are usually small.  They are usually fast.  And they are usually reproducible and precise.  Remember one-person objective.  If there is a degradation, we want to detect it and we want to detect with a one-person  objective.  No, we have some pitfalls with performance unit test.  And I'll describe a real-life example with maloc.  So maloc performance unit test, we have developed a performance unit test to compare the JLPS  maloc with a TC maloc and JEMaloc.  So seven years ago, we switched from the JLPS maloc to TC maloc because we had less  fragmentation and it was faster with TC maloc.  All fine, all good.  But when we parallelize the start simulation, where we had, for example, to recreate this  20 million in memory indexes, we saw some not understandable 25-person variation.  So sometimes start simulation was taking four seconds and sometimes it was taking five  seconds.  And this, we saw that the performance difference was varying depending on linking a little bit  more or less code in the executable, but this code was not called.  And so the size, minimal change to the size of the executable were causing difference.  And so we said, let's analyze where this is coming from.  And we started to analyze this val-green called green to really see in detail the instruction  executed and we saw no difference.  We used the Linux Perf tool to analyze then what was the behavior not under the val-green  called green simulator, but the real stuff.  And Perf shows effectively that the TC maloc slow pass was called a lot more when we had  maybe 10 bytes more or 100 bytes less of executable code not called.  We couldn't understand this mystery.  We saw it was more often in the slow pass, but we couldn't determine why.  So we said, that's easy.  Now we will re-measure the maloc library.  So we do a maloc performance unit test and tasks, simulating or indexer tasks, doing  a million maloc and then a million free.  What have we seen?  We saw that JLIP say was slower, but had consistent performance with this unit test.  JE maloc with this performance unit test was significantly faster than TC maloc.  We were really happy.  But when we use JE maloc with the real code, the real start simul with the complete system  was slower with JE maloc.  So what was the conclusion?  More work needed on the unit test so that it better simulates what we have to do.  So we continue to work on the unit test after improving the unit test to better reflect the  start simulation work.  What have we seen?  TC maloc was slower with many threads, but became faster when the unit test was doing  L loops of simulating start-top simul.  So you start the unit test TC maloc once, you start the unit test start simul once with  JE maloc.  TC maloc slower, but when you say to TC maloc, start, stop, start, stop, start, stop, then  it was faster.  So our unit test was still not OK.  With JE maloc, we have observed that then doing the M million free in the main task  was slower.  And that is, in fact, when we stop simul is a main task that is doing the free.  The unit test, it does not evaluate fragmentation.  And I have not put all the mysteries that we have seen with this unit test.  But still, based on what we have measured, we obtain a very clear conclusion with this  unit test and what to do about maloc.  The very clear conclusion is that we cannot conclude from the maloc performance unit test.  So currently, we have decided to keep TC maloc and we will reevaluate with a newer JLPSA  in red at 8.  We are currently on red at 7 because on red at 7, the JLPSA is quite old.  OK.  So pitfall of performance unit test.  As we have seen, it's difficult to have a performance unit test which is representative  of the reload.  Maloc, we obtain no conclusion.  The pitfall takes timing that we have seen.  Well, it was a very simple measure.  In fact, we have measured without contention.  But what should we do?  We should maybe also measure with contention and what type of contention and so on.  And if we have to have a unit test representative of the reload, what kind of contention do  you have in the reload?  And if you measure and you see where you have contention and you simulate this in your  performance unit test that might be for your current release, but if you change your code,  the pattern of contention might change.  And so performance unit tests are nice, small, fast, but it's difficult to have them representative.  Even for hash table, binary trees and so on, the real behavior depends on the key types  of the hash function, of the compare function, of the distribution of the key values, et  cetera.  So if it is already difficult to have performance unit tests for such low level algorithms, what  about performance unit tests for more complex algorithms?  For example, how to have a representative trajectory calculation performance unit test?  You remember the picture at the beginning?  How do you do a performance unit test for that?  With which data?  How many airports?  Root air spaces?  With what flights?  A lot of short all, a lot of long all, flying where?  You might have a lot of variation in the data.  So conclusion on performance unit tests, these are useful, somewhat useful, but they are  largely insufficient.  And so for this, the solution is to complement, in fact, to do most of the performance tracking,  not with performance unit tests, but to measure and track performance with a full system and  the real data.  So we want to replay one day of operational data.  So replay operational data.  The operational system, ETFMS, it records all its external input.  So it records the messages that are modifying the state of the system, the flight plan,  the radar position, et cetera.  It records the query messages, you know, the flying in front end systems, don't really  ETFMS recording it.  For example, queries such as flight list entering France between 10 and 12, which might  be asked by France, for example, by the France control center, this is recorded.  And so we have a replay tool, ETFMS has a replay tool which can replay the input data.  So of course, it means that the new release that we are preparing, we must be able to  replay the somewhat recent old input format.  So this brings a little bit of constraint on the development.  And with this, we have some difficulties.  We need several days of input to replay one day because you can have flights that are  filed several days in advance.  So if you have a flight today, you see the flight plan might have been filed two days  in advance, and so we have to replay more days.  Now, what is the elapsed time that we need to replay several days of operational data?  This is a problem, of course.  What is the hardware needed to replay the full operational data?  Well, we have seen that we have, let's say, a medium-sized server and workstation.  If a developer wants to evaluate the impact of a performance or we want to track daily,  well, we have to ask them quite a lot of hardware.  So that's a problem to replay the full operational data.  Now also, we want, as you remember, our objective of one person or better, how to have a sufficiently  deterministic replay in a multiprocess system, multi-trader system.  This is quite a challenge, and we will describe it a little bit after.  Remember one person.  So volume of data to replay.  So replaying the full operational input is too heavy, and so the compromise is to replay  the full data that changes state of the system, flight plan, radar data, et cetera, but to  replay only a subset of the query load.  So we replay only one hour of the query load of the real system.  And even in this, we replay a subset of the background and the bad jobs.  We also have a problem that replaying in real time is too slow, as I have said.  If we have to replay several days to have the result of one day, if we replay in real  time, it means several days.  If you want to do daily tracking of the performance, you have a lot of replays that will be in  parallel a little bit everywhere.  And so we have to try to reduce the time needed to replay.  But we can't just take all the input and replay it instantaneously as fast as possible  without doing something, because an input must be replayed at the time it was received  on OPS.  If you have a flight plan at 9 o'clock and a radar data at 9.30 and another one at 9.35,  you can't replay the radar data together with the arrival of the flight plan.  You have to wait till the radar data is a correct time to process it.  So many actions also happen on timer events.  And so what we need is an accelerated fast time replay mode.  So what is it?  The replay tool controls the clock value and the clock value jumps over the time periods  with no input and no event.  And so we are processing the data at the correct simulated clock.  But when there is nothing to do, the clock jumps instead of doing nothing.  So the fast time mode with all these limitations on the data and with this time control, we  can no replaying one day and the data needed for one day in about 13 hours on a fast Linux  workstation.  We don't need anymore a big server and we can use a workstation which is not so huge.  Still, we have to take attention to the source of non-deterministic results.  So one source of non-deterministic result is a network, DNFS and so on.  If the database files or whatever are on the network, we are not then the replay is not  the only user of the network and that can introduce quite a lot of variation, much bigger  than the one person we want to detect.  So the solution for this is very easy.  We replay on isolated workstations.  They have their local file system, their local database and so on.  Now another source of non-deterministic result is a system administrator.  You might say, but how can a system administrator give non-deterministic result on a replay?  Well, they are running jobs to audit to see what's happening, whatever and these jobs,  if they suddenly intervene during a replay, it changes the performance.  And so the solution is to discuss with a system administrator so that we can disable their  job on the replay workstation and the solution was not too difficult.  Another source of non-deterministic is the security officer.  They absolutely want to be sure that there is no virus, no strange thing, no route executable,  not allowed.  And so they also want to run audit jobs and scan jobs and so on.  And here also the solution is to discuss with them, but it was a little bit more difficult.  We also see that non-deterministic results are obtained because of the input or put past  the story.  You start to replay from scratch, but because you use files, a file system and a database  that was already used previously, even if you removed all the files, if you clear the  database, you still do not obtain exactly the same and this was annoying us with a one-person  objective.  So removing a file clearing the database was not good enough and so before each replay,  we completely recreate the file system and the database for each replay.  Even with that, it was not okay because the operating system uses the story itself.  If you do twice, two things, one after each other on Linux, the second time it might have  been, I don't know, the memory in the kernel, whatever might have been changed.  And so what we do is we recreate the file system, the database, and we reboot the workstation  before each replay.  With this, we still have some remaining source of non-deterministic results.  For example, the time control tool, we serialize most of the input processing, most but not  all.  For example, because if we would serialize everything, it would slow down a lot the replay.  For example, the radar position that they've arrived at the same second, we are not replaying  them one by one.  We have several processes and the several processes will process the radar data that  arrived at the same second in parallel and that can introduce some non-determinism.  The replays are done on identical workstations, same hardware, as I've said, file system  recreate, database recreate, reboot.  Still, despite same hardware, same operating system, restart from scratch and so on, we are  observing difference between workstations.  So all workstations are not born equal.  That's our conclusion.  Small difference in the clock of the CPU or whatever, but we see some impact.  With all these caveats and the limitation and all what we did, we have finally achieved  a reasonably deterministic replay performance with three levels of results.  We have a global tracking where we track the elapsed user and system CPU for a replay  in the complete system.  We do a per-process tracking, user and system CPU and some per-stat recording.  And we have a detailed tracking.  We run one or of replays under Valgrind called Grind.  This we run on the side because this takes quite a lot of time.  It's very slow.  It takes 26 hours, but it is very precise.  So this is a drawing.  These are all the baselines that we are building.  So we do continuous integration.  Every day, developers are integrating.  We are building and replaying.  And this is the release.  This is a drawing showing the performance of the tracking of the global performance of  the release we are developing, where we have done this optimization for the start CPU.  In Grind, you see the total user CPU of all the processes that are replaying.  In blue, it is a total system CPU for all the processes.  And in red, it is the elapsed type.  So what can you see is that we have seen a very gradual improvement of the performance  during the development.  And this gradual improvement, we have managed to track, to see that the optimizations that  we are doing were effectively optimizing or we were able to see performance degradation  on a global level using this.  By the way, this, this, and this, and that is not that suddenly we got a quantum computer  that did the replay in zero points on thing second.  It's because we had a problem during the replay, of course.  I'll discuss more in details later on this part of the graph.  Remember a little bit the pattern.  So that's a global tracking.  We also have a per process tracking where we record the user system CPU, the EAP status,  how much was used, free TC maloc details because we are using TC maloc and so on.  So that's the kind of thing that we will record for each process so that we can see what's  happening here.  We see four, four process which are processing flight.  The counter there is the one which is processing flight list and cons and so on.  And for each of these process we are recording data that allows us to understand which process  if we see a difference on the big global graph which process has increased.  And then the third level is when we have to analyze what has happened inside the process.  Then we have a one hour of replay under Valgrind call green and then we use the excellent K-cache  green tool and the excellent call green tool to record the cold stack who has spent what  we can see the functions which have consumed the top most and we can see the code which  jumps we have, which condition was often true or false and we can go up to the assembly  language level.  So this is the main tool we are using when we want to optimize some specific algorithms  or when we see degradation.  By the way I am also the organizer of the debugging tool deaf room and tomorrow there  is a talk about, at least one talk about Valgrind.  So if you are interested that was the advertisement.  So another interesting thing to discuss is that what we measure is we want to avoid performance  degradation but we also want to see that if we believe we are doing an optimization is  that really an optimization.  And as a real life example of what we believe was a mist optimization, something that we  could optimize, that we try to optimize and then it was becoming a failed optimization.  So this is the slide with a little bit of data code I promised to Jean-Pierre and we  would have some lines of data code here they are.  So here what we see, we see a little bit of code of a task, a data task and it has two  rendezvous, it accepts, so this task is maintaining the automatic loading of data like when the  airports are changing and whatever, it's synchronizing the access and the load of this  data.  So while someone is accessing this data it must be locked and when it is not locked the  task will load new data in the memory.  And so this task among others has to maintain the number of locks and so it accepts, so  it has a rendezvous which is called unlock where when a client task says I don't need  a lock anymore, it's calling unlock and the task decrements the number of locks and there  is also a rendezvous called getlock count which returns a current lock to the client.  What is it used for?  This is used for because when a process is activated to handle something like a flight  plan it will take some locks but when it is finished to process the flight plan it's not  supposed to have a lock anymore and so at the top level of the processing we are checking  that there is no lock by calling this, this getlock count.  Okay, a rendezvous, a rendezvous is a data task is something which is relatively costly  because it is a task calling between quotes another task so there is a task which there  is some system called to synchronize and so it's relatively expensive.  And so the optimization idea was to decrease the number of rendezvous by using lower level  synchronization based on volatile.  That was the optimization idea.  The idea was to take locks not as a task maintain variable but as a global volatile variable.  We have a function that returns the value and inside the task we know accept unlock  and then we are doing the decrement in the body.  Why do we have to do it in the body?  While above the unlock the decrement was done outside of the body because imagine that you  have the task which is processing my flight plan.  It has finished.  It releases the last lock and just after says no I will check that there is zero lock.  Here when with this solution we must absolutely be sure that when the task that says unlock  gets the lock on that it must have been done and so we must do it in the body because otherwise  possibly we could have a race condition in our check.  So this should be faster because we will have the same number of unlock rendezvous but we  will have much faster get lock calls accessing a volatile variable is much faster than a  rendezvous and so this is supposed to be much much faster.  That was the idea.  In reality we detected with the performance tracking that this was a pessimization this  thing.  The compiler the compiler is quite efficient and helps us to build high level synchronization  algorithms an efficient way.  And so for example if you have a no body rendezvous then the compiler will optimize this and it  will be a lot less costly to have a no body rendezvous than to have a body in the unlock  calls and so because of this becoming more heavy in fact it was a pessimization and this  was detected this is the extract of the big of the big drawing at the beginning.  So this was without the optimization here we had the little bit of problem and here  was with the optimization and you see that the system time slightly increased.  So here we start to say what have we done during several release and after we have said  we better roll back and so here we have rolled back the optimization so that we are we are  back at the original system time performance.  So what can we conclude about that is that you need to track performance you need to  track performance because otherwise you can have problems you can try you have to track  performance of your optimization because otherwise it might it might become a pessimization  and a third thing to remember is that you can believe you are maybe smarter than the  compiler but it's difficult difficult but not impossible because here you see we did  the optimization correctly not using volatile but using atomic and doing a part of the decrement  the decrement operation was done outside of the task and so it all went okay.  So we detect here that we can optimize something we optimize but it is a pessimization we ask  ourselves why are we so stupid we understand it and then after we become more intelligent  so that's quite positive as an evolution.  So performance tracking a summary we have good depth performance tracking using a mix  of performance unit tests replay operational data as deterministic as possible the replay  day we have to change it because the pattern of usage might change we might have new companies  appearing airspace being restructured new route old route and so on so we change the  reference day that we compare relatively frequently so that's too much new usage pattern we use  various tools Valgrin call green kash green perth top and so on and but we have to take  care about blind spot of your tools for example Valgrin call green kash green is very easy  to use it's really a main tool we are using to optimize but it is very slow and it serializes  multi-traded application and so it measures something for contention for example you never  have any contention when you run something under Valgrin call green and it had a limited  system call measurement it was not measuring the number of system call and the elapsed  time in system call but not the system CPU and we saw here in our case a system CPU had  to measure as I also happen to be a valgrin developer we I change I change a valgrin so  that now it also measure the system CPU spending system calls so we need to have global indicators  we have to zoom on the details where needed and as I have said improvement in the pipeline  the next version of our green call green will measure system CPU and we are also working  on developing a cold cream cold green diff to help visualizing the difference because  currently comparing the kash green graphs is a little bit difficult so it looks wonderful  we have a nice tracking system nice graph we can measure from the top from the global  system to the details of the process and can we be happy with that is that good enough  to go operational what about you are on call or me I'm on call because I'm on call from  time to time and you are waking up Saturday 4 o'clock and because the users are complaining  that the system is slow well I can't reply I will replay tomorrow Sunday the day and  on Monday I'll explain you why we had big problems on Saturday this is not acceptable  we have other questions like is the reference day that we replay is it representative of  what happens on ops we need some indication for that what about evolution of the ops  workload and capacity planning for example if we say we believe that our users will do  a lot more queries to optimize their roots because we have improved because we have new  users that will say we want to use our service will the system cope or do we have to change  the hardware setup do we have to upgrade the hardware put more hardware for this we need  something else than the replay for example what additional capacity hardware capacities  need to support each time more queries of that specific type the solution for this is  to have permanently activate response time monitoring and statistics so this I'm speaking  about the tactical response time tactical because it's mostly useful during tactical  operation but we of course also use this during the replay or the development the idea of  this is that the application contains measurement code at critical points such as every remote  procedure calling vocation begin and so one process is invoking something in another process  it will measure how long it took at the at the process side that execute the remote procedure  call we will also measure how much it took to process and send back the reply we are  measuring the database access time begin and of the database and see if you can algorithm  begin and such as calculate a vertical trajectory and so on and so on so the measurements are  typically nested for example inside an RPC execution begin and we will order begin and  for sub things that are being used by the thing so this tactical response time package  maintains a circular a circular buffer with the last m measurement so the begin and measurement  and for each begin and measurement it records the elapsed time the thread CPU time optionally  the full process CPU time you remember the beginning I said clock get time monotonic  and clock get time thread CPU this is measured with clock get time monotonic and this is  measured with clock get time thread CPU this is relatively heavy costly this is in fact  a virtual system called implemented for those I know in the VDSO of Linux and this is really  switching to kernel to get the data so if they are kernel developers in the room if  you could improve the the clock get time thread CPU that would be really nice so we the dispatch  also maintain statistics how many measurements of what kind we're done an histogram of elapsed  thread CPU and details about the n-worth case this is giving a reasonable overhead about  one point seven percent of the CPU is spent in measuring what the application is doing  rather than doing real work and for this we largely prefer to have it always activated  on ops and on on our test and replay and so on because this is critical for us to understand  what's happening in our system so this is an example of we have an online access to  this data structure so this is a screen which allows to look interactively if you are called  for example during the night what's happening and you see here a kind of tree a kind of  tree of actions where we we have received a flight plan we had a flight deviation then  we did several derivation phase over for the calculation of the trajectory we read the  flight in the database we calculate another concept finally we have distributed some data  and we have we have committed at the end the data in the database and so we can track the  details of what's what's happening in the last m measurements this is the statistics  that are maintained so we maintain how many measurement we had sorry the total time spent  elapsed he'll try to CPU the average and a distribution an histogram of the distribution  and the n worst case in case something is really abnormal it will appear in this in  this data structure so this tax response time as I've indicated is used from development  to operation on development it helps to understand how the system works see message exchange  between process the algorithms executed the statistics are used to analyze the performance  replay we can compare using this tax response time the profile of the replay day with the  ops profile we can measure the resource consumption for new functionalities because we are recording  things that are need to be recorded and some on ops online investigation of performance  problem bug investigation in our system in our code the policies that exceptions are  used for bugs not for normal behavior and so if we have an exception we take a core  dump we can take a core dump without stopping the process we take a core dump we drop the  input that has created the problem and we process the next message and the core dump  contains the full status of what the thing was doing including in the tax response time  the last n measurement of what the process did because possibly the bug might be created  by something that was done a little bit before as always this we can see what what what the  process was doing and recently did we also use this for post ops analysis and trade analysis  and input for our capacity planning so performance tracking of a big application a summary so  we have a reasonably deterministic per performance tracking during development we it allows us  to detect performance regression on a daily basis we can verify that the optimization  that we believe our optimization have the desired effect it allows us to plan capacity  upgrade for demand growth and new functionality etc we are using a mix of various techniques  and tools such as performance unit test replay real data application self measurement we  have to take care to avoid blind spots by using various tools Perf val green cold green  top s trace all we learn you something about your application the tooling we are also using  for other purpose for example the replay tool is also a automatic testing tool because  if we say we can replay 50,000 flight plans we can also say inject to fly plan and verify  that this one is this characteristic and that is it has received this delay for example  so the replay tool is also the test tool it is also used by your users to analyze optimize  operational actions and procedure if they did something and they say maybe we should  have done something else they can replay of course on offline systems and then they can  decide to try other actions that what they did during operations so as I have indicated  your operational system needs to have performance tracking and statistics is not only for development  but also that finishes my presentation yes typically  okay so the question is when we implement new features do we inject some data in the  replay in order to see how it behaves yes the idea is that during the development when  we implement a new feature we have we are of course developing unit test for this new  feature which are done with the replay tool and whenever we have relevant data that we  can use we have we might have to discuss with our users or we might have to create them  ourselves we will measure how the code that we have developed behaves now it's very difficult  to know what to do exactly because the real usage depends a lot of the pattern that the  real users will do and sometimes users are a big source of non-determinism yes most of  the load yes okay so the question is I've said that it runs on one server and so how  is this highly reliable yes I said when it is working it's enough to have one server  but of course we have several servers we have cluster of server and if one server has an  hardware problem for example we move the system to another hardware server so operationally  it's good enough to have well let's say medium-sized Linux server but we can move the application  to another place yes yes no we are just taking so  first another second measure so the question was how do we ensure precise measurement in  the multiprocess so first the measurement that I showed where performance unit test  as all this we just have a single typically a single executable which is the performance  unit test in the real system we just say this process as consume this this threat of a this  threat sorry this task as consume this CPU to do this action in that elapsed time no what  we want to measure is what is really happening let's imagine for example that we have somewhere  a kind of critical sections and two process are accessing some data and are looking this  data you might have that one process has to wait a long time so that we will see because  it says I see that I have spent let's say 0.1 second waiting and consume only one millisecond  either there was not enough CPU on the system it was overloaded or there is wait time and  so we just measure what the kernel gives back via the clock get time system calls other  questions no thank you all speakers.
ada_performance_0_2303.png ada_performance_2304_2759.png ada_performance_2760_3107.png ada_performance_3108_5087.png ada_performance_5088_5915.png ada_performance_5916_9947.png ada_performance_9948_16823.png ada_performance_16824_20495.png ada_performance_20496_21083.png ada_performance_21084_24059.png ada_performance_24060_24467.png ada_performance_24468_27947.png ada_performance_27948_28607.png ada_performance_28608_28883.png ada_performance_28884_30419.png ada_performance_30420_33095.png ada_performance_33096_42575.png ada_performance_42576_45095.png ada_performance_45096_47111.png ada_performance_47112_57875.png ada_performance_57876_58007.png ada_performance_58008_58067.png ada_performance_58068_60887.png ada_performance_60888_61163.png ada_performance_61164_68471.png ada_performance_68472_72635.png ada_performance_72636_72899.png ada_performance_72900_73067.png ada_performance_73068_73127.png ada_performance_73128_73187.png ada_performance_73188_73487.png ada_performance_73488_73559.png ada_performance_73560_73607.png ada_performance_73608_73691.png ada_performance_73692_73919.png ada_performance_73920_74099.png ada_performance_74100_74279.png ada_performance_74280_74447.png ada_performance_74448_74567.png ada_performance_74568_74651.png ada_performance_74652_74699.png ada_performance_74700_74747.png ada_performance_74748_74975.png ada_performance_74976_75119.png ada_performance_75120_75167.png ada_performance_75168_75275.png ada_performance_75276_75467.png ada_performance_75468_75635.png ada_performance_75636_75707.png ada_performance_75708_75791.png ada_performance_75792_75839.png ada_performance_75840_75887.png ada_performance_75888_75959.png ada_performance_75960_76031.png ada_performance_76032_76103.png ada_performance_76104_76163.png ada_performance_76164_76343.png ada_performance_76344_76703.png ada_performance_76704_76799.png ada_performance_76800_76859.png ada_performance_76860_76979.png ada_performance_76980_77027.png ada_performance_77028_77075.png ada_performance_77076_77123.png ada_performance_77124_77375.png ada_performance_77376_77891.png ada_performance_77892_77987.png ada_performance_77988_78059.png ada_performance_78060_78119.png ada_performance_78120_78191.png ada_performance_78192_78395.png ada_performance_78396_78514.png