Measuring Performance on OpenBSD
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Bluhm, Alexander
When you try a new software version, something may be wrong or slow. You want to figure out when the regression was introduced. After updating OpenBSD you might see that something does not work as it used to be. To simplify debugging it is helpful to determine the point in time when the change was introduced and search for the relevant commit. For functional regressions a test suite that is executed on a daily basis is sufficient. By providing recent test results, the relevant day can be seen and the responsible developer of that area is informed. Making statements about performance is more difficult. The requirements for measurements may change, new test programs are needed, test hardware is only available for a limited time. So it is not sufficient to store historic data on a daily basis. You want to change granularity or look back into the past. For that purpose I have created a system that can create a performance measurement of an OpenBSD kernel in the past. The kernel is compiled from a certain CVS checkout. That may sound easier than it is as the OpenBSD kernel is not self contained. It belongs to a base system, there may be incompatibilities with userland. The performance also depends on the compiler version that changes over an OpenBSD development cycle. My solution installs an OpenBSD release and updates the kernel from source in fixed time steps. If there is a relevant user land change within the step, the necessary parts of the system are updated. This allows quick progress while compiling the kernel without rebuilding the whole system. With each new kernel the performance is measured.
 So, my name is Alexander Blum. I'm an open VSD developer for more than 10 years now. I work at the company in Munich. We build firewall and VPN appliances based on open VSD. So, it's also my job to make open VSD reliable and fast and suitable for our products. So, what I want to do is detect changes in performance in open VSD when they occur. So, when everything is something is committed and it affects performance, I want to see okay, there was a difference and also see how does it develop or release as we have the regressions here. Instead, has there something that is good and I want to measure real-world performance. So, it's basically network performance that I'm interested. I started with network stack performance, not the forwarding performance. It's a different story. And so, I want to see how does our TCP stack behave. So, what I will do is, first of all, I will talk about where we're coming from, what did we have before. I will also talk about my regression tests, then how does this performance testing over the time work, what results did I find and what's the conclusion where I want to give you on your way. So, it starts with the same that was before I started the testing performance. So, in Innova, that's a company of web form, we have a high performance firewall test that and we use it for multiple purposes. You want to generate numbers for that we control our customers that are basically high and the marketing also was once high numbers. The developers want to know how the system works and then we want to see when we buy it, when we choose new hardware revisions, does it go faster because it's not faster, what are the pensions we release, do we have regressions there, we want to have forwarding performance again, performance and everything. So, everything results in bad results for that what I want. I want results for developers. And what you see here is you can't see really what we are measuring, you have some dots jumping around, you don't know why, you have some lines in there and it's also unclear. By the way, we are testing the 6.0 release, always the same software, you have varying results, it goes up and down and nobody knows why. So the hardware setup is also rather complex because it's for multiple people doing testing the same, doing tests at the same time. We have different requirements as I just told you before and we want to test all the different hardware so we have several machines where we can install the software on and it works basically like this. We have some Linux lines here and some Linux servers and we have some old machines there, we bought them 10 years ago and so they had only one gigabit interface and we take several of the machines with multiple interfaces, put them with a big switch and go with 10 gigabit through those targets and measure how much forward or with the end performance we get there. If you do that, it depends on how many other people are working simultaneously, then the switch drops some packets or not and you have 20% variance in throughput depending on the phase of the moon or whatever. What I want, the problems are, we have too many requirements as I told you at the beginning, we have too much complexity which is bad if you want to reliable numbers and if you want to dig down what's really the problem there, you have not enough flexibility because it's all set up in a static way and other people are using it in parallel. If you start moving the cables around and reconfiguring the switches, other set of break or you don't have enough permissions to configure the switch or people complain when you remove some cables. What I want to have as a setup is this. I have an open BSD machine here, I have another open BSD machine here and I have only a cable in between, no switch, no nothing. As much less things that influence my results and everything, I measure is somehow based on the open BSD I want to get information about. The other thing I had before is my regression tests. I worked on that, I started that I think four years ago and I run the open BSD regression tests on a daily basis. Those regression tests existed before, they were just checked into the source tree and every developer could do a make in some sub-directory and good pass or not. But there was no regular execution of them and a lot of them were un-maintained or just failed. So what I started, I just took the test that already existed here and run them daily. I can run them on multiple architectures to have a comparison, how does it work on 32 bit or 64 bit. And I'm planning to extend it on Spark and we have little engine, big engine. And what the nice thing is when you do it daily and record it and publish it on a web page, then you get a history. You can say, okay, three weeks ago this test started failing, so what was going on there? I can link that to the commit or to the source change that caused the problem. And that's something I also want to do for a regular performance. And what I also do is collecting useful information. I create log files, you can download the object files from the test run, you can debug it more or less online and if it's really complicated you can also log in at the machine and do your analysis there. And everything is accessible, you just go to this website and you can see the results of the things. So here's how it looks like. I didn't do it. You didn't do what? Break. Ah, no, no. But we didn't fix it. So what we have here is a display of everything and it's quite red because we have new architectures here that's ARM V7 and ARM 64 if it doesn't crash during the test run and I haven't spent too much time on those ARM platforms so quite a lot of them is red. But we are working on that. So what I show here or what is told before, I want to have links and information on what's going on. So I put a lot of links here. For example, the run history, you can click on that and you see when each test was executed, how the machine was set up. So what I do there, I get a snapshot from OpenBSD every time and every day or so, the CDRAR builds a snapshot with the current release, no, not release, with the current software version and just install it on the machine. There's an auto installer for OpenBSD and I can just set it up. The config is setting and here you can see how it installed and if they broke anything or if the auto installer is broken or snapshot has some problem that they cannot install. So then I have a link here to the CBS web of the test. So here are the tests and you can just go in there and see, oh, did somebody commit something in the test so that's the reason why it broke. Here I have some color scheme to show what is wrong with the test. The path is always green. The red has failed. You have skipped and that means there's something missing in the hardware setup so it cannot run there. So we have architecture specific tests, for example here is a GCC test and on R64 we don't have GCC so we skip it. And there are some other reasons why I skip it. We also have some tests that use that to something with the keyboard driver and if there's no mouse attached then it's skipped. And if you click on such a fail or in the past you get a link to the log to the test output. So you can see or get an idea why the test fails. So here I have a link to the logs of the setup of this specific date. So this is the date when I run it. See this one is the most recent one. So I made the slides last month. And here every day you get four new architectures. And when you click on that you get the logs how it was installed and you can also download the object files that were generated while the test was run and perhaps you see it gets some more information from there. And I also have a cron job to delete that here. So I cut it a little bit and here the test results are deleted so that the drive doesn't fill up. And here you can also click on it. You get a D-Mess from the machine and see what version was actually tested, who built the terminal and what's the test target. So what we get from it now I clicked on the only show me the IP86 tests. It's IP86 because that's the first architecture I started with. And so we have the history here when we did run the test. So we see, okay, this here has some problems forever. And here this libcrop door something here between 13th and 15th went wrong. So here I have to start and investigate. So those GCC tests failed since we switched to the playing linker because it misses some features and then so it basically checks whether some warning of some unsafe functions are emitted by the linker. And that doesn't work currently. But Kden is promised to fix it. And here we have other tests that pass sometimes and fail sometimes that basically means it's not a stable test. There should be some more investigation, make it more stable. So that's a fragment test here. Send some IP fragments with escapee to another machine and maybe some packets get lost and when they get lost the test fails and it's a little bit fragile. So the other thing I do is I sort them here. So if something, so this test I fixed it four days ago. But if this test here for example it doesn't terminate. So it just takes too long and it takes more than an hour. After an hour every test is supported. And so it's no-term category. So we have some failures. They are sorted here. We have some paths here that they are, they fail in some other architecture. That's why they are sorted here and here we have some skip tests. And we have a lot of really passing tests down here but I cut it away from the slides because it didn't fit. And the interesting thing for the developers is that you should concentrate on that and fix that. So I run all the tests on hardware. So most of them are just running one machine. So we have some kernel installed here, some user-landed installed here, some tests here in user-source. Just type make and then something happens and it either passes or fails. But there are some other tests that are more complicated. For example, I have a test that tests ARC, the ARC protocol. And they have a remote machine and I send some SKP packets to it. And I send some fake ARC packets and look at the ARC replies and check whether that fits. So you can do a test with two machines. This one controls everything. The tests create this run, the make file run. And it may log in with SSH to here to make some setup and it may send some packets here that do the actual test. There are tests that have more machines involved. The most complex one are testing PF, forwarding, redirection and fragmentation. So I run the PF. The PF that it says that it's basically run here. So I can send a packet here. Then it goes through PF, gets modified or reassembled whatever tests I'm doing here. Then it goes to a router. I need the router here because I want to do pathMTU tests. So this router has a limit pathMTU at the interface. It's only 1,300. So the packet has to be fragmented somewhere here with its IPv6. So here, if it's IPv4, then it hits to the server. The server is, I am just running the normal ping service of the current. So if it's been packet, it's just returned. And it also runs the echo service on part 7. If it's a RubyP packet, it's just returned. And there's also TCP echo service. You open the connection. Everything you write in gets back out. So I can just write a TCP stream through this. And here it has to do some pathMTU. Here the PF has to do some magic things. And then I get everything. So back, if it works. If it doesn't get the data back, the test fails. So how do we set up that? I told a little bit about it. So I have this master here. That's also the web server. You can publish the results. And this master controls the serial console server that also handles all the installations. So here I can say, OK, install or upgrade all the machines. And it will install the latest snapshot on these machines with the outer installer. And the OpenBSD installer allows to put a site, TGZ, somewhere that's where all the configuration files are in there. They are stored here. And here it gets all the configuration files and gets the IP addresses. And it's configured statically so that it's set up to run those tests. OK, now we switch from regression testing to performance testing. What did you do? Fix the test? So what do we want? First of all, we want to see the history. When did things get slower or faster? You want to have reproducible results. So if you run the test once, get a number. If I run it again, I want to have the same number. That's pretty hard. I want to have the details so I can see, OK, in which situation was the machine, was everything set up correctly, also see which commits were in that region, and what date of the curve was run there, what user line was running there. I want to have a drill down. So if I see, OK, one month ago something broke or got slower. Then I want to drill down and say, OK, which commit was it? Which hour was it? What did it really cause that? And I want to have it somehow automatic so I don't want to run everything manually. The performance tests are run by a cron job and it's just in the background, just needs a little bit of babysitting if something breaks. Hopefully it doesn't break, doesn't add nothing breaks. And so I also want to see automatic and basically I do the test runs manually now but I have set up a cron job that updates the last week of commits every week. So how does it work? So in the years, I installed an OpenBSD release. And I have a defined point where I can start from. I have a kernel installed and the user line installed and it's publicly released and everyone knows what it is. And then I check out the kernel at a certain date. So I want to test this kernel work. Then I compile it. I compile it on both machines. You remember, I have two machines on the table in between it and the same thing for on both machines. So I compile it. Then I run the tests and then I can have a data how much I want to step forward. So maybe a day or month or week or an hour. Usually when I want a day, a day based stepping is a good resolution. So I step one day ahead and do the next check out. I compile the kernel, run the tests and then do that until I'm at the end date that might be the next release. And then I connect the results to make some statistics about it and present them. And you're not rebuild the user line? Oh, that's on the next slide. So basically I want to avoid to rebuild a user line because it takes forever. So compiling the kernel is two minutes, compiling the user line is two hours. So what does the result look like? Basically I copied the protocol that generates those results from the regression test. So it looks quite similar. So what we have here is the run history. So you can see when was it installed and how was the kernel compiled and everything. We have here some details in this line. I'll just play it later. Then here we have the tests I run. Basically it's an IPer 3 from port system I installed there. I run some TCP tests. It also has a minus R option that normally when you have an IPer server, the client makes the server and all data is sent from client to server. And with minus R, it's the other direction. And we run some TCP bench. It's TCP, TCP bench. Here it's with 100 simultaneous connections. Here's only one connections. We run IPer with Udiki and reverse. And I have two test setups. So we have one test setup with one kind of machine and there's another IP address. The IP address is coded here somewhere. So the one machine is here and the other machines, the test results are down there. They're basically the same test. And here I compiled the kernel and measured the time how long it takes to convert the kernel. The main idea that I got is that I got... So what we did before, before we did some performance testing, people compile their machines. So developers compile the kernel, compile their user land and they always say, oh, it's getting slower. And to measure that it's getting slower, I also just do a kernel compile, make some A clean and then run J4 on the 4 CPU machine and J8 on the 8 CPU machine. And I also do some disk drive, filesystem performance, but I'm not satisfied with that. I had to find you in the... So now when you click... Oh no, I forgot to explain this. So here we have the date when I run the test. Here it's the version I installed. So in this case it's 64 release. I installed it. And then I have the first CVS checkout that's here. So it started here at 16th of March and I stopped at 23rd of March and it's 8 test runs in between but one day's there. And I repeated the test five times. I'll give more detail about that later. So if you click on one, it's such test runs. So if you click here, you get to the next slide. That shows what happened in one of those test runs. So here you still... If you again have those logs, you can look at more details. And here's the checkout date. So here this test I did run in February and I checked out October 2017. And here I checked out... It's 4th of October, that's 11th of October, 18th of October, so I have a week's step. I stepped weekly through the 6th to release, I think. The idea is the 6th to it was installed and I was there weekly through it. So what I do now, I have the checkout date and between this date and this date, there were 49 kernel commits. And if you click on the CVS log, you also see all the commits. I go through the CVS history and collect everything and show what was between here and here. Later there will be this library. Show it. And then we have those build works. So as Bob asked already what happens with userland, I have to recompile the userland from time to time. The thing is that netBSD can be compiled, the kernel can be compiled on any POSIX system. That's not the case with OpenBSD. The OpenBSD kernel requires that it's compiled and built on a system, on a userland system that matches more or less its date. And so I have to update the userland so that it's compatible to the kernel. So basically what I have to do is recompile the kernel, recompile the compiler, the client compiler, when it's updated. And that's also good performance wise because I'm not only interested in the kernel performance, but I also want to know what do the compiler and its options have an influence on the performance. So I also measure the important steps in userland. So every time I have such a build work, then I know, okay, here I have to do something with a userland. I'll explain that later in more detail. So and now here you have the numbers of this test. That's the I-curve TCP in forward direction. And I repeat those tests multiple times to get more stable numbers. My observation was that the test goes up and down and gets faster and slower. And so I try to do it here five times and here present average numbers. It's not quite correct. For network performance throughput, I take the maximum number. So that's the maximum number. And when those repetitions if results that are not stable that are jumping a lot, then I print them in the right here. So this one has a lot of jumping in it. You see it's very right. So here we should look into more detail. So I get to the next slide. So if you click on the top, click here. And you go to the details of all the repetition. So you see here it looks like here we have some steps of log files. And here it's repeat one, zero, one, two, three, four. And here we have some strange values. All of them are right. You can't read the numbers, but here it's four gigabits throughput. And here it's only 1.6. So what happens? This is an 8-core machines with two sockets. And these two socket machines are very unreliable performance wise. And it happens that once a while the throughput goes down to 1.6 gigabit and I don't know why. So if you reboot it, it goes up again. So here we have four, so that's 3, 9, 4, 3, 8, 1.6, 3, 8. And if you reboot it, it gets into this state. And if you reboot it again, it gets out. I don't know why it's so. So here in the back I calculate the standard deviation. And I also print it in red when it's too big. So I see easily where the problem is. And report that to the next slide. So it can also be generated some graphs out of that. So here we install 6.2 here. And we run until 6.3. And we make one measurement every week. So as you told you, I have several tests that have shown us different symbols here. So there is an explanation on the website, but it doesn't matter which test it is. And what you see here, we have this. That's the occasional 1.6 throughput. Nobody knows why. And most of them are in this band here. You also see that the numbers are jumping up and down. So it's not one line. Here we have one line. But here on the top it's more or less. The different tests also give different results. And here we have one test that's faster. I'll explain later what it is. But just remember, we saw it here. What I also do is those vertical lines that are the quirks. When I have to touch user lens to make the kernel compile again or to change the compiler or whatever, then I have such a curve. I have a static file with the build quirks where I have a date. It says, OK, at this date, we have to update the compiler. And if the previous test is before the date and the next test is after the date, in the meantime, the compiler is recompiled. So here's a small example of the quirks that happened in the previous slide. So to make it easily detectable, I also have a line for the release date. I don't build anything there. I just want to show you where the release was. So first thing I had to do, I wanted to check out the compiler. And I saw that there's a bug in CBS. And you cannot check out a vendor branch if there was a commit on a non-vendor branch later. Bug in CBS. So I fixed CBS for two and added a patch to the quirks and recompiled CBS. Then I can check out LLBM that was released 5.0 at that point and recompile the kernel with that. What we also have is there's a header file that's called PNBar where the PF structure is in there. So when we change the PF structure in the kernel, the PF CTL tutor to change the kernel has to be compatible. Otherwise, kernel and tool don't have the same idea to construct. And the update of the PF rule fails. And usually you see that the performance difference. Because I run it with PF because the default rule. And if you remove the PF rules, it can be loaded. Then you have a different PF set. And then we see other performance. And so I figure out, when I look at the graph, oh, there's a problem. Then I add a perk to say, OK, here, when we have the PNBar header file change, then I have to recompile the PF CTL. That's rather short. It's just a few seconds to recompile it. So the next problem we had, somebody committed here a driver and he made a mistake. So the tree was broken. It was only broken for half a day or so. But if my metamatter just hits that moment, it can't compile the kernel. So I have to look at the fix that's committed later and patch that and the sources that were before and then it compiles again. So what did here? Here was another PF CTL set change. So the CTL has the same problem. If the system CTL strikes change, then it doesn't work anymore. That may have some influence how the system was booted, what's this here that's perceived to be removed. So if that changes, I just recompile it. Here we updated LLVM again. Here we had some PF change. And here we have the LLVM Vs series. So that's how the perks work. That's the segify where everything, it's a per module where I've described all the things that change there. Bob, you look like you have a question. Why did you choose to just do the kernel and then kind of band-aid in user-land to just make it work? To make it possible. As opposed to just doing your own like main build. So testing a release on a daily basis from one release to the next takes me more than a week. Only compiling the kernel. And it would be 10 times slower if I compile it. Now here it's again in the state diagram how the user-land building works. So we come from here, check out the kernel, compile it, run the test, advance. And now we have to look. Okay, was there at work? If not, we can just go check out the next set. If yes, we check out the user-land at the date that's given in the build works, build our tool chain and then continue here. As explained just before. So what I want to do next, what I already have is I don't have only one setup. I have two setups. The difference between the two is here I have two sockets, eight cores. And here it's the same hardware with one socket, four cores. So I ripped out one CPU here physically and then the performance numbers are much more stable. So running on a two-socket CPU is dead especially if the operating system is not aware of the CPU, the socket architecture and the numer and whatever. So what I also want to do, I install the machine but don't need more, is to have a Linux here to make some performance test and compare it to Linux. So then we have a stable Linux here. It's always the same and we can look up what's the difference when we only change the system. But it's not running yet. Just last. So how does the installation work? It's the same setup as for the regression test, just different machines here. So there's a master that publishes it in the back. It says the console server. Okay, install the release. Installs it here and installs it here. And from the master there comes the configuration and there comes the recompile command. So it says, okay, here log in, check out LLVM, check out LLVM, build it. It builds it in parallel. And it goes here and says run the test. So the running the test is done locally. You can also just log into this machine and say, okay, run, run now. And then you will get, you can look at the machine, how it's running the test and what the kernel is doing and what top is doing and take some statistics there so you can really dig into it and that's a script that's just running here and doing all the, doing one test run. So does the I-PROF test the T-Speed and Bench, the Make, Timer and the Palses test. And it also logs made up in here and say, okay, here we start the I-PROF daemon as a receiver. Okay, now I have some results. So you remember the slides before, we had one that had a single performance peak and it was weekly. But it didn't always. I did the same test but it did it daily. So it just took two weeks. Yeah, it's 14 days. 14 days and daily checking the linear, we still have those strange numbers of two subject machines. We know that for now and now we see, ah, it's here and I could look at all the CVS commits here and here and I found out that DLG, an OpenVisity Committer, committed a T-SIM integration here where he collects multiple packets and sends it to the network card once instead of doing the hardware output for every packet. And it was committed here and it was back out here because it broke, suspended some resumes on laptops. So what I can see is, okay, that's a very valuable or interesting approach to increase performance. We just have to fix this problem. And I wrote DLG in mail and he fixed it later. So then there's another thing. Did you see the performance in Paris when you put it back? Yes, that's on the latest slide. So what we have now. In OpenBSD we recompile the kernel every time we reboot. So what I was trying to figure out why we have such changing numbers especially sometimes it goes up and down especially in the second run. So I disabled the recompiling the kernel because recompiling the kernel is done in the background after the machine has booted up and using starting some background jobs while you do performance testing is not good at you. So I disabled it. And then I asked myself what happens if I enable it? So I have three different things now what I can do when I repeat the test. So I come here, compile the kernel and I run the tests. Then I can repeat them and run them again and run them again and run them again. Or I can say, okay, what happens if I reboot the machine in between? So I run the test, reboot the machine, run it again, reboot the machine, run it again and go this way. Or I can say, okay, what happens if I relink the kernel? And the kernel in OpenVC means that we have a random sort order of our object files. The idea why we do this is security. We want to avoid that you can do blind rock attacks that you, when you don't know where the object files, how the object files are sorted within a binary, it's very hard to guess where the guesses are. You do it for rock. And that means you, it's very hard to exploit the kernel. And that's why we relink the kernel in a random order at every boot. So every OpenVC machine has a variation of the kernel layout. It's 1600 object files, I think, and you have all possible orders. So it's a very large, it's very hard to hit the right order. So I try to do that. So now what happens? I compare the results. So here I install 64, run it for five days, do 10 tests each, and keep the machine running at each test. So here, that's 10 tests executed here, 10 here, 10 here, 10 here, 10 here. And in between, I recompile the kernel and do the TBS checkout. But it's just after release. Here's the OpenVC release. And you go in the kernel, look and know nothing, performance relevant has been committed there. So either there was nothing committed or just some other things. And what you see, it's going up and down a little bit, but it looks quite stable. Then what happens if I reboot the machine between tests? You see it's going a little bit more, spreading a little bit more, but it looks quite similar. And that happens if I relink the kernel and sort and reorder the object files. You see it's a large spreading here, and I guess that's the locality of the caches, where we cache the execute of the instructions. And if you reorder the instructions, the cache locality is bad or at least random based on the layout of your kernel. So what you have to care about when you do performance testing, you have one kernel tested and get a number, then you apply it patch, recompile it, test it again. You may see this variation and not the patch you're testing. You have to repeat the testing, doing it multiple times to see the band of the one target and compare it with another band that might be above or below. Just by recompiling the kernel, you get variation. When you were rebooting the machine there, you were rebooting the machine with kernel relinking on reboot disabled. Yes, I disabled the relinking every time. Otherwise, the relinking. The relinking is the same enough as the plus unit. And the relinking would run in background when I run my tests. That would be bad also. So I disabled relinking and relink explicitly if I turn it on. So it's just doing it in front of the test. And now I go back to my two CPU socket machine, that's the 8-core machine, because I have another effect here. So I install 64. I do the test for 15 days. It's five tests each, and we have one line here, and we have another line here sometimes. So what's that? It's another unexplainable phenomenon. So when I click on the tables, I see here in the second cycle that I have those red numbers. And I have only the red numbers here and here and here and here. So and it's always that. It doesn't matter which one I take, it just has to select one of those with those numbers here. And if I click on it, I see this result. Here's red and there's red. So the second run is slower. So I reboot here. I have to keep. That means I don't reboot the machine. And here it's slower, and it's only slower when I'm sending it forward direction. This is a minus R. That means the eye curve is sending the packets in the other direction. That's not effective. But if I send it in the forward direction, it gets slower. And what I figured out is I run the tests in sequence. So I do the network test, and then I run kernel compiler, minus J8. And this minus J8 means that I have eight compilers running on all cores, and it affects the statistics that are relevant for scheduling. And after I compile the kernel, the eye curve is scheduled on another core, on another socket, and then performance drops, which didn't happen before. OK, that's halfway explainable. What's even more strange is, if I compile the kernel again, this effect goes away. Then they are always scheduled on the first socket, and then I have a maximum performance again. So I asked several developers, I asked the tennis, and couldn't believe that it is that way, and they asked NBI, and he thought that's very strange, and gave me two diffs that, in front scheduler, that didn't change anything. So that's an ununderstood effect. So what did I make to next? I did run from 6.3, that's here, to 6.4, that's here, and that's 202 days. It's a day resolution again. So you see the colors are different. The reason is that I used Galer machine. So I ripped out one of those sockets, and those CPUs run sockets because I had those unreliable resites, and that's the same hardware, but only one CPU with four cores. And then we don't have those ghost numbers here, we don't have the second line here. So what happens here? We installed it here, and here it goes down, I don't know why, and this N, there we added RedpoLine. RedpoLine is a spectrum mitigation recommended by Intel, and here in N I recompiled the compiler, and we set it, but it enables the RedpoLine workaround, and enabled in the kernel compiler to drop the quality. So I don't know what this is, that's probably a measurement error, and I went on, and here we enabled witness. Witness is a debugging tool to find out if there are any lock ordering problems in the kernel. So we got it from free VSD, and it saves some stack traces in the background, does some checks on locking, and if there's a problem, it panics and gives you a result. So it was just a debugging thing that we enabled during the release, during the development cycle, to get some results so that users use snapshot C, that the, see those stack traces and report them and we can fix it. Pouldy bugger. Pouldy bugger is always off. I have set the CCTL to, so I have a bunch of CCTLs that turn off anything that might affect performance, I try to be very fast. Except for one thing, in the BIOS I disabled the CPU turbo thingy, so CPUs can change their CPU speed, and when it's cool, and I say, okay, don't do that, don't get higher when it's cool, just keep the same level to make the numbers constant, but in the kernel I disabled all the debugging aids, like Pouldy bug, and the CPU CCTL to get stable numbers, because normally for release we disable it, and before we enable it, and you would see a jump here, but you don't see it. I've disabled it already. So what we get here is we turn on witness. The next one here is we enable RedPolyte, no, not RedPolyte, RedProtect. So RedProtect is a feature from Mortimer. Red guard. Red guard. Red guard. Sorry, Red guard. So it's Red guard. So it adds some additional checks when we leave each function to see if the stack pointer has been changed, and that gives a little bit of overhead to the attackant here, and what we did here is we turned off witness again. So witness was just a temporary experiment. You see, do we have locking problems in the kernel? We turned it on, got some reports, fixed some stuff, turned it on again, so we went back here. So we have what we've said here is RedPolyte and Red guard. Here I've added reference counting to the PCB. Let's think of it like a socket stutter structure in the kernel. So they are Red guard, Red guard now, and I think that's this little performance drop here, because it gives a terminated operation, it's synchronization between the CPUs. So now we go on. That's the next release. That's from 6.4 to 6.5. So here we are in April this year. Performance wise, that did not much happen. Sorry, I forgot the H. What happens here is at the I, that's an L, small L, there we disabled stack protector if Red guard is enabled. So the Red guard is a more paranoid check in the stack, and then you don't need the stack protector anymore, because everything the stack protector would find, Red guard would also find that it's a little performance optimization that may be related to that. The main problem is that we don't see precise numbers, but we always see the stand, and if there's a huge change, we can see it, but if it's only a little bit, it's hard to see and hard to guess what's going on. I remember what page it is. We added some feature that we get kind of stack phrases. We get them reported from our user, we see them in DVD, and some architectures print the arguments that were for the functions when you call them. For example, I3886 does it because it saves the most stack, and we see them on stack. I3886 doesn't do that because everything is passed to registers, and the stack trace cannot find the register values. And then we added a feature that we print, and then we write the register values to the stack so that we can see the arguments that were passed to the function from the stack trace, and we enabled it here. So that's the H. It's a little, a small drop. So now we have six, five, and as I told you, I created the slides here, so that's my last measurement. And you remember the TX mitigation from DOG? I wrote in my mail, it has a performance increase, and DOG said, okay, let's try to make it in a way that doesn't break the cloudier's laptop because it was cloudier reported in the ICP so it got back out. Here we implemented it again, and we see the same jump that we saw two releases before, and it wasn't back out. Still there. I wanted to tell you about the CVS log. So between every measurements, I go through the CVS and get all the, and sort all the changes that are relevant between those checkouts, so I have the begin checkout, that's the first measurement, and the second checkout, and everything in source SUS that was committed there, that was a commit, and they are all consecutive here, and a show which files were touched, and you can click on the diff, and you go to the CVS web, and you can look at the diffs of the file, and here you see the commit message, and here is the TX mitigation. That's the one that increased the performance. So you can just, from the front table, it's one click to this, and you see what was going on between the two columns. So I want to summarize my findings. Okay all this measuring sucks. It's a lot of work, and it's strange things going on, and it's hard. The multi-summet CPU sucks performance wise, especially when you want to accept precise numbers. Reproducing numbers is very hard, and even if you do, it's quite unclear what are the effects behind it. Do not trust your numbers, test it again, see that you can reproduce it, and keep the setup as simple and stupid as you can, otherwise you don't have a chance to get the cause of the changes, because you have too much complexity. So what else can be done? Nowadays I'm only testing between two open BSD and the stacks. I could also add another machine and measure some forwarding. I can set up this Linux machine, I've shown before, and use this as a gliding server. So I have one stable Linux release, and it has all open BSD variants between it, or I could use free BSD if you like. And the UDP test, I run UDP tests, but those numbers really suck. It's quite hard to say what does it mean if you get a number in UDP, because do you measure what you're sending or what you're receiving, and if you send too much, the kernel is busy with freeing things, without receiving things, and the I-PARF tool, I install the I-PARF tool of the release version, I have three different versions of I-PARF there, and they all have different algorithms to measure UDP. So that's very, very potential to improve. So what I'm ultimately going to do is when I get a diff, then I want to see, does this diff make the performance better or not, and I have no way to do that automatically right now. So what I want to do is put the diff in there, run the test, and have a website that's okay with this and without the diff, something else. I want to go farther back with releases, look at 59566061, just comparing them. The file system performance, the main problem with this machine is that we have our rate controller there, it's an LSI controller, which doesn't like the driver and firmware, and the performance is really bad, and so either you have to change the rate controller or fix the driver or take another hardware, and then I can do more reasonable calls to some performance. And then this line I added last week, because we have this band of results because of the sorting of the kernel, and I should add another mode where I sort the kernel object files in a constant way, so that I try to get more reliable numbers, even if I apply patches or change the checkout date. I didn't do everything on myself, I had some help. So there was a young planco, he does the hardware administration for all the machines, and creates the possibility to set up that. Then Moritz Fuhl is a student working for me, and he did all the guts with a loop plot to get those nice graphs, so I didn't have to dig into that. And Inua, my employer, is hosting the machines, giving the electricity, and gave me work time for that. So I have some links for you, so that's the top level thing, you can find the results of the rigorous tests, results of the performance tests, you can also get the raw data of the performance tests, that's only a loop plot form, just the numbers down there, and then the loop plot scripts get the right numbers out of it to create the graphs. I've committed the power scripts that create that on GitHub. This is the scripts that do the installation of the hardware machines, and that's the source code of those large slides. That's the final slide for the questions. Bob first. I didn't run that yet. So I'm also repeating another measurement, and those machines are busy right now. But youasına the Scriptureanas are flew. So everything the developer thought would be nice to test so we have regression test for LibreSys. For example, where just the crypto things We have So much in the test those libraries. I have written some tests for this log I Re-wrote this log more or less and I test all the things to test CLS there That's that knows what messages get lost and have a relay demon test So we have if you think about Tests Component tests so to give you an example from LibreSysL We'll have places where we decided we know what are for what exactly what should come out of a TLS handshake at these Steps and so there's literally hex encoded bytes of what the handshake should be and we brought it and we compare the output That would be like a component test and yet you've probably got something more in the lungs of a unit or an integration test So we we brought glue over and fed him a lot of state Sack him on my couch and he worked with the ports people and got it to where that same regression test There's also stuff that effectively got it where we could run have all the versions of open SSL installed along with LibreSysL and do Compatibility tests connecting between all of those. So it's kind of a it's everything from like component tests for functions to effectively almost And then the end of the process from 3bc And I wrote it so that they also run on NFS NFS and that's the test that Bob just mentioned here it's around the Client and server code he's a client SSL client server and then compile it with Libre SSL open SSL and SSL 1 1 and do a cross An interoperability interoperability test So I take the same binary and compile it with three kinds of libraries and then the other side and compile it with three kinds of libraries and do all tests So yes, yeah, so the bag that you're always talking about is in the 3Gbps secondary Yes So we started at 4 I was back up to 3 and a half That was before our MSI But how do we get more? Do I get a processing queue? Do I link the blocks? You help removing the big blocks in the open BPC cards And the socket is there again related to some issues? Is there also a relation in our kernel? Yes, so the basic problem in the open BPC kernel and that's why I did that because we want to get beyond the point is that we have Locking that only allows to networking on one CPU At the same time So and you have a lot of tension there Yes So I have a rather old machine there so that's a seon machine from 2012 So in anyway in the open BPC lab I collect all the hardware that nobody else needs anymore and And run my tests there and it more or less doesn't matter for the regression test It doesn't matter at all and for the Performance test. Yes, I need a machine with a lot of CPUs to make the CPU problem visible, but if it's 10 years old or Very expensive the new streets doesn't matter much because we have concentrated to get the kernel limitations solved And there it's not that important what hardware you have here around I still answer your question It was important just that would it invent a processor that explores more problems than a really fast processor? I would say we are not at that point currently Okay Any reason you're testing in bits per second rather than packets per second Basically because Doesn't matter much That's just I know the Yeah, not an understanding of all the magnitude number I expect from the previous Easter Interesting Oh, no, no How much of these tests is automatic and how much of it you have to do things manually to kick them off? So for the the problem for the regression test run every day across. Yeah, and the performance test I Am just go to the website What I do here and cross job that repeats the the last week So what what I do is when I say okay, I want to test to release I have a make line It's a man. It's controlled by make part so I can say okay do the Do this here make Something and then I can find you made so the make line I can make in minus and then it prints the line and then there's this okay It's it's a day step and I change the day to three hours and I say change to start in the end and Then then I fire it off so at the release of manually, but I've added is a crumb job It does the last week here. So every week Every Friday I get the daily resolution of the last week and heads down to the total slide So that's the slide with all measurements see here and Then you can pick on the other ones Single emitter and you always have the last one here and that's that the blue one is the yeah a core machine And you see it's lower than the the for machine Because we have this big lock and then it's scheduled the wrong core and then more lock contention By the way, I'll type some make outputs UDP and that's total crap The UDP Don't don't trust the UDP don't trust the unit so that's the machine that that's the The a core machine or yeah, that's the a core. I don't know what that is. Maybe that the dex mitigation Made faster for TCG and slower But but that's just the guess So so you piece this crap anyway, I guess one or two gigabit. I don't know why The pool is spreading that all the version of I perv Printed completely different format. So they are missing here We are to really missing because the output performance has changed and the change They how they interpret the the lost packets Okay, we have a question there Or I wanted to have multiple tools to To see if one tool has back Okay, but you think you see the bench as well. Yes. Yes. Yes. So what what what can do here movements? Was not much nicer here. You have to scroll Oh, why is it so crummy? Oh, yeah, so now So I have to go to the Somewhere so it's called out. So normally you have here first when I make it bigger So Now I had it so you can have the TCP bench only you can click on this and I perv vanish And now we are only testing TCP batch. Oh Now we can see the interesting part So they have to move it this way. There we go. Yeah, okay, so so you can you can choose what what measurements you want to see and And we can and I said those don't do it and Probably once our course and the ours And I made several measurements and this is the summary everything is in that side. I'm What are you wrapping this all in a key or QA Or some tests framework, how do you launch all that? You know, then there's not much for a road. It's ever You may want to consider turning off witness Because my experience with this is been that it is stored reality it is on So I always said the city out to turn it off. Yeah, but just compiling it in may slow Yeah, so that's that's the thing you saw there But it had no particular way to to patch it out and I also wanted to show what's the thing that the People who built the current at that time see yeah, so free BSE has an explicit disclaimer when we boot up with witness and in the configuration file because certain idiots Orics Run tests with witness enable them to play about that performance Witnesses fantastic to the water reversals, but if you're interested in performance turn them off because it will be garbage Yeah, yes indeed and then here it is show the feeling so that's the best Yeah, yeah, we use those packet generating tools in the company for our forwarding and VPN tests for our product and we have a Network stack that works in parallel the other we get much higher performance But for open BSD as I'm doing TCP here. That's that's really hard with those packet generators and And I'm using stack performance. I'm not doing forwarding performance here and Doing a real TCP. They don't do a real TCP high shake and generate packets there And so here I'm just flying open BSD with itself and every problem I see is somehow in the open It was more sorry more for the UDP But they're still the problem if you measure the sec What do you measure do you measure the number of packets that hit the stack or that they're received by the system and if you Send more packets to a sec you will receive less packets in the user that because the kernel is busy throwing them away If you're testing the phone forwarding. Yes, that's more suitable for forwarding. I didn't do forwarding yet. And you also have to be careful. So what we do at Genoa, we say we increase the sending rate until 1% of the packets get lost and we take the receiving rate of that and that's our UDP performance. But that's not that what the iCurve tool is doing there. So perhaps it's just writing our own tool for the stack or... Even with an iCurve 3 itself, I've seen a lot weirder stuff on previous games in the last year or two than I had seen in the past. Just stuff that wanting more stuff getting locked at Magical Model and stuff that even...not to say about iCurve 2 but iCurve 3 because that's a whole other monster. But I just wanted to throw that out there for anyone in the room. So what's possible with that, I can repeat those tests easily. So if I have a script that runs all the tests, it's not being healed to add another test and run the whole thing in the past again because I can generate a mystery at every moment. So any more questions? Okay, no questions. Thank you. Thank you.
 So, my name is Alexander Blum.  I'm an open VSD developer for more than 10 years now.  I work at the company in Munich.  We build firewall and VPN appliances based on open VSD.  So, it's also my job to make open VSD reliable and fast and suitable for our products.  So, what I want to do is detect changes in performance in open VSD when they occur.  So, when everything is something is committed and it affects performance, I want to see  okay, there was a difference and also see how does it develop or release as we have  the regressions here.  Instead, has there something that is good and I want to measure real-world performance.  So, it's basically network performance that I'm interested.  I started with network stack performance, not the forwarding performance.  It's a different story.  And so, I want to see how does our TCP stack behave.  So, what I will do is, first of all, I will talk about where we're coming from, what did  we have before.  I will also talk about my regression tests, then how does this performance testing over  the time work, what results did I find and what's the conclusion where I want to give  you on your way.  So, it starts with the same that was before I started the testing performance.  So, in Innova, that's a company of web form, we have a high performance firewall test that  and we use it for multiple purposes.  You want to generate numbers for that we control our customers that are basically high and  the marketing also was once high numbers.  The developers want to know how the system works and then we want to see when we buy  it, when we choose new hardware revisions, does it go faster because it's not faster,  what are the pensions we release, do we have regressions there, we want to have forwarding  performance again, performance and everything.  So, everything results in bad results for that what I want.  I want results for developers.  And what you see here is you can't see really what we are measuring, you have some dots  jumping around, you don't know why, you have some lines in there and it's also unclear.  By the way, we are testing the 6.0 release, always the same software, you have varying  results, it goes up and down and nobody knows why.  So the hardware setup is also rather complex because it's for multiple people doing testing  the same, doing tests at the same time.  We have different requirements as I just told you before and we want to test all the different  hardware so we have several machines where we can install the software on and it works  basically like this.  We have some Linux lines here and some Linux servers and we have some old machines there,  we bought them 10 years ago and so they had only one gigabit interface and we take several  of the machines with multiple interfaces, put them with a big switch and go with 10  gigabit through those targets and measure how much forward or with the end performance  we get there.  If you do that, it depends on how many other people are working simultaneously, then the  switch drops some packets or not and you have 20% variance in throughput depending on the  phase of the moon or whatever.  What I want, the problems are, we have too many requirements as I told you at the beginning,  we have too much complexity which is bad if you want to reliable numbers and if you want  to dig down what's really the problem there, you have not enough flexibility because it's  all set up in a static way and other people are using it in parallel.  If you start moving the cables around and reconfiguring the switches, other set of break or you don't  have enough permissions to configure the switch or people complain when you remove some cables.  What I want to have as a setup is this.  I have an open BSD machine here, I have another open BSD machine here and I have only a cable  in between, no switch, no nothing.  As much less things that influence my results and everything, I measure is somehow based  on the open BSD I want to get information about.  The other thing I had before is my regression tests.  I worked on that, I started that I think four years ago and I run the open BSD regression  tests on a daily basis.  Those regression tests existed before, they were just checked into the source tree and  every developer could do a make in some sub-directory and good pass or not.  But there was no regular execution of them and a lot of them were un-maintained or just  failed.  So what I started, I just took the test that already existed here and run them daily.  I can run them on multiple architectures to have a comparison, how does it work on 32  bit or 64 bit.  And I'm planning to extend it on Spark and we have little engine, big engine.  And what the nice thing is when you do it daily and record it and publish it on a web  page, then you get a history.  You can say, okay, three weeks ago this test started failing, so what was going on there?  I can link that to the commit or to the source change that caused the problem.  And that's something I also want to do for a regular performance.  And what I also do is collecting useful information.  I create log files, you can download the object files from the test run, you can debug it  more or less online and if it's really complicated you can also log in at the machine and do  your analysis there.  And everything is accessible, you just go to this website and you can see the results  of the things.  So here's how it looks like.  I didn't do it.  You didn't do what?  Break.  Ah, no, no.  But we didn't fix it.  So what we have here is a display of everything and it's quite red because we have new architectures  here that's ARM V7 and ARM 64 if it doesn't crash during the test run and I haven't spent  too much time on those ARM platforms so quite a lot of them is red.  But we are working on that.  So what I show here or what is told before, I want to have links and information on what's  going on.  So I put a lot of links here.  For example, the run history, you can click on that and you see when each test was executed,  how the machine was set up.  So what I do there, I get a snapshot from OpenBSD every time and every day or so, the  CDRAR builds a snapshot with the current release, no, not release, with the current software  version and just install it on the machine.  There's an auto installer for OpenBSD and I can just set it up.  The config is setting and here you can see how it installed and if they broke anything  or if the auto installer is broken or snapshot has some problem that they cannot install.  So then I have a link here to the CBS web of the test.  So here are the tests and you can just go in there and see, oh, did somebody commit something  in the test so that's the reason why it broke.  Here I have some color scheme to show what is wrong with the test.  The path is always green.  The red has failed.  You have skipped and that means there's something missing in the hardware setup so it cannot  run there.  So we have architecture specific tests, for example here is a GCC test and on R64 we don't  have GCC so we skip it.  And there are some other reasons why I skip it.  We also have some tests that use that to something with the keyboard driver and if there's no  mouse attached then it's skipped.  And if you click on such a fail or in the past you get a link to the log to the test  output.  So you can see or get an idea why the test fails.  So here I have a link to the logs of the setup of this specific date.  So this is the date when I run it.  See this one is the most recent one.  So I made the slides last month.  And here every day you get four new architectures.  And when you click on that you get the logs how it was installed and you can also download  the object files that were generated while the test was run and perhaps you see it gets  some more information from there.  And I also have a cron job to delete that here.  So I cut it a little bit and here the test results are deleted so that the drive doesn't  fill up.  And here you can also click on it.  You get a D-Mess from the machine and see what version was actually tested, who built  the terminal and what's the test target.  So what we get from it now I clicked on the only show me the IP86 tests.  It's IP86 because that's the first architecture I started with.  And so we have the history here when we did run the test.  So we see, okay, this here has some problems forever.  And here this libcrop door something here between 13th and 15th went wrong.  So here I have to start and investigate.  So those GCC tests failed since we switched to the playing linker because it misses some  features and then so it basically checks whether some warning of some unsafe functions are  emitted by the linker.  And that doesn't work currently.  But Kden is promised to fix it.  And here we have other tests that pass sometimes and fail sometimes that basically means it's  not a stable test.  There should be some more investigation, make it more stable.  So that's a fragment test here.  Send some IP fragments with escapee to another machine and maybe some packets get lost and  when they get lost the test fails and it's a little bit fragile.  So the other thing I do is I sort them here.  So if something, so this test I fixed it four days ago.  But if this test here for example it doesn't terminate.  So it just takes too long and it takes more than an hour.  After an hour every test is supported.  And so it's no-term category.  So we have some failures.  They are sorted here.  We have some paths here that they are, they fail in some other architecture.  That's why they are sorted here and here we have some skip tests.  And we have a lot of really passing tests down here but I cut it away from the slides  because it didn't fit.  And the interesting thing for the developers is that you should concentrate on that and  fix that.  So I run all the tests on hardware.  So most of them are just running one machine.  So we have some kernel installed here, some user-landed installed here, some tests here  in user-source.  Just type make and then something happens and it either passes or fails.  But there are some other tests that are more complicated.  For example, I have a test that tests ARC, the ARC protocol.  And they have a remote machine and I send some SKP packets to it.  And I send some fake ARC packets and look at the ARC replies and check whether that fits.  So you can do a test with two machines.  This one controls everything.  The tests create this run, the make file run.  And it may log in with SSH to here to make some setup and it may send some packets here  that do the actual test.  There are tests that have more machines involved.  The most complex one are testing PF, forwarding, redirection and fragmentation.  So I run the PF.  The PF that it says that it's basically run here.  So I can send a packet here.  Then it goes through PF, gets modified or reassembled whatever tests I'm doing here.  Then it goes to a router.  I need the router here because I want to do pathMTU tests.  So this router has a limit pathMTU at the interface.  It's only 1,300.  So the packet has to be fragmented somewhere here with its IPv6.  So here, if it's IPv4, then it hits to the server.  The server is, I am just running the normal ping service of the current.  So if it's been packet, it's just returned.  And it also runs the echo service on part 7.  If it's a RubyP packet, it's just returned.  And there's also TCP echo service.  You open the connection.  Everything you write in gets back out.  So I can just write a TCP stream through this.  And here it has to do some pathMTU.  Here the PF has to do some magic things.  And then I get everything.  So back, if it works.  If it doesn't get the data back, the test fails.  So how do we set up that?  I told a little bit about it.  So I have this master here.  That's also the web server.  You can publish the results.  And this master controls the serial console server that also handles all the installations.  So here I can say, OK, install or upgrade all the machines.  And it will install the latest snapshot on these machines with the outer installer.  And the OpenBSD installer allows to put a site, TGZ, somewhere that's where all the  configuration files are in there.  They are stored here.  And here it gets all the configuration files and gets the IP addresses.  And it's configured statically so that it's set up to run those tests.  OK, now we switch from regression testing to performance testing.  What did you do?  Fix the test?  So what do we want?  First of all, we want to see the history.  When did things get slower or faster?  You want to have reproducible results.  So if you run the test once, get a number.  If I run it again, I want to have the same number.  That's pretty hard.  I want to have the details so I can see, OK, in which situation was the machine, was everything  set up correctly, also see which commits were in that region, and what date of the curve  was run there, what user line was running there.  I want to have a drill down.  So if I see, OK, one month ago something broke or got slower.  Then I want to drill down and say, OK, which commit was it?  Which hour was it?  What did it really cause that?  And I want to have it somehow automatic so I don't want to run everything manually.  The performance tests are run by a cron job and it's just in the background, just needs  a little bit of babysitting if something breaks.  Hopefully it doesn't break, doesn't add nothing breaks.  And so I also want to see automatic and basically I do the test runs manually now but I have  set up a cron job that updates the last week of commits every week.  So how does it work?  So in the years, I installed an OpenBSD release.  And I have a defined point where I can start from.  I have a kernel installed and the user line installed and it's publicly released and everyone  knows what it is.  And then I check out the kernel at a certain date.  So I want to test this kernel work.  Then I compile it.  I compile it on both machines.  You remember, I have two machines on the table in between it and the same thing for on both  machines.  So I compile it.  Then I run the tests and then I can have a data how much I want to step forward.  So maybe a day or month or week or an hour.  Usually when I want a day, a day based stepping is a good resolution.  So I step one day ahead and do the next check out.  I compile the kernel, run the tests and then do that until I'm at the end date that might  be the next release.  And then I connect the results to make some statistics about it and present them.  And you're not rebuild the user line?  Oh, that's on the next slide.  So basically I want to avoid to rebuild a user line because it takes forever.  So compiling the kernel is two minutes, compiling the user line is two hours.  So what does the result look like?  Basically I copied the protocol that generates those results from the regression test.  So it looks quite similar.  So what we have here is the run history.  So you can see when was it installed and how was the kernel compiled and everything.  We have here some details in this line.  I'll just play it later.  Then here we have the tests I run.  Basically it's an IPer 3 from port system I installed there.  I run some TCP tests.  It also has a minus R option that normally when you have an IPer server, the client makes  the server and all data is sent from client to server.  And with minus R, it's the other direction.  And we run some TCP bench.  It's TCP, TCP bench.  Here it's with 100 simultaneous connections.  Here's only one connections.  We run IPer with Udiki and reverse.  And I have two test setups.  So we have one test setup with one kind of machine and there's another IP address.  The IP address is coded here somewhere.  So the one machine is here and the other machines, the test results are down there.  They're basically the same test.  And here I compiled the kernel and measured the time how long it takes to convert the  kernel.  The main idea that I got is that I got...  So what we did before, before we did some performance testing, people compile their machines.  So developers compile the kernel, compile their user land and they always say, oh, it's getting  slower.  And to measure that it's getting slower, I also just do a kernel compile, make some  A clean and then run J4 on the 4 CPU machine and J8 on the 8 CPU machine.  And I also do some disk drive, filesystem performance, but I'm not satisfied with that.  I had to find you in the...  So now when you click...  Oh no, I forgot to explain this.  So here we have the date when I run the test.  Here it's the version I installed.  So in this case it's 64 release.  I installed it.  And then I have the first CVS checkout that's here.  So it started here at 16th of March and I stopped at 23rd of March and it's 8 test runs  in between but one day's there.  And I repeated the test five times.  I'll give more detail about that later.  So if you click on one, it's such test runs.  So if you click here, you get to the next slide.  That shows what happened in one of those test runs.  So here you still...  If you again have those logs, you can look at more details.  And here's the checkout date.  So here this test I did run in February and I checked out October 2017.  And here I checked out...  It's 4th of October, that's 11th of October, 18th of October, so I have a week's step.  I stepped weekly through the 6th to release, I think.  The idea is the 6th to it was installed and I was there weekly through it.  So what I do now, I have the checkout date and between this date and this date, there  were 49 kernel commits.  And if you click on the CVS log, you also see all the commits.  I go through the CVS history and collect everything and show what was between here and here.  Later there will be this library.  Show it.  And then we have those build works.  So as Bob asked already what happens with userland, I have to recompile the userland  from time to time.  The thing is that netBSD can be compiled, the kernel can be compiled on any POSIX system.  That's not the case with OpenBSD.  The OpenBSD kernel requires that it's compiled and built on a system, on a userland system  that matches more or less its date.  And so I have to update the userland so that it's compatible to the kernel.  So basically what I have to do is recompile the kernel, recompile the compiler, the client  compiler, when it's updated.  And that's also good performance wise because I'm not only interested in the kernel performance,  but I also want to know what do the compiler and its options have an influence on the performance.  So I also measure the important steps in userland.  So every time I have such a build work, then I know, okay, here I have to do something  with a userland.  I'll explain that later in more detail.  So and now here you have the numbers of this test.  That's the I-curve TCP in forward direction.  And I repeat those tests multiple times to get more stable numbers.  My observation was that the test goes up and down and gets faster and slower.  And so I try to do it here five times and here present average numbers.  It's not quite correct.  For network performance throughput, I take the maximum number.  So that's the maximum number.  And when those repetitions if results that are not stable that are jumping a lot, then  I print them in the right here.  So this one has a lot of jumping in it.  You see it's very right.  So here we should look into more detail.  So I get to the next slide.  So if you click on the top, click here.  And you go to the details of all the repetition.  So you see here it looks like here we have some steps of log files.  And here it's repeat one, zero, one, two, three, four.  And here we have some strange values.  All of them are right.  You can't read the numbers, but here it's four gigabits throughput.  And here it's only 1.6.  So what happens?  This is an 8-core machines with two sockets.  And these two socket machines are very unreliable performance wise.  And it happens that once a while the throughput goes down to 1.6 gigabit and I don't know  why.  So if you reboot it, it goes up again.  So here we have four, so that's 3, 9, 4, 3, 8, 1.6, 3, 8.  And if you reboot it, it gets into this state.  And if you reboot it again, it gets out.  I don't know why it's so.  So here in the back I calculate the standard deviation.  And I also print it in red when it's too big.  So I see easily where the problem is.  And report that to the next slide.  So it can also be generated some graphs out of that.  So here we install 6.2 here.  And we run until 6.3.  And we make one measurement every week.  So as you told you, I have several tests that have shown us different symbols here.  So there is an explanation on the website, but it doesn't matter which test it is.  And what you see here, we have this.  That's the occasional 1.6 throughput.  Nobody knows why.  And most of them are in this band here.  You also see that the numbers are jumping up and down.  So it's not one line.  Here we have one line.  But here on the top it's more or less.  The different tests also give different results.  And here we have one test that's faster.  I'll explain later what it is.  But just remember, we saw it here.  What I also do is those vertical lines that are the quirks.  When I have to touch user lens to make the kernel compile again or to change the compiler  or whatever, then I have such a curve.  I have a static file with the build quirks where I have a date.  It says, OK, at this date, we have to update the compiler.  And if the previous test is before the date and the next test is after the date, in the  meantime, the compiler is recompiled.  So here's a small example of the quirks that happened in the previous slide.  So to make it easily detectable, I also have a line for the release date.  I don't build anything there.  I just want to show you where the release was.  So first thing I had to do, I wanted to check out the compiler.  And I saw that there's a bug in CBS.  And you cannot check out a vendor branch if there was a commit on a non-vendor branch  later.  Bug in CBS.  So I fixed CBS for two and added a patch to the quirks and recompiled CBS.  Then I can check out LLBM that was released 5.0 at that point and recompile the kernel  with that.  What we also have is there's a header file that's called PNBar where the PF structure  is in there.  So when we change the PF structure in the kernel, the PF CTL tutor to change the kernel  has to be compatible.  Otherwise, kernel and tool don't have the same idea to construct.  And the update of the PF rule fails.  And usually you see that the performance difference.  Because I run it with PF because the default rule.  And if you remove the PF rules, it can be loaded.  Then you have a different PF set.  And then we see other performance.  And so I figure out, when I look at the graph, oh, there's a problem.  Then I add a perk to say, OK, here, when we have the PNBar header file change, then I  have to recompile the PF CTL.  That's rather short.  It's just a few seconds to recompile it.  So the next problem we had, somebody committed here a driver and he made a mistake.  So the tree was broken.  It was only broken for half a day or so.  But if my metamatter just hits that moment, it can't compile the kernel.  So I have to look at the fix that's committed later and patch that and the sources that  were before and then it compiles again.  So what did here?  Here was another PF CTL set change.  So the CTL has the same problem.  If the system CTL strikes change, then it doesn't work anymore.  That may have some influence how the system was booted, what's this here that's perceived  to be removed.  So if that changes, I just recompile it.  Here we updated LLVM again.  Here we had some PF change.  And here we have the LLVM Vs series.  So that's how the perks work.  That's the segify where everything, it's a per module where I've described all the things  that change there.  Bob, you look like you have a question.  Why did you choose to just do the kernel and then kind of band-aid in user-land to just  make it work?  To make it possible.  As opposed to just doing your own like main build.  So testing a release on a daily basis from one release to the next takes me more than  a week.  Only compiling the kernel.  And it would be 10 times slower if I compile it.  Now here it's again in the state diagram how the user-land building works.  So we come from here, check out the kernel, compile it, run the test, advance.  And now we have to look.  Okay, was there at work?  If not, we can just go check out the next set.  If yes, we check out the user-land at the date that's given in the build works, build  our tool chain and then continue here.  As explained just before.  So what I want to do next, what I already have is I don't have only one setup.  I have two setups.  The difference between the two is here I have two sockets, eight cores.  And here it's the same hardware with one socket, four cores.  So I ripped out one CPU here physically and then the performance numbers are much more  stable.  So running on a two-socket CPU is dead especially if the operating system is not aware of the  CPU, the socket architecture and the numer and whatever.  So what I also want to do, I install the machine but don't need more, is to have a Linux here  to make some performance test and compare it to Linux.  So then we have a stable Linux here.  It's always the same and we can look up what's the difference when we only change the  system.  But it's not running yet.  Just last.  So how does the installation work?  It's the same setup as for the regression test, just different machines here.  So there's a master that publishes it in the back.  It says the console server.  Okay, install the release.  Installs it here and installs it here.  And from the master there comes the configuration and there comes the recompile command.  So it says, okay, here log in, check out LLVM, check out LLVM, build it.  It builds it in parallel.  And it goes here and says run the test.  So the running the test is done locally.  You can also just log into this machine and say, okay, run, run now.  And then you will get, you can look at the machine, how it's running the test and what  the kernel is doing and what top is doing and take some statistics there so you can  really dig into it and that's a script that's just running here and doing all the, doing  one test run.  So does the I-PROF test the T-Speed and Bench, the Make, Timer and the Palses test.  And it also logs made up in here and say, okay, here we start the I-PROF daemon as a  receiver.  Okay, now I have some results.  So you remember the slides before, we had one that had a single performance peak and  it was weekly.  But it didn't always.  I did the same test but it did it daily.  So it just took two weeks.  Yeah, it's 14 days.  14 days and daily checking the linear, we still have those strange numbers of two subject  machines.  We know that for now and now we see, ah, it's here and I could look at all the CVS commits  here and here and I found out that DLG, an OpenVisity Committer, committed a T-SIM integration  here where he collects multiple packets and sends it to the network card once instead  of doing the hardware output for every packet.  And it was committed here and it was back out here because it broke, suspended some  resumes on laptops.  So what I can see is, okay, that's a very valuable or interesting approach to increase  performance.  We just have to fix this problem.  And I wrote DLG in mail and he fixed it later.  So then there's another thing.  Did you see the performance in Paris when you put it back?  Yes, that's on the latest slide.  So what we have now.  In OpenBSD we recompile the kernel every time we reboot.  So what I was trying to figure out why we have such changing numbers especially sometimes  it goes up and down especially in the second run.  So I disabled the recompiling the kernel because recompiling the kernel is done in  the background after the machine has booted up and using starting some background jobs  while you do performance testing is not good at you.  So I disabled it.  And then I asked myself what happens if I enable it?  So I have three different things now what I can do when I repeat the test.  So I come here, compile the kernel and I run the tests.  Then I can repeat them and run them again and run them again and run them again.  Or I can say, okay, what happens if I reboot the machine in between?  So I run the test, reboot the machine, run it again, reboot the machine, run it again  and go this way.  Or I can say, okay, what happens if I relink the kernel?  And the kernel in OpenVC means that we have a random sort order of our object files.  The idea why we do this is security.  We want to avoid that you can do blind rock attacks that you, when you don't know where  the object files, how the object files are sorted within a binary, it's very hard to  guess where the guesses are.  You do it for rock.  And that means you, it's very hard to exploit the kernel.  And that's why we relink the kernel in a random order at every boot.  So every OpenVC machine has a variation of the kernel layout.  It's 1600 object files, I think, and you have all possible orders.  So it's a very large, it's very hard to hit the right order.  So I try to do that.  So now what happens?  I compare the results.  So here I install 64, run it for five days, do 10 tests each, and keep the machine running  at each test.  So here, that's 10 tests executed here, 10 here, 10 here, 10 here, 10 here.  And in between, I recompile the kernel and do the TBS checkout.  But it's just after release.  Here's the OpenVC release.  And you go in the kernel, look and know nothing, performance relevant has been committed there.  So either there was nothing committed or just some other things.  And what you see, it's going up and down a little bit, but it looks quite stable.  Then what happens if I reboot the machine between tests?  You see it's going a little bit more, spreading a little bit more, but it looks quite similar.  And that happens if I relink the kernel and sort and reorder the object files.  You see it's a large spreading here, and I guess that's the locality of the caches, where  we cache the execute of the instructions.  And if you reorder the instructions, the cache locality is bad or at least random based on  the layout of your kernel.  So what you have to care about when you do performance testing, you have one kernel tested  and get a number, then you apply it patch, recompile it, test it again.  You may see this variation and not the patch you're testing.  You have to repeat the testing, doing it multiple times to see the band of the one target and  compare it with another band that might be above or below.  Just by recompiling the kernel, you get variation.  When you were rebooting the machine there, you were rebooting the machine with kernel  relinking on reboot disabled.  Yes, I disabled the relinking every time.  Otherwise, the relinking.  The relinking is the same enough as the plus unit.  And the relinking would run in background when I run my tests.  That would be bad also.  So I disabled relinking and relink explicitly if I turn it on.  So it's just doing it in front of the test.  And now I go back to my two CPU socket machine, that's the 8-core machine, because I have  another effect here.  So I install 64.  I do the test for 15 days.  It's five tests each, and we have one line here, and we have another line here sometimes.  So what's that?  It's another unexplainable phenomenon.  So when I click on the tables, I see here in the second cycle that I have those red  numbers.  And I have only the red numbers here and here and here and here.  So and it's always that.  It doesn't matter which one I take, it just has to select one of those with those numbers  here.  And if I click on it, I see this result.  Here's red and there's red.  So the second run is slower.  So I reboot here.  I have to keep.  That means I don't reboot the machine.  And here it's slower, and it's only slower when I'm sending it forward direction.  This is a minus R. That means the eye curve is sending the packets in the other direction.  That's not effective.  But if I send it in the forward direction, it gets slower.  And what I figured out is I run the tests in sequence.  So I do the network test, and then I run kernel compiler, minus J8.  And this minus J8 means that I have eight compilers running on all cores, and it affects  the statistics that are relevant for scheduling.  And after I compile the kernel, the eye curve is scheduled on another core, on another socket,  and then performance drops, which didn't happen before.  OK, that's halfway explainable.  What's even more strange is, if I compile the kernel again, this effect goes away.  Then they are always scheduled on the first socket, and then I have a maximum performance  again.  So I asked several developers, I asked the tennis, and couldn't believe that it is that  way, and they asked NBI, and he thought that's very strange, and gave me two diffs that,  in front scheduler, that didn't change anything.  So that's an ununderstood effect.  So what did I make to next?  I did run from 6.3, that's here, to 6.4, that's here, and that's 202 days.  It's a day resolution again.  So you see the colors are different.  The reason is that I used Galer machine.  So I ripped out one of those sockets, and those CPUs run sockets because I had those  unreliable resites, and that's the same hardware, but only one CPU with four cores.  And then we don't have those ghost numbers here, we don't have the second line here.  So what happens here?  We installed it here, and here it goes down, I don't know why, and this N, there we added  RedpoLine.  RedpoLine is a spectrum mitigation recommended by Intel, and here in N I recompiled the compiler,  and we set it, but it enables the RedpoLine workaround, and enabled in the kernel compiler  to drop the quality.  So I don't know what this is, that's probably a measurement error, and I went on, and here  we enabled witness.  Witness is a debugging tool to find out if there are any lock ordering problems in the  kernel.  So we got it from free VSD, and it saves some stack traces in the background, does some  checks on locking, and if there's a problem, it panics and gives you a result.  So it was just a debugging thing that we enabled during the release, during the development  cycle, to get some results so that users use snapshot C, that the, see those stack traces  and report them and we can fix it.  Pouldy bugger.  Pouldy bugger is always off.  I have set the CCTL to, so I have a bunch of CCTLs that turn off anything that might  affect performance, I try to be very fast.  Except for one thing, in the BIOS I disabled the CPU turbo thingy, so CPUs can change their  CPU speed, and when it's cool, and I say, okay, don't do that, don't get higher when  it's cool, just keep the same level to make the numbers constant, but in the kernel I  disabled all the debugging aids, like Pouldy bug, and the CPU CCTL to get stable numbers,  because normally for release we disable it, and before we enable it, and you would see  a jump here, but you don't see it.  I've disabled it already.  So what we get here is we turn on witness.  The next one here is we enable RedPolyte, no, not RedPolyte, RedProtect.  So RedProtect is a feature from Mortimer.  Red guard.  Red guard.  Red guard.  Sorry, Red guard.  So it's Red guard.  So it adds some additional checks when we leave each function to see if the stack pointer  has been changed, and that gives a little bit of overhead to the attackant here, and  what we did here is we turned off witness again.  So witness was just a temporary experiment.  You see, do we have locking problems in the kernel?  We turned it on, got some reports, fixed some stuff, turned it on again, so we went back  here.  So we have what we've said here is RedPolyte and Red guard.  Here I've added reference counting to the PCB.  Let's think of it like a socket stutter structure in the kernel.  So they are Red guard, Red guard now, and I think that's this little performance drop  here, because it gives a terminated operation, it's synchronization between the CPUs.  So now we go on.  That's the next release.  That's from 6.4 to 6.5.  So here we are in April this year.  Performance wise, that did not much happen.  Sorry, I forgot the H.  What happens here is at the I, that's an L, small L, there we disabled stack protector  if Red guard is enabled.  So the Red guard is a more paranoid check in the stack, and then you don't need the stack  protector anymore, because everything the stack protector would find, Red guard would  also find that it's a little performance optimization that may be related to that.  The main problem is that we don't see precise numbers, but we always see the stand, and  if there's a huge change, we can see it, but if it's only a little bit, it's hard to see  and hard to guess what's going on.  I remember what page it is.  We added some feature that we get kind of stack phrases.  We get them reported from our user, we see them in DVD, and some architectures print the  arguments that were for the functions when you call them.  For example, I3886 does it because it saves the most stack, and we see them on stack.  I3886 doesn't do that because everything is passed to registers, and the stack trace cannot  find the register values.  And then we added a feature that we print, and then we write the register values to the  stack so that we can see the arguments that were passed to the function from the stack  trace, and we enabled it here.  So that's the H.  It's a little, a small drop.  So now we have six, five, and as I told you, I created the slides here, so that's my last  measurement.  And you remember the TX mitigation from DOG?  I wrote in my mail, it has a performance increase, and DOG said, okay, let's try to make it in  a way that doesn't break the cloudier's laptop because it was cloudier reported in the ICP  so it got back out.  Here we implemented it again, and we see the same jump that we saw two releases before,  and it wasn't back out.  Still there.  I wanted to tell you about the CVS log.  So between every measurements, I go through the CVS and get all the, and sort all the  changes that are relevant between those checkouts, so I have the begin checkout, that's the first  measurement, and the second checkout, and everything in source SUS that was committed  there, that was a commit, and they are all consecutive here, and a show which files were  touched, and you can click on the diff, and you go to the CVS web, and you can look at  the diffs of the file, and here you see the commit message, and here is the TX mitigation.  That's the one that increased the performance.  So you can just, from the front table, it's one click to this, and you see what was going  on between the two columns.  So I want to summarize my findings.  Okay all this measuring sucks.  It's a lot of work, and it's strange things going on, and it's hard.  The multi-summet CPU sucks performance wise, especially when you want to accept precise  numbers.  Reproducing numbers is very hard, and even if you do, it's quite unclear what are the  effects behind it.  Do not trust your numbers, test it again, see that you can reproduce it, and keep the  setup as simple and stupid as you can, otherwise you don't have a chance to get the cause of  the changes, because you have too much complexity.  So what else can be done?  Nowadays I'm only testing between two open BSD and the stacks.  I could also add another machine and measure some forwarding.  I can set up this Linux machine, I've shown before, and use this as a gliding server.  So I have one stable Linux release, and it has all open BSD variants between it, or I  could use free BSD if you like.  And the UDP test, I run UDP tests, but those numbers really suck.  It's quite hard to say what does it mean if you get a number in UDP, because do you measure  what you're sending or what you're receiving, and if you send too much, the kernel is busy  with freeing things, without receiving things, and the I-PARF tool, I install the I-PARF  tool of the release version, I have three different versions of I-PARF there, and they  all have different algorithms to measure UDP.  So that's very, very potential to improve.  So what I'm ultimately going to do is when I get a diff, then I want to see, does this  diff make the performance better or not, and I have no way to do that automatically right  now.  So what I want to do is put the diff in there, run the test, and have a website that's okay  with this and without the diff, something else.  I want to go farther back with releases, look at 59566061, just comparing them.  The file system performance, the main problem with this machine is that we have our rate  controller there, it's an LSI controller, which doesn't like the driver and firmware,  and the performance is really bad, and so either you have to change the rate controller  or fix the driver or take another hardware, and then I can do more reasonable calls to  some performance.  And then this line I added last week, because we have this band of results because of the  sorting of the kernel, and I should add another mode where I sort the kernel object files  in a constant way, so that I try to get more reliable numbers, even if I apply patches  or change the checkout date.  I didn't do everything on myself, I had some help.  So there was a young planco, he does the hardware administration for all the machines, and creates  the possibility to set up that.  Then Moritz Fuhl is a student working for me, and he did all the guts with a loop plot  to get those nice graphs, so I didn't have to dig into that.  And Inua, my employer, is hosting the machines, giving the electricity, and gave me work time  for that.  So I have some links for you, so that's the top level thing, you can find the results  of the rigorous tests, results of the performance tests, you can also get the raw data of the  performance tests, that's only a loop plot form, just the numbers down there, and then  the loop plot scripts get the right numbers out of it to create the graphs.  I've committed the power scripts that create that on GitHub.  This is the scripts that do the installation of the hardware machines, and that's the source  code of those large slides.  That's the final slide for the questions.  Bob first.  I didn't run that yet.  So I'm also repeating another measurement, and those machines are busy right now.  But youasına the Scriptureanas are flew.  So everything the developer thought would be nice to test so we have regression test for  LibreSys. For example, where just the crypto things  We have  So much in the test those libraries. I have written some tests for this log  I  Re-wrote this log more or less and I test all the things to test CLS there  That's that knows what messages get lost and have a relay demon test  So we have  if you think about  Tests  Component tests so to give you an example from LibreSysL  We'll have places where we decided we know what are for what exactly what should come out of a TLS handshake at these  Steps and so there's literally hex encoded bytes of what the handshake should be and we brought it and we compare the output  That would be like a component test and yet you've probably got something more in the lungs of a unit or an integration test  So we we brought glue over and fed him a lot of state  Sack him on my couch and he worked with the ports people and got it to where that same regression test  There's also stuff that effectively got it where we could run have all the versions of open SSL installed along with LibreSysL and do  Compatibility tests connecting between all of those. So it's kind of a it's everything from like component tests for functions to  effectively almost  And then the end of the process from 3bc  And I wrote it so that they also run on NFS  NFS and that's the test that Bob just mentioned here it's around the  Client and server code he's a client SSL client server and then compile it with Libre SSL open SSL and SSL 1 1 and do a cross  An interoperability interoperability test  So I take the same binary and compile it with three kinds of libraries and then the other side and compile it with three kinds of libraries and do all  tests  So yes, yeah, so the bag that you're always talking about is in the 3Gbps secondary  Yes  So we started at 4  I was back up to 3 and a half  That was before our MSI  But how do we get more? Do I get a processing queue? Do I link the blocks?  You help removing the big blocks in the open BPC cards  And the socket is there again related to some issues? Is there also a relation in our kernel?  Yes, so the basic problem in the open BPC kernel and that's why I did that because we want to get beyond the point is that we have  Locking that only allows to networking on one CPU  At the same time  So and you have a lot of tension there  Yes  So I have a rather old machine there so that's a seon machine from 2012  So in anyway in the open BPC lab I collect all the hardware that nobody else needs anymore and  And run my tests there and it more or less doesn't matter for the regression test  It doesn't matter at all and for the  Performance test. Yes, I need a machine with a lot of CPUs to make the  CPU problem visible, but if it's 10 years old or  Very expensive the new streets doesn't matter much because we have concentrated to get the kernel limitations solved  And there it's not that important what hardware you have here around  I still answer your question  It was important just that would it invent a processor that explores more problems than a really fast processor?  I would say we are not at that point currently  Okay  Any reason you're testing in bits per second rather than packets per second  Basically because  Doesn't matter much  That's just I know the  Yeah, not an understanding of all the magnitude number I expect from the previous Easter  Interesting  Oh, no, no  How much of these tests is automatic and how much of it you have to do things manually to kick them off?  So for the the problem for the regression test run every day across. Yeah, and the performance test I  Am just go to the website  What I do here and cross job that repeats the the last week  So what what I do is when I say okay, I want to test to release I have a make line  It's a man. It's controlled by make part so I can say okay do the  Do this here make  Something and then I can find you made so the make line  I can make in minus and then it prints the line and then there's this okay  It's it's a day step and I change the day to three hours and I say change to start in the end and  Then then I fire it off so at the release of manually, but I've added is a crumb job  It does the last week here. So every week  Every Friday I get the daily resolution of the last week and heads down to the total slide  So that's the slide with all measurements  see here and  Then you can pick on the other ones  Single emitter and you always have the last one here and that's that the blue one is the yeah a core machine  And you see it's lower than the the for machine  Because we have this big lock and then it's scheduled the wrong core and then more lock contention  By the way, I'll type some make outputs  UDP and that's total crap  The UDP  Don't don't trust the UDP don't trust the unit so that's the machine that that's the  The a core machine or yeah, that's the a core. I don't know what that is. Maybe that the dex mitigation  Made faster for TCG and slower  But but that's just the guess  So so you piece this crap anyway, I guess one or two gigabit. I don't know why  The pool is spreading that all the version of I perv  Printed completely different format. So they are missing here  We are to really missing because the output performance has changed and the change  They how they interpret the the lost packets  Okay, we have a question there  Or I wanted to have multiple tools to  To see if one tool has back  Okay, but you think you see the bench as well. Yes. Yes. Yes. So what what what can do here movements?  Was not much nicer here. You have to scroll  Oh, why is it so crummy?  Oh, yeah, so now  So I have to go to the  Somewhere so it's called out. So normally you have here first when I make it bigger  So  Now I had it so you can have the TCP bench only you can click on this and I perv vanish  And now we are only testing TCP batch. Oh  Now we can see the interesting part  So they have to move it this way. There we go. Yeah, okay, so so you can you can choose what what  measurements you want to see and  And we can and I said those don't do it and  Probably once our course and the ours  And I made several measurements and this is the summary everything is in that side. I'm  What are you wrapping this all in a key or QA  Or some tests framework, how do you launch all that?  You know, then there's not much for a road. It's ever  You may want to consider turning off witness  Because my experience with this is been that it is stored reality it is on  So I always said the city out to turn it off. Yeah, but just compiling it in may slow  Yeah, so that's that's the thing you saw there  But it had no particular way to to patch it out and I also wanted to show what's the thing that the  People who built the current at that time see yeah, so free BSE has an explicit  disclaimer when we boot up with witness and in the configuration file because certain idiots  Orics  Run tests with witness enable them to play about that performance  Witnesses fantastic to the water reversals, but if you're interested in performance turn them off because it will be garbage  Yeah, yes indeed and then here it is show the feeling so that's the best  Yeah, yeah, we use those packet generating tools in the company for our forwarding and VPN tests for our product and we have a  Network stack that works in parallel the other we get much higher performance  But for open BSD as I'm doing TCP here. That's that's really hard with those packet generators and  And I'm using stack performance. I'm not doing forwarding performance here and  Doing a real TCP. They don't do a real TCP high shake and generate packets there  And so here I'm just flying open BSD with itself and every problem I see is somehow in the open  It was more sorry more for the UDP  But they're still the problem if you measure the sec  What do you measure do you measure the number of packets that hit the stack or that they're received by the system and if you  Send more packets to a sec you will receive less  packets in the user that because the kernel is busy throwing them away  If you're testing the phone  forwarding. Yes, that's more suitable for forwarding. I didn't do forwarding yet. And  you also have to be careful. So what we do at Genoa, we say we increase the sending rate  until 1% of the packets get lost and we take the receiving rate of that and that's our  UDP performance. But that's not that what the iCurve tool is doing there. So perhaps  it's just writing our own tool for the stack or...  Even with an iCurve 3 itself, I've seen a lot weirder stuff on previous games in the  last year or two than I had seen in the past. Just stuff that wanting more stuff getting  locked at Magical Model and stuff that even...not to say about iCurve 2 but iCurve 3 because that's  a whole other monster. But I just wanted to throw that out there for anyone in the room.  So what's possible with that, I can repeat those tests easily. So if I have a script that  runs all the tests, it's not being healed to add another test and run the whole thing  in the past again because I can generate a mystery at every moment.  So any more questions? Okay, no questions. Thank you.  Thank you.
Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_0_107.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_108_203.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_204_311.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_312_899.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_900_2543.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_2544_2711.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_2712_3683.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_3684_4475.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_4476_6251.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_6252_7307.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_7308_8003.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_8004_10751.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_10752_11819.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_11820_16307.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_16308_19535.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_19536_23327.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_23328_24755.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_24756_25175.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_25176_27515.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_27516_29999.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_30000_33467.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_33468_33599.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_33600_34487.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_34488_34571.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_34572_34655.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_34656_39443.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_39444_39539.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_39540_39683.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_39684_41567.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_41568_44039.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_44040_48071.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_48072_48263.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_48264_48863.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_48864_49751.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_49752_51299.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_51300_53195.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_53196_53315.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_53316_55439.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_55440_58787.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_58788_61667.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_61668_63239.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_63240_63743.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_63744_63875.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_63876_66143.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_66144_72263.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_72264_75251.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_75252_76283.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_76284_77663.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_77664_77891.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_77892_79067.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_79068_82655.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_82656_83411.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_83412_84431.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_84432_84479.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_84480_85043.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_85044_85175.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_85176_85955.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_85956_86087.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_86088_86327.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_86328_86507.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_86508_86591.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_86592_86711.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_86712_86759.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_86760_86807.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_86808_86987.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_86988_87059.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_87060_87143.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_87144_87191.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_87192_87491.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_87492_87563.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_87564_87611.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_87612_87779.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_87780_87923.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_87924_93839.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_93840_93935.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_93936_93983.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_93984_94031.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_94032_94079.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_94080_94139.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_94140_95639.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_95640_95687.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_95688_96299.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_96300_96347.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_96348_96395.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_96396_96443.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_96444_96491.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_96492_96563.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_96564_96611.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_96612_98231.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_98232_98291.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_98292_98339.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_98340_98411.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_98412_98483.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_98484_98531.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_98532_99011.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_99012_99167.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_99168_99299.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_99300_99467.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_99468_99755.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_99756_100007.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_100008_100079.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_100080_107495.png Visualization_of_Regression_and_Performance_Know_when_something_went_wrong_s6rAXaHylFM_107496_107589.png