Automated firewall testing
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Provost, Kristof
We're all convinced that automated tests are a good idea. For some applications (e.g. grep, awk, cc, ...) this is very straightforward. Others are a lot harder to test, for example firewalls. Typically testing firewalls takes two to three hosts. One to send traffic, the firewall test host and one to receive traffic. This makes automated test orchestration complex and brittle. This in turn means that tests either don't get written, are difficult to write and/or suffer random failures unrelated to issues in the firewall itself. Virtualisation has made this all somewhat easier, but it's still fiddly and difficult to make robust. It's also slow. The new FreeBSD network stack virtualisation lets us build on the existing jails system to build test setups, execute tests and clean up in mere seconds, without any requirement for additional hardware, or even hardware virtualisation support. FreeBSD 12 will ship with network stack virtualisation (known as VIMAGE or vnet). This is an important feature for many applications, one of which is automated network stack and firewall testing. As of FreeBSD 12 PF fully support VIMAGE, allowing users to configure a firewall for each jail. This talk will introduce VIMAGE and show how it can be used to easily write firewall tests. If there's time a few interesting bugs and their test cases will also be discussed.
 Alright, talk, yes. Welcome. I'm going to be talking to you about automated firewall testing and to give you the entire point of the talk right up front as a spoiler, testing is good, you should all do more testing. Any questions? Excellent, okay we can go now. To do the typical shameless self promotion, I'm Kristof, I'm a FreeBSD developer. In FreeBSD I mostly try to maintain PF. PF is one of the many firewalls, well one of the three firewalls in FreeBSD. PF we borrowed from OpenBSD and we'll return it to them any day now. Professionally I tend to do a lot of embedded projects, quite a lot of them on that other Unix operating system nobody's ever heard of. I want to make it very clear that I'm not for sale, but I am for rent at very reasonable rates. Send me email. Onto more interesting things, PF is a packet filter. I trust we all have a vague idea of what firewalls do. They look at your packets, clock their tongues disapprovingly and then throw some of them away. Sometimes they let packets through, I mostly think that's usually a mistake. We imported it from OpenBSD and we did that a while ago. I actually forgot to look up when, but it's been many years since we did the last import, which means that there have been differences. The OpenBSD people keep changing their mind about what the syntax should be and we haven't taken any of those changes in yet. We do take occasional bug fixes. There's even one or two commits with my name on it in OpenBSD, which should carry you off that particular operating system. Now that I've said what we don't have in FreeBSD, some of the things we do have is VNAT, and I'll talk more about VNAT later. What we also have is multi-core capability. PF in FreeBSD tends to be quite a good bit faster than the one in OpenBSD. The OpenBSD people are working hard on unlocking their kernel. They are, I think, discovering as FreeBSD did in FreeBSD 5 that this is not easy and it's going to take a while, but they'll get there. On to the actual topic, testing. Why do we want automated testing? Well, obviously because I'm really, really, really lazy. So if I can get the computer to do it, I don't have to do it. That's clearly a win. Testing is a good idea because our users are very spoiled and they have this vague notion that if we give them software, it should actually work. Yeah, I know. They're very unreasonable, but what are you going to do? It's also nice that when we fix the bug, it doesn't spontaneously reemerge. Regressions are a thing that happens. I've got some examples later on. Actually, on the next slide, but that's later. It's also really nice when you're doing development. I recently did some work on PF Sync and it's really nice that you can run some tests and have an idea that I may have broken it, but at least the breakage will be subtle and it will take a while for people to notice. It doesn't explode immediately when you try to do this. I will confess that during development, I had quite a lot of episodes of, I think this will work and then you run the test and it just massively explodes. That's actually when you want your tests to fail as when you're developing rather than when you've delivered it to the customer and they're going to start using it. Some example regressions. There's a pattern here. I wonder if you can spot it. One of the things I did to PF was make it understand V6 fragmentation. I hate V6 fragmentation now. Long, long ago, that went in and that worked and it was wonderful and then somebody made the V6 forwarding path faster and broke PF's fragment handling. I didn't notice for nine months, which is a kind of painfully long period for something to be broken. That was before we had any sort of tests for PF. That would only happen when I manually built a test setup and tried this out. My weekends are boring, but they're not that boring. It did eventually end up being fixed and a little while later, actually just last year around August, I think, V6 fragment handling broke again. This time as a result of a security fix, there was an issue with fragment handling not just in FreeBSD but in Linux and in a bunch of other operating systems where somebody decided to try what would happen if I generate an evil sequence of fragmented packets. It turns out that a lot of operating systems had list operations and you could exploit this to make it take a really, really, really long time to do fragment lookups. That was a denial of service vector. Someone fixed this. That really should have looked up his name. It turns out that the fix had a bug in it. I'll show you the code and give you a moment to try to find the bug. The test picked it up immediately. It only took about two weeks between the bug actually being introduced and the fix going in, which it's two weeks that it was broken. That's not ideal, but two weeks is a lot better than nine months. It was also really annoying bug, which, you know, a Heisen bug. You try to debug it, you attach detrace probes to the function and all of a sudden the bug goes away. And you can't really tell all of your users, you know, just run this detrace probe and your code will work just fine. Also annoyingly, the PF tests failed over this. So everybody thought it's a PF bug, Crystal messed it up again. Nobody needs to look at this. So eventually I looked at it and discovered, you know, very confused because I had not actually touched the V6 fragment code in PF in months. Why is this broken? Turns out it wasn't PF that was broken. It was the rest of the network stack that was broken. So here's a little bit of code. These are the fragments that entered the stack. And has anyone spotted the bug yet? It is actually, I have actually made it slightly obvious. So basically what it does now is it hashes source and destination IP address and what else does it hash? Presumably it hashes an ID, oh yeah, my identification somewhere. And it uses this to divide up the fragments into buckets to make it much, much easier to find them rather than run through lists. And it turns out that this hash, so, you know, we allocate an array and then we hash over the entire size of the array. The array of 4 byte UIN32s is four times as big as it should be. So we hash not only the source and destination IP address and the ID, we also hash whatever random garbage happens to be on the stack at that moment. Which makes this hash unpredictable, which means that, you know, some parts of your fragmented packet will end up in one bucket, some parts will end up in another bucket, and so you never, ever manage to reassemble them. Unless, of course, you attach Dtrace and Dtrace leaves predictable garbage on the stack. So they all end up in the correct buckets and everything just works. Yeah, I, there was some amount of swearing when I finally figured out what went on there. And it is subtle enough that it's easy to miss. So that's the patch that went in is, you know, just make this array a bit smaller, use less stack real estate, hash only the bits that you actually mean to hash, and then everything is fine. So what do we want out of tests? Well, again, we're lazy, so we want them to be easy to write. We also want other people to be able to run them, not only so that they can run them and find their own bugs, but also so that they can write them. Because, you know, no matter how easy it is to write a test, if somebody else does it, I don't have to do the work. We also wanted to integrate with our existing continuous integration infrastructure. Free BSD actually has a fairly large number of tests, and we do actually run them regularly. So CI.FreeBSD.org if you're interested. So we run all of these tests, we build, you know, it's always nice if your software actually compiles. And then we actually test this. Some tests failed here. It's math related, and math is something computers do. I can't help you with that one. About 7,300 tests, and these run a couple times a day basically, you know, whenever it got done doing the last test run, it does another one. We now have someone who will actually follow up on those tests, so that's really nice. So I maintain the firewall, so the bit of code that I would like to test is the firewall. How do you go about testing firewall code? The typical approach that you might think first time around is, okay, we get a bunch of hardware. We get, you know, like three machines. We have machine A, machine B, and then in the middle we stick the machine where we run the firewall, and we have machine A's on the packet through the firewall machine, and machine B will go and look at, you know, did this packet make it through and molested? That's all fine. You know, you end up with having to need a fair bit of hardware, and then when you want to test more complicated setups like PFCync where you synchronize states of one firewall to another, well, you need two firewall machines, so you need four machines, and then you might want to test Karp to do failover, so you also need four or possibly even more machines. So this gets to be a bit of expensive. Not only that, you know, how would you configure this machine? Well, you SSH into it, except that we have just configured the firewall to drop all traffic. Okay, fine. You can attach a serial port, or you can use IPMI on a different interface, but, you know, your hardware budget starts to grow fairly significantly if you want to do this. Then you've got another issue that, you know, you really want to be running the latest and greatest code, so you can plausibly net boot this thing, which is really annoying if you want to test a firewall. Also, you know, you need a server to host this on, and your setup gets to become, you know, really, really complicated. You need to deal with, you know, this box panicked, so now we need to reboot it, and we can't log into it, so we need to do this over serial, but if you ever tried to do interactive scripting against something that's intended to be used by a human, it's hideously unreliable, so you need a remote power switch. It's to be, you know, very complicated, very expensive, slow, because machines take forever to boot. And, you know, where would this hardware live? The FreeBSD project has some friends who would, you know, lend them the hardware and who would host it for them, but then you only have the one setup. So how do other people write tests? So this is a very unsatisfactory sort of solution. It turns out that these days you can have virtual machines. Has anyone ever heard of this concept? FreeBSD even has its own hypervisor, Beehive, and this is the approach that was actually taken in a Google Summer of Code project. And it has a bunch of advantages because you don't need to buy all of this hardware, and a lot more people can build this setup. But you still got, you know, what if we block all the traffic? Okay, we need to use the serial interface. Your configuration gets to be a bit complicated. Another issue is that the way the tests are currently run is they run in a virtual machine, because that's a really convenient way to run your tests. But if we want our tests to start up a virtual machine, we have to have nested virtual machines, and that becomes a lot more interesting. It's also, and that's probably more an artifact of how we do our builds in our tests, but it is really, really annoying to build a virtual machine from your test setup, where you might not have the source code installed, you might not have already compiled it, and building that virtual machine, actually building it can be slow, never mind booting it. So this is better, but it is still an annoying setup. So what we've finally wound up with is VNet. And before we actually go into that, it might be useful to explain what VNet is. So how many of you know what VNet does? Oh, excellent, then you guys can explain it, and I don't have to. Basically, VNet is virtual IP stacks. This has jails, like containers, but we've had them for a lot longer. And you can associate an IP stack with this jail now, which means that you can, from inside the jail, you can set an IP address, you can run a DHCP client, and you can configure a firewall. This has been enabled by default in 12, the latest release, everybody should be running 12, it's awesome. PF supports this now, by which I mean I don't know of any way to make it panic, I'm sure there are some, and if you know of any, you know, file bugs and tell me about it. But it actually does mostly work now, which means you can start up a jail and give it its own firewall, which means we can just start the jail, throw some traffic at it, see what happens. So how do we do this? You know, I don't know how many of you have played with Linux containers, but there's a lot of stuff that goes on to actually make that container behave the way you want it to behave. Any number of moving parts and any number of things that have to be configured, and you know, your abstraction layers like Docker will do that for you, but there's still a lot of moving parts. On 3BSD, you know, I bet it's actually not that difficult. That's it. That's a jail with its own IP stack. You start the jail, you give it a name, I've named it Alcatraz because I think that sort of thing is funny. And then you tell it that you want it to have its own IP stack, so vNut, and you want the jail to stay to remain running even if no processes are running in it, so persist. That's it. Of course, I've lied to you slightly because that's not quite everything that there is to it because while this jail now has its own IP stack, it doesn't have any network interfaces. It has loopback, but yeah, you know, firewalling loopback is perhaps not the most productive thing you could be doing with yourself. I'm not going to tell you how to spend your Saturday, but, you know, consider picking up a different hobby. So what you need to do is we need to create, well, what we do in the tests is we create an ePair and an ePair is basically two network cards with a cable between them, virtual network cards. You can't actually use them to link up two different machines, but within one machine you can link up virtual machines or jails or stuff with them. So we create one, we assign an IP address to it, and then we tell the jail that, you know, you can have this network interface. And an additional really fun thing about the jails is they make it really easy to execute things inside the jail. So there we go, you know, executing the jail, if config ePair 0B set this IP address and up the jail. And after we've done that, we have one external interface on the host, host system. We have one internal inside the jail. And I should actually include the name of the jail here. So there's a typo in the slides, and then you can ping the jail. There you go. We're testing that the network stack works. I mean, ICMP eCos are not very exciting, but they do actually exercise a fair bit of your functionality. So why don't we take a look at the basic test? That's not the entire test. That's basically just, that's just the header. We use ATF that we have, you know, we noticed that NetBSD had left their front door unlocked one night. So we went in and took their testing framework. Quite entirely possible. I haven't actually looked at the new one yet because they keep locking their door now. So what do we do? You know, we declare a test case for v4. It's got a cleanup function. We'll see the cleanup function after we've done the really interesting things. You can set the description and, you know, this test actually wants to run as root. The Mami state of FreeBSD has decided that, you know, if you want to create network interfaces and start jails and configure the firewall, maybe we shouldn't let every single user on the system do this. Yeah, I know, I know, but that's how it is. So that's just setup code. You can forget this. It's not important. This is the actual test. And this is, it's a very basic test, but it is already a useful test. It tests that the firewall works and blocks packets. So what do we do? Some initialization codes. Everything the initialization code does is make sure that, you know, you've actually loaded the BF module. It is very hard to test the firewall if the code's not in the kernel. So if it's not loaded, this test will just be skipped. Next step, create an ePair. Set up an IP address on the ePair. So basically what we saw before is we want an interface in the host. We want an interface in the jail so that we can send traffic back and forth. Make a jail. There's a wrapper around this just to make it slightly easier to type. What it also does is it keeps track of the list of jails you've created so that in the cleanup, it will automatically destroy all of these jails. Quite nice to actually clean up after yourself. Set an IP address in the jail and then a sanity test check. Can we actually ping this jail now? So if things fail here, it's not actually a PF bug because PF is not enabled yet. But we might as well test that this works. So ATF check, you know, run this command and check that the exit status is zero. Ignore the output and then run a ping command, you know, one ping only, with a timeout of one second because if your echo request hasn't made it to a jail running on the same host within a second, something's probably wrong. Next step is we enable PF and PF defaults to allowing all of the traffic to go through and then we can still ping. And then when we tell PF to block all of the traffic, we can't ping anymore. Hence the exit status two. It won't work anymore. And that's a very basic test, of course. You know, the firewall should be able to make more fine-grained decisions on your packet, let's say, than, you know, allow all packets or deny all packets. But it's basic functionality test. The rest of this test actually does a little bit more, but this is all I can fit on the slide. But you can see how you can fairly trivially test that I can filter out only ICMP echoes and then I can still tell that into it or whatever. Cleanup codes, clean up, and then add the test case to the list of test cases. So if you run this using QA, which is another tool that we happen to spot lying in an unlocked garage from the SD, you can tell QA, you know, run this one particular test and it will run the test and you will see that the test took 1.2 seconds. 1.2 seconds of which it spent an entire second waiting for an ICMP reply that we knew would never turn up. So 0.2 seconds to start up a machine, a jail, configure the networking on it, configure the firewall, and tear it all down again. I think we've satisfied that, you know, these tests are quick. We've also satisfied that everyone can run this because you don't need anything. If you've got a FreeBSD system running, FreeBSD12 system or current, or even an 11 system, although there are a number of bugs still in Vimage in 11, so it's not enabled by default there. You could build your own, but if you want Vimage, you really want to be running 12. It also stores the test results so you can later on go and have a look at what was the output, what environment variables were set, what was the timing, all sorts of things like that. This was a very simple test. I've mentioned PF-Sync before, so let's take a look at a really complicated test. This is just half of the test, but this is a very basic test for PF-Sync. It synchronizes states from one firewall to another. So what do we need? Well, we need an interface over which they can sync. We need two jails that will have states, and we will just send our traffic from the host. So we create jail one and jail two. Yeah, the jail, you know, naming the jails after real life jails joke got a bit old, so they're one and two here. So what do we do? We set up IP addresses for the external interface, for the sync interface. We configure PF-Sync, so we tell it what device it should synchronize over. All this does is if you don't configure IP addresses in PF-Sync, it will just multicast. So if you have two machines connected over a straight up link cable, it will just magically work. You don't need to worry about it. Set that up, do the same thing on the other interface. So that's all just set up. What do we do next? Well, we turn on PF. We set some rules on both hosts. We set up the interfaces, and then we send a ping to it. Select the correct source interface just in case that would go wrong. Give PF-Sync some time to actually synchronize the states because this is not an instant process. And then we go and look not at the jail that saw the traffic, but at the other one, do you have the corresponding state for this? So we just grab and go look, does this state exist? If it does not exist, we arrow out. If it does exist, everything works, and it's awesome. You had a question? So the question is, have I tried to use TCP dump to see if it's arriving? I have not. Mostly, what I've tried to do is make it really easy on myself to actually test this. So rather than look at, do I actually get this packet, we check, do we get the effects of this packet having been delivered? So typically, things like ICMP echoes. I don't check that the reply arrives. I check, is ping happy that the reply arrived? So theoretically, that does mean that you could have bugs like the firewall corrupts the payload of an ICMP echo reply packet and ping doesn't check for this corruption or expect this corruption. That is a bug that you wouldn't notice in this setup. So that is arguably a weakness of this setup is that you're not actually doing formal testing that is the TCP or IP output actually standards compliant. All we're testing is, is FreeBSD itself happy with the packets it sees? I think that's a price worth paying because doing the full formal verification is much, much harder and it turns out that, you know, if FreeBSD does TCP wrong, we're going to notice. If nothing else, the Netflix people will shout at us. Well, they won't shout, they're actually nice people, but, you know, they will tell us. Now that I've all got you all fired up for these tests, where can you find them? The source code for these tests lives in user source assuming you've installed your source to user source. Test says net pfill pf which mirrors the structure of where the code for pf lives. They get installed to user's test says net pfill pf which again mirrors the structure of where they're installed. So you can, if you want to run them yourself, you might want to run some, might want to install some tools. QR to actually run the tests. And I never know how to pronounce this. Skape. It's a Python tool that lets you generate and analyze packets. So there are some tests where I actually deliberately create very specifically formatted packets and actually go look at them. Some of the fragment, fragmentation tests just do this where you can deliberately create corrupted packets for instance, which might also be a really interesting test to do. If I knew of any, you know, malformed packets that cause panics for instance, that is something I would definitely do is create this packet to attempt to provoke this panic. You want to load, well pf sync because a pf sync module depends on pf so it will implicitly load that one. And then you can go to that directory and as root run qr test and it will run through all of the tests for pf. Takes, I forget about 30 seconds. We don't have that many of them yet but it turns out that having a couple of tests already gives you a lot of value. So you don't need to think that, you know, it's a firewall with hundreds of features so I need to write thousands of tests. Ideally you want to have thousands of tests but it turns out that just having, you know, five of them already tests a lot of the functionality of pf and you can get a lot of, you know, fall out accidental tests along the way. For instance, this ipv6 fragmentation bug, well the reassembly bug, we didn't have code to specifically test reassembly. Pf just tested this accidentally while it was testing some, well something related but not quite that code. So you get a lot of value out of relatively little investment for tests. At this point, you know, I would appeal to authority and get some sort of really profound quote from someone to reinforce my point. I didn't find one because I was lazy and I didn't look for one. So the quote that you're getting is, you know, the tests are good and you should write more tests and it's for me just now. So you know, it is a very well sourced quote. I hope, you know, to continue to try to persuade you that tests are a good idea and that you should try to write more of them. What is in it for you? Well, you can prototype your own setups. You know, if you want to play with Carport or PFCinc or PF and you don't happen to have a box lying around to run this on, just spool up a jail and test this. And if you write them as a test, you can keep running this and you can be sure that this won't actually break. So if you have a very specific use case that relies on a feature that you're not sure other people are using, you know, send me a test case, we'll include it and you can be sure that the next time somebody breaks it, it will get fixed rather than you noticing, you know, when you try to upgrade to a release two years after somebody broke the feature for you. Another case and this is what I really want to focus on is when you report a bug to PF, I have to try to reproduce it. I have to try to work out what your test setup is and quite often in most cases, in fact, I spend more time trying to work out what your setup is, how it breaks than actually debugging the problem. So if you give me a test case, you've already done more than half of the work. Not only that, if you give me the test case, when I fix the bug, I would like to have a test for it. Well, you've given me a test case. Most of the work is already done. So if you have a pet bug that's been really, really been annoying you and you want it fixed, that's the best way. Actually, that's not true. That's the second best way. The best way is to pay me. No, no, that's not a joke. You know, I'm very money motivated. If you pay me to do work for you, I will do work for you. But if you don't want to pay me, you should, but if you don't want to pay me, write the test. And I promise you, well, I'm not going to promise you that I will absolutely fix your bug for you, but it does vastly increase the chances of it getting fixed. I've had someone report a bug to me in the NAT system and it was a very strange bug and I had no idea of where it was or how to fix it, right up to the point where he gave me a trivial test case for it. Not quite in this format yet, but he gave me a configuration file and a little shell script and, you know, you do this and I see these problems and then I could suddenly reproduce the problem and it only took a day or two to fix. For a, you know, a problem that started out as a description of, you know, I have this very large NAT setup, you know, very large. I, you know, I run a university, so a lot of students, a lot of traffic flowing through this and every once in a while without any obvious cause, the machine will just stop forwarding packets and then it will sit in that state for a few minutes and then it will start forwarding packets again. Until we have the, you know, the active description of this is how you trigger it and then it only took a day to, well, it took barely any time at all to figure out where the problem was and fix it. So write tests, illustrate your, your problem, make it easy for other people to fix the bug for you. Or, you know, you can also fix the bug yourself and that's awesome, but it turns out that the reviewing code means that I need to understand what the problem is, which means that I need to know what your setup is and what the bug is, which is also accomplished by a test case. If you want to make it even more likely that your bug gets fixed, write the test case, write the patch and submit that. But, you know, whatever you do, test case, test case, test case. If you don't want to be the first one to do this, you won't be. Olivier Couchard-Labe was, is one of our, one of the contributors to the FreeBSD project and he's really, really good at benchmarking things. And for his previous employer, he really cared about IPsec and it kept breaking. He was really sick of this. So he wrote tests for it and now there are tests and IPsec isn't broken. And if somebody does break it, Lee Wen will shout at them. Well, that's a lie. Lee Wen's a really nice guy. He won't shout, but he will tell people that it's broken and that they should fix it. And this also means that, you know, because we run these tests a couple of times a day, if you commit something that breaks IPsec today, we're going to notice tomorrow or on Monday and you're going to have a bug assigned to you on Monday or on Tuesday. So it will be very fresh in your memory. You know, before I belabor the point any more, does anyone have any questions? Have I been that clear? Oh, yes, go ahead. Well, thinking of PF as a filter, why don't test PF as a filter on the low level? I mean, when you, for example, use PING, there's a lot of things that can go wrong when the PING doesn't exit the code with the exit code. And you're not quite sure if, as you mentioned before, that PF is broken. Yes. So the question is, you know, when we test PF based on the PING tool, there is a lot of codes that needs to run. So why don't we simplify this down so that we're only testing PF? That is a valid approach to take in testing. I think the advantage of not doing it that way, of doing the, you know, we just use the PING tool to send the packet, is two-fold. The first is that it's much easier. We don't have to write any code to send PING messages. We don't have to write any code to receive them, to validate them. That code is already there. So we can just use it. It makes it much easier to write the test. Another side effect of this, and you can argue if that's good or bad, is that you test much more code. So that's the incidental testing I was talking about, like we had with the V6 reassembly code. You are incidental to this test. You are also testing PING. And that's an upside because you test more of the code and the functionality of the system. It's also a downside because when it breaks, it's not immediately obvious where the breakage is. So if somebody breaks the PING tool, changes the return code, changes it, so that it no longer sends ICMP echo requests, or it thinks that they're corrupted when they're not corrupted, this test is going to fail and is going to look like PF broke. That's arguably a downside, but hey, at least we know that something's broken. And especially if you run these tests fairly frequently, you will typically not be looking at thousands upon thousands of commits that could have caused a problem. So if you look and you see, today it was working, tomorrow it was broken. In this time span we had 50 commits, 40 of them touched unrelated driver code, five of them touched the IP stack in some way, one of them touched the PING tool. There's not too many things that could have caused the breakage. It's a valid point, but you can argue it both ways. Yes? So the question is, do we have test cases for the other firewalls in FreeBSD? So FreeBSD ships out of the box with three firewalls, IPF, IPFW and PF. I'm not going to go into the reasons for this because honestly I don't know if I understand and agree with all of them, but the fact is we have three firewalls. We currently don't have tests for the other firewalls. Right now PF is the only one that has these tests. It would be trivial to look at the PF tests and go, okay, now I'm going to rewrite them or I'm going to write similar tests for IPFW or IPF. I believe both of them also support VNet. It does actually take some changes to the firewall to make this work, but to make VNet work, not to make the test work. The tests do assume that you have VNet. So we don't have them right now. I actually just this morning posted an idea for a Google Summer Code project. So if there are any students who are interested in working on firewalls and testing, there is a project, write more tests. Ideally it should be possible to write a test so that it works on all three firewalls. There is some functionality like PF sync that is common. That is, sorry, is unique to PF, so you would have to write a test specifically for PF. But for things like, you know, drop this packet, allow this packet, keep state, you could write tests that would work on all three. We don't have them right now. It would be awesome to have them. Any other questions? Okay, thank you very much for your attention. I want to see the back of your hand. So the question is, do the tests include any randomness? Right now they do not. Again, randomness in tests is something you can argue both ways. So it's good because fast testing is great because you can trip over unexpected bugs. It's bad because you can't necessarily reproduce your results. Right now there is no randomness in it or no deliberate randomness anyway.
 Alright, talk, yes. Welcome. I'm going to be talking to you about automated firewall  testing and to give you the entire point of the talk right up front as a spoiler, testing  is good, you should all do more testing. Any questions? Excellent, okay we can go now.  To do the typical shameless self promotion, I'm Kristof, I'm a FreeBSD developer. In FreeBSD  I mostly try to maintain PF. PF is one of the many firewalls, well one of the three firewalls  in FreeBSD. PF we borrowed from OpenBSD and we'll return it to them any day now. Professionally  I tend to do a lot of embedded projects, quite a lot of them on that other Unix operating  system nobody's ever heard of. I want to make it very clear that I'm not for sale, but I  am for rent at very reasonable rates. Send me email. Onto more interesting things, PF  is a packet filter. I trust we all have a vague idea of what firewalls do. They look  at your packets, clock their tongues disapprovingly and then throw some of them away. Sometimes  they let packets through, I mostly think that's usually a mistake. We imported it from OpenBSD  and we did that a while ago. I actually forgot to look up when, but it's been many years  since we did the last import, which means that there have been differences. The OpenBSD  people keep changing their mind about what the syntax should be and we haven't taken  any of those changes in yet. We do take occasional bug fixes. There's even one or two commits  with my name on it in OpenBSD, which should carry you off that particular operating system.  Now that I've said what we don't have in FreeBSD, some of the things we do have is VNAT,  and I'll talk more about VNAT later. What we also have is multi-core capability. PF  in FreeBSD tends to be quite a good bit faster than the one in OpenBSD. The OpenBSD people  are working hard on unlocking their kernel. They are, I think, discovering as FreeBSD  did in FreeBSD 5 that this is not easy and it's going to take a while, but they'll get  there. On to the actual topic, testing. Why do we want automated testing? Well, obviously  because I'm really, really, really lazy. So if I can get the computer to do it, I don't  have to do it. That's clearly a win. Testing is a good idea because our users are very  spoiled and they have this vague notion that if we give them software, it should actually  work. Yeah, I know. They're very unreasonable, but what are you going to do? It's also nice  that when we fix the bug, it doesn't spontaneously reemerge. Regressions are a thing that happens.  I've got some examples later on. Actually, on the next slide, but that's later. It's  also really nice when you're doing development. I recently did some work on PF Sync and it's  really nice that you can run some tests and have an idea that I may have broken it, but  at least the breakage will be subtle and it will take a while for people to notice. It  doesn't explode immediately when you try to do this. I will confess that during development,  I had quite a lot of episodes of, I think this will work and then you run the test and  it just massively explodes. That's actually when you want your tests to fail as when you're  developing rather than when you've delivered it to the customer and they're going to start  using it. Some example regressions. There's a pattern here. I wonder if you can spot it.  One of the things I did to PF was make it understand V6 fragmentation. I hate V6 fragmentation  now. Long, long ago, that went in and that worked and it was wonderful and then somebody  made the V6 forwarding path faster and broke PF's fragment handling. I didn't notice for  nine months, which is a kind of painfully long period for something to be broken. That  was before we had any sort of tests for PF. That would only happen when I manually built  a test setup and tried this out. My weekends are boring, but they're not that boring. It  did eventually end up being fixed and a little while later, actually just last year around  August, I think, V6 fragment handling broke again. This time as a result of a security  fix, there was an issue with fragment handling not just in FreeBSD but in Linux and in a  bunch of other operating systems where somebody decided to try what would happen if I generate  an evil sequence of fragmented packets. It turns out that a lot of operating systems  had list operations and you could exploit this to make it take a really, really, really  long time to do fragment lookups. That was a denial of service vector. Someone fixed  this. That really should have looked up his name. It turns out that the fix had a bug  in it. I'll show you the code and give you a moment to try to find the bug. The test  picked it up immediately. It only took about two weeks between the bug actually being introduced  and the fix going in, which it's two weeks that it was broken. That's not ideal, but  two weeks is a lot better than nine months. It was also really annoying bug, which, you  know, a Heisen bug. You try to debug it, you attach detrace probes to the function and  all of a sudden the bug goes away. And you can't really tell all of your users, you  know, just run this detrace probe and your code will work just fine. Also annoyingly,  the PF tests failed over this. So everybody thought it's a PF bug, Crystal messed it  up again. Nobody needs to look at this. So eventually I looked at it and discovered,  you know, very confused because I had not actually touched the V6 fragment code in PF  in months. Why is this broken? Turns out it wasn't PF that was broken. It was the rest  of the network stack that was broken. So here's a little bit of code. These are the  fragments that entered the stack. And has anyone spotted the bug yet? It is actually,  I have actually made it slightly obvious. So basically what it does now is it hashes  source and destination IP address and what else does it hash? Presumably it hashes an  ID, oh yeah, my identification somewhere. And it uses this to divide up the fragments  into buckets to make it much, much easier to find them rather than run through lists.  And it turns out that this hash, so, you know, we allocate an array and then we hash over  the entire size of the array. The array of 4 byte UIN32s is four times as big as it  should be. So we hash not only the source and destination IP address and the ID, we  also hash whatever random garbage happens to be on the stack at that moment. Which makes  this hash unpredictable, which means that, you know, some parts of your fragmented packet  will end up in one bucket, some parts will end up in another bucket, and so you never,  ever manage to reassemble them. Unless, of course, you attach Dtrace and Dtrace leaves  predictable garbage on the stack. So they all end up in the correct buckets and everything  just works. Yeah, I, there was some amount of swearing when I finally figured out what  went on there. And it is subtle enough that it's easy to miss. So that's the patch that  went in is, you know, just make this array a bit smaller, use less stack real estate,  hash only the bits that you actually mean to hash, and then everything is fine. So what  do we want out of tests? Well, again, we're lazy, so we want them to be easy to write.  We also want other people to be able to run them, not only so that they can run them and  find their own bugs, but also so that they can write them. Because, you know, no matter  how easy it is to write a test, if somebody else does it, I don't have to do the work.  We also wanted to integrate with our existing continuous integration infrastructure. Free  BSD actually has a fairly large number of tests, and we do actually run them regularly.  So CI.FreeBSD.org if you're interested. So we run all of these tests, we build, you  know, it's always nice if your software actually compiles. And then we actually test this.  Some tests failed here. It's math related, and math is something computers do. I can't  help you with that one. About 7,300 tests, and these run a couple times a day basically,  you know, whenever it got done doing the last test run, it does another one. We now have  someone who will actually follow up on those tests, so that's really nice. So I maintain  the firewall, so the bit of code that I would like to test is the firewall. How do you go  about testing firewall code? The typical approach that you might think first time around is,  okay, we get a bunch of hardware. We get, you know, like three machines. We have machine  A, machine B, and then in the middle we stick the machine where we run the firewall, and  we have machine A's on the packet through the firewall machine, and machine B will go  and look at, you know, did this packet make it through and molested? That's all fine.  You know, you end up with having to need a fair bit of hardware, and then when you want  to test more complicated setups like PFCync where you synchronize states of one firewall  to another, well, you need two firewall machines, so you need four machines, and then you might  want to test Karp to do failover, so you also need four or possibly even more machines.  So this gets to be a bit of expensive. Not only that, you know, how would you configure  this machine? Well, you SSH into it, except that we have just configured the firewall  to drop all traffic. Okay, fine. You can attach a serial port, or you can use IPMI on a different  interface, but, you know, your hardware budget starts to grow fairly significantly if you  want to do this. Then you've got another issue that, you know, you really want to be running  the latest and greatest code, so you can plausibly net boot this thing, which is really annoying  if you want to test a firewall. Also, you know, you need a server to host this on, and  your setup gets to become, you know, really, really complicated. You need to deal with,  you know, this box panicked, so now we need to reboot it, and we can't log into it, so  we need to do this over serial, but if you ever tried to do interactive scripting against  something that's intended to be used by a human, it's hideously unreliable, so you need  a remote power switch. It's to be, you know, very complicated, very expensive, slow, because  machines take forever to boot. And, you know, where would this hardware live? The FreeBSD  project has some friends who would, you know, lend them the hardware and who would host  it for them, but then you only have the one setup. So how do other people write tests?  So this is a very unsatisfactory sort of solution. It turns out that these days you can have virtual  machines. Has anyone ever heard of this concept? FreeBSD even has its own hypervisor, Beehive,  and this is the approach that was actually taken in a Google Summer of Code project.  And it has a bunch of advantages because you don't need to buy all of this hardware, and  a lot more people can build this setup. But you still got, you know, what if we block  all the traffic? Okay, we need to use the serial interface. Your configuration gets  to be a bit complicated. Another issue is that the way the tests are currently run is  they run in a virtual machine, because that's a really convenient way to run your tests.  But if we want our tests to start up a virtual machine, we have to have nested virtual machines,  and that becomes a lot more interesting. It's also, and that's probably more an artifact  of how we do our builds in our tests, but it is really, really annoying to build a virtual  machine from your test setup, where you might not have the source code installed, you might  not have already compiled it, and building that virtual machine, actually building it  can be slow, never mind booting it. So this is better, but it is still an annoying setup.  So what we've finally wound up with is VNet. And before we actually go into that, it might  be useful to explain what VNet is. So how many of you know what VNet does? Oh, excellent,  then you guys can explain it, and I don't have to. Basically, VNet is virtual IP stacks.  This has jails, like containers, but we've had them for a lot longer. And you can associate  an IP stack with this jail now, which means that you can, from inside the jail, you can  set an IP address, you can run a DHCP client, and you can configure a firewall. This has  been enabled by default in 12, the latest release, everybody should be running 12, it's  awesome. PF supports this now, by which I mean I don't know of any way to make it panic,  I'm sure there are some, and if you know of any, you know, file bugs and tell me about  it. But it actually does mostly work now, which means you can start up a jail and give  it its own firewall, which means we can just start the jail, throw some traffic at it,  see what happens. So how do we do this? You know, I don't know how many of you have played  with Linux containers, but there's a lot of stuff that goes on to actually make that  container behave the way you want it to behave. Any number of moving parts and any number  of things that have to be configured, and you know, your abstraction layers like Docker  will do that for you, but there's still a lot of moving parts. On 3BSD, you know, I  bet it's actually not that difficult. That's it. That's a jail with its own IP stack. You  start the jail, you give it a name, I've named it Alcatraz because I think that sort of thing  is funny. And then you tell it that you want it to have its own IP stack, so vNut, and  you want the jail to stay to remain running even if no processes are running in it, so  persist. That's it. Of course, I've lied to you slightly because that's not quite everything  that there is to it because while this jail now has its own IP stack, it doesn't have any  network interfaces. It has loopback, but yeah, you know, firewalling loopback is perhaps  not the most productive thing you could be doing with yourself. I'm not going to tell  you how to spend your Saturday, but, you know, consider picking up a different hobby. So  what you need to do is we need to create, well, what we do in the tests is we create  an ePair and an ePair is basically two network cards with a cable between them, virtual network  cards. You can't actually use them to link up two different machines, but within one  machine you can link up virtual machines or jails or stuff with them. So we create one,  we assign an IP address to it, and then we tell the jail that, you know, you can have  this network interface. And an additional really fun thing about the jails is they make  it really easy to execute things inside the jail. So there we go, you know, executing  the jail, if config ePair 0B set this IP address and up the jail. And after we've done that,  we have one external interface on the host, host system. We have one internal inside the  jail. And I should actually include the name of the jail here. So there's a typo in the  slides, and then you can ping the jail. There you go. We're testing that the network stack  works. I mean, ICMP eCos are not very exciting, but they do actually exercise a fair bit of  your functionality. So why don't we take a look at the basic test? That's not the entire  test. That's basically just, that's just the header. We use ATF that we have, you know,  we noticed that NetBSD had left their front door unlocked one night. So we went in and  took their testing framework. Quite entirely possible. I haven't actually looked at the  new one yet because they keep locking their door now. So what do we do? You know, we declare  a test case for v4. It's got a cleanup function. We'll see the cleanup function after we've  done the really interesting things. You can set the description and, you know, this test  actually wants to run as root. The Mami state of FreeBSD has decided that, you know, if  you want to create network interfaces and start jails and configure the firewall, maybe  we shouldn't let every single user on the system do this. Yeah, I know, I know, but  that's how it is. So that's just setup code. You can forget this. It's not important. This  is the actual test. And this is, it's a very basic test, but it is already a useful test.  It tests that the firewall works and blocks packets. So what do we do? Some initialization  codes. Everything the initialization code does is make sure that, you know, you've actually  loaded the BF module. It is very hard to test the firewall if the code's not in the kernel.  So if it's not loaded, this test will just be skipped. Next step, create an ePair. Set  up an IP address on the ePair. So basically what we saw before is we want an interface  in the host. We want an interface in the jail so that we can send traffic back and forth.  Make a jail. There's a wrapper around this just to make it slightly easier to type. What  it also does is it keeps track of the list of jails you've created so that in the cleanup,  it will automatically destroy all of these jails. Quite nice to actually clean up after  yourself. Set an IP address in the jail and then a sanity test check. Can we actually  ping this jail now? So if things fail here, it's not actually a PF bug because PF is not  enabled yet. But we might as well test that this works. So ATF check, you know, run this  command and check that the exit status is zero. Ignore the output and then run a ping  command, you know, one ping only, with a timeout of one second because if your echo request  hasn't made it to a jail running on the same host within a second, something's probably  wrong. Next step is we enable PF and PF defaults to allowing all of the traffic to go through  and then we can still ping. And then when we tell PF to block all of the traffic, we  can't ping anymore. Hence the exit status two. It won't work anymore. And that's a very  basic test, of course. You know, the firewall should be able to make more fine-grained decisions  on your packet, let's say, than, you know, allow all packets or deny all packets. But  it's basic functionality test. The rest of this test actually does a little bit more,  but this is all I can fit on the slide. But you can see how you can fairly trivially test  that I can filter out only ICMP echoes and then I can still tell that into it or whatever.  Cleanup codes, clean up, and then add the test case to the list of test cases. So if  you run this using QA, which is another tool that we happen to spot lying in an unlocked  garage from the SD, you can tell QA, you know, run this one particular test and it will run  the test and you will see that the test took 1.2 seconds. 1.2 seconds of which it spent  an entire second waiting for an ICMP reply that we knew would never turn up. So 0.2 seconds  to start up a machine, a jail, configure the networking on it, configure the firewall,  and tear it all down again. I think we've satisfied that, you know, these tests are  quick. We've also satisfied that everyone can run this because you don't need anything.  If you've got a FreeBSD system running, FreeBSD12 system or current, or even an 11 system,  although there are a number of bugs still in Vimage in 11, so it's not enabled by default  there. You could build your own, but if you want Vimage, you really want to be running  12. It also stores the test results so you can later on go and have a look at what was  the output, what environment variables were set, what was the timing, all sorts of things  like that. This was a very simple test. I've mentioned  PF-Sync before, so let's take a look at a really complicated test. This is just half  of the test, but this is a very basic test for PF-Sync. It synchronizes states from one  firewall to another. So what do we need? Well, we need an interface over which they can  sync. We need two jails that will have states, and we will just send our traffic from the  host. So we create jail one and jail two. Yeah, the jail, you know, naming the jails after  real life jails joke got a bit old, so they're one and two here. So what do we do? We set  up IP addresses for the external interface, for the sync interface. We configure PF-Sync,  so we tell it what device it should synchronize over. All this does is if you don't configure  IP addresses in PF-Sync, it will just multicast. So if you have two machines connected over  a straight up link cable, it will just magically work. You don't need to worry about it. Set  that up, do the same thing on the other interface. So that's all just set up. What do we do next?  Well, we turn on PF. We set some rules on both hosts. We set up the interfaces, and then  we send a ping to it. Select the correct source interface just in case that would go wrong.  Give PF-Sync some time to actually synchronize the states because this is not an instant  process. And then we go and look not at the jail that saw the traffic, but at the other  one, do you have the corresponding state for this? So we just grab and go look, does this  state exist? If it does not exist, we arrow out. If it does exist, everything works, and  it's awesome. You had a question?  So the question is, have I tried to use TCP dump to see if it's arriving? I have not.  Mostly, what I've tried to do is make it really easy on myself to actually test this. So rather  than look at, do I actually get this packet, we check, do we get the effects of this packet  having been delivered? So typically, things like ICMP echoes. I don't check that the reply  arrives. I check, is ping happy that the reply arrived? So theoretically, that does mean  that you could have bugs like the firewall corrupts the payload of an ICMP echo reply  packet and ping doesn't check for this corruption or expect this corruption. That is a bug that  you wouldn't notice in this setup. So that is arguably a weakness of this setup is that  you're not actually doing formal testing that is the TCP or IP output actually standards  compliant. All we're testing is, is FreeBSD itself happy with the packets it sees? I think  that's a price worth paying because doing the full formal verification is much, much harder  and it turns out that, you know, if FreeBSD does TCP wrong, we're going to notice. If  nothing else, the Netflix people will shout at us. Well, they won't shout, they're actually  nice people, but, you know, they will tell us. Now that I've all got you all fired up  for these tests, where can you find them? The source code for these tests lives in user  source assuming you've installed your source to user source. Test says net pfill pf which  mirrors the structure of where the code for pf lives. They get installed to user's test  says net pfill pf which again mirrors the structure of where they're installed. So you  can, if you want to run them yourself, you might want to run some, might want to install  some tools. QR to actually run the tests. And I never know how to pronounce this. Skape.  It's a Python tool that lets you generate and analyze packets. So there are some tests  where I actually deliberately create very specifically formatted packets and actually  go look at them. Some of the fragment, fragmentation tests just do this where you can deliberately  create corrupted packets for instance, which might also be a really interesting test to  do. If I knew of any, you know, malformed packets that cause panics for instance, that  is something I would definitely do is create this packet to attempt to provoke this panic.  You want to load, well pf sync because a pf sync module depends on pf so it will implicitly  load that one. And then you can go to that directory and as root run qr test and it will  run through all of the tests for pf. Takes, I forget about 30 seconds. We don't have that  many of them yet but it turns out that having a couple of tests already gives you a lot  of value. So you don't need to think that, you know, it's a firewall with hundreds of  features so I need to write thousands of tests. Ideally you want to have thousands of tests  but it turns out that just having, you know, five of them already tests a lot of the functionality  of pf and you can get a lot of, you know, fall out accidental tests along the way. For  instance, this ipv6 fragmentation bug, well the reassembly bug, we didn't have code to  specifically test reassembly. Pf just tested this accidentally while it was testing some,  well something related but not quite that code. So you get a lot of value out of relatively  little investment for tests. At this point, you know, I would appeal to authority and  get some sort of really profound quote from someone to reinforce my point. I didn't find  one because I was lazy and I didn't look for one. So the quote that you're getting is,  you know, the tests are good and you should write more tests and it's for me just now.  So you know, it is a very well sourced quote. I hope, you know, to continue to try to persuade  you that tests are a good idea and that you should try to write more of them. What is  in it for you? Well, you can prototype your own setups. You know, if you want to play  with Carport or PFCinc or PF and you don't happen to have a box lying around to run this  on, just spool up a jail and test this. And if you write them as a test, you can keep running  this and you can be sure that this won't actually break. So if you have a very specific use case  that relies on a feature that you're not sure other people are using, you know, send me  a test case, we'll include it and you can be sure that the next time somebody breaks  it, it will get fixed rather than you noticing, you know, when you try to upgrade to a release  two years after somebody broke the feature for you. Another case and this is what I really  want to focus on is when you report a bug to PF, I have to try to reproduce it. I have  to try to work out what your test setup is and quite often in most cases, in fact, I  spend more time trying to work out what your setup is, how it breaks than actually debugging  the problem. So if you give me a test case, you've already done more than half of the  work. Not only that, if you give me the test case, when I fix the bug, I would like to  have a test for it. Well, you've given me a test case. Most of the work is already done.  So if you have a pet bug that's been really, really been annoying you and you want it fixed,  that's the best way. Actually, that's not true. That's the second best way. The best  way is to pay me. No, no, that's not a joke. You know, I'm very money motivated. If you  pay me to do work for you, I will do work for you. But if you don't want to pay me,  you should, but if you don't want to pay me, write the test. And I promise you, well, I'm  not going to promise you that I will absolutely fix your bug for you, but it does vastly increase  the chances of it getting fixed. I've had someone report a bug to me in the NAT system  and it was a very strange bug and I had no idea of where it was or how to fix it, right  up to the point where he gave me a trivial test case for it. Not quite in this format  yet, but he gave me a configuration file and a little shell script and, you know, you do  this and I see these problems and then I could suddenly reproduce the problem and it only  took a day or two to fix. For a, you know, a problem that started out as a description  of, you know, I have this very large NAT setup, you know, very large. I, you know, I run a  university, so a lot of students, a lot of traffic flowing through this and every once  in a while without any obvious cause, the machine will just stop forwarding packets  and then it will sit in that state for a few minutes and then it will start forwarding  packets again. Until we have the, you know, the active description of this is how you  trigger it and then it only took a day to, well, it took barely any time at all to figure  out where the problem was and fix it. So write tests, illustrate your, your problem, make  it easy for other people to fix the bug for you. Or, you know, you can also fix the bug  yourself and that's awesome, but it turns out that the reviewing code means that I  need to understand what the problem is, which means that I need to know what your setup  is and what the bug is, which is also accomplished by a test case. If you want to make it even  more likely that your bug gets fixed, write the test case, write the patch and submit  that. But, you know, whatever you do, test case, test case, test case. If you don't want  to be the first one to do this, you won't be. Olivier Couchard-Labe was, is one of our,  one of the contributors to the FreeBSD project and he's really, really good at benchmarking  things. And for his previous employer, he really cared about IPsec and it kept breaking.  He was really sick of this. So he wrote tests for it and now there are tests and IPsec isn't  broken. And if somebody does break it, Lee Wen will shout at them. Well, that's a lie.  Lee Wen's a really nice guy. He won't shout, but he will tell people that it's broken and  that they should fix it. And this also means that, you know, because we run these tests  a couple of times a day, if you commit something that breaks IPsec today, we're going to notice  tomorrow or on Monday and you're going to have a bug assigned to you on Monday or on  Tuesday. So it will be very fresh in your memory. You know, before I belabor the point  any more, does anyone have any questions? Have I been that clear? Oh, yes, go ahead.  Well, thinking of PF as a filter, why don't test PF as a filter on the low level?  I mean, when you, for example, use PING, there's a lot of things that can go wrong when the PING doesn't exit the code with the exit code.  And you're not quite sure if, as you mentioned before, that PF is broken.  Yes. So the question is, you know, when we test PF based on the PING tool, there is a lot of codes that needs to run.  So why don't we simplify this down so that we're only testing PF?  That is a valid approach to take in testing.  I think the advantage of not doing it that way, of doing the, you know, we just use the PING tool to send the packet, is two-fold.  The first is that it's much easier. We don't have to write any code to send PING messages.  We don't have to write any code to receive them, to validate them. That code is already there.  So we can just use it. It makes it much easier to write the test.  Another side effect of this, and you can argue if that's good or bad, is that you test much more code.  So that's the incidental testing I was talking about, like we had with the V6 reassembly code.  You are incidental to this test. You are also testing PING.  And that's an upside because you test more of the code and the functionality of the system.  It's also a downside because when it breaks, it's not immediately obvious where the breakage is.  So if somebody breaks the PING tool, changes the return code, changes it, so that it no longer sends ICMP echo requests,  or it thinks that they're corrupted when they're not corrupted, this test is going to fail and is going to look like PF broke.  That's arguably a downside, but hey, at least we know that something's broken.  And especially if you run these tests fairly frequently, you will typically not be looking at thousands upon thousands of commits  that could have caused a problem. So if you look and you see, today it was working, tomorrow it was broken.  In this time span we had 50 commits, 40 of them touched unrelated driver code,  five of them touched the IP stack in some way, one of them touched the PING tool.  There's not too many things that could have caused the breakage.  It's a valid point, but you can argue it both ways. Yes?  So the question is, do we have test cases for the other firewalls in FreeBSD?  So FreeBSD ships out of the box with three firewalls, IPF, IPFW and PF.  I'm not going to go into the reasons for this because honestly I don't know if I understand and agree with all of them,  but the fact is we have three firewalls. We currently don't have tests for the other firewalls.  Right now PF is the only one that has these tests. It would be trivial to look at the PF tests and go,  okay, now I'm going to rewrite them or I'm going to write similar tests for IPFW or IPF.  I believe both of them also support VNet. It does actually take some changes to the firewall to make this work,  but to make VNet work, not to make the test work. The tests do assume that you have VNet.  So we don't have them right now. I actually just this morning posted an idea for a Google Summer Code project.  So if there are any students who are interested in working on firewalls and testing, there is a project,  write more tests. Ideally it should be possible to write a test so that it works on all three firewalls.  There is some functionality like PF sync that is common. That is, sorry, is unique to PF,  so you would have to write a test specifically for PF. But for things like, you know,  drop this packet, allow this packet, keep state, you could write tests that would work on all three.  We don't have them right now. It would be awesome to have them. Any other questions?  Okay, thank you very much for your attention.  I want to see the back of your hand.  So the question is, do the tests include any randomness? Right now they do not.  Again, randomness in tests is something you can argue both ways. So it's good because fast testing is great  because you can trip over unexpected bugs. It's bad because you can't necessarily reproduce your results.  Right now there is no randomness in it or no deliberate randomness anyway.
Automated_firewall_testing_RYtwx7Jj8aE_0_143.png Automated_firewall_testing_RYtwx7Jj8aE_144_647.png Automated_firewall_testing_RYtwx7Jj8aE_648_1991.png Automated_firewall_testing_RYtwx7Jj8aE_1992_14363.png Automated_firewall_testing_RYtwx7Jj8aE_14364_14615.png Automated_firewall_testing_RYtwx7Jj8aE_14616_16043.png Automated_firewall_testing_RYtwx7Jj8aE_16044_17915.png Automated_firewall_testing_RYtwx7Jj8aE_17916_19451.png Automated_firewall_testing_RYtwx7Jj8aE_19452_20075.png Automated_firewall_testing_RYtwx7Jj8aE_20076_27767.png Automated_firewall_testing_RYtwx7Jj8aE_27768_28583.png Automated_firewall_testing_RYtwx7Jj8aE_28584_29399.png Automated_firewall_testing_RYtwx7Jj8aE_29400_29615.png Automated_firewall_testing_RYtwx7Jj8aE_29616_32483.png Automated_firewall_testing_RYtwx7Jj8aE_32484_32711.png Automated_firewall_testing_RYtwx7Jj8aE_32712_32795.png Automated_firewall_testing_RYtwx7Jj8aE_32796_33503.png Automated_firewall_testing_RYtwx7Jj8aE_33504_34727.png Automated_firewall_testing_RYtwx7Jj8aE_34728_35867.png Automated_firewall_testing_RYtwx7Jj8aE_35868_36335.png Automated_firewall_testing_RYtwx7Jj8aE_36336_36767.png Automated_firewall_testing_RYtwx7Jj8aE_36768_36887.png Automated_firewall_testing_RYtwx7Jj8aE_36888_37295.png Automated_firewall_testing_RYtwx7Jj8aE_37296_39995.png Automated_firewall_testing_RYtwx7Jj8aE_39996_40079.png Automated_firewall_testing_RYtwx7Jj8aE_40080_40439.png Automated_firewall_testing_RYtwx7Jj8aE_40440_40535.png Automated_firewall_testing_RYtwx7Jj8aE_40536_40727.png Automated_firewall_testing_RYtwx7Jj8aE_40728_43799.png Automated_firewall_testing_RYtwx7Jj8aE_43800_44279.png Automated_firewall_testing_RYtwx7Jj8aE_44280_44351.png Automated_firewall_testing_RYtwx7Jj8aE_44352_45491.png Automated_firewall_testing_RYtwx7Jj8aE_45492_51863.png Automated_firewall_testing_RYtwx7Jj8aE_51864_58355.png Automated_firewall_testing_RYtwx7Jj8aE_58356_58487.png Automated_firewall_testing_RYtwx7Jj8aE_58488_58571.png Automated_firewall_testing_RYtwx7Jj8aE_58572_58847.png Automated_firewall_testing_RYtwx7Jj8aE_58848_59063.png Automated_firewall_testing_RYtwx7Jj8aE_59064_59207.png Automated_firewall_testing_RYtwx7Jj8aE_59208_59267.png Automated_firewall_testing_RYtwx7Jj8aE_59268_59315.png Automated_firewall_testing_RYtwx7Jj8aE_59316_59363.png Automated_firewall_testing_RYtwx7Jj8aE_59364_59495.png Automated_firewall_testing_RYtwx7Jj8aE_59496_59543.png Automated_firewall_testing_RYtwx7Jj8aE_59544_59685.png