SUSE Security retrospective for last year
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Meissner, Marcus
Another year - another security retrospective. The talk will introduce the SUSE security team and its members, our areas of work and responsibilities. The talk will show some statistics and interesting details of last year security issues, and go in detail on some of the specific codenamed ones. A special focus will be on updated classification and overview of last years Meltdown and Spectre like CPU issues, describing them and the mitigations that SUSE has been deployed. As the SUSE security team has grown in the last year, we also increased our work in both proactive security and related areas, which the talk will briefly highlight.
 Yeah. Welcome to security review of the last year. My name is Marcus. I will give you a bit of retrospective of the last year, put on some highlights, but also tell you about our work that we are doing just in a pretty high level way so it will not have many technical details. So first, who are we? We are the SUSE security team. We are currently, let me count, eight people. Our boss has changed recently, so his name is Ivan Teplin. He's been with us for one month now. We have one project manager, that's me, and we have several security engineers. So in alphabetical order, we have Alexander, Alexsandros, Johannes, Malte, Matthias and Robert, and Alexsandros and Robert are sitting close to the back over there. The rest is unfortunately not here. We are still not complete. We've been still looking for a good security engineer. We have one very good candidate that we are trying to get and one open position that we are also planning to fill. So what are we doing? The work that we are doing is split into two major parts that we will go about separately. We have the proactive part and the reactive part. The reactive part is handling security issues, handling security incidents, reacting on the security incidents. We are doing that for all the ZUSA products, for all the open ZUSA products and distributions, but not for the kind of security issues that address physical security or website security for ZUSA. We are only working for the ZUSA Linux and open ZUSA products. The team is split into two parts. The reactive part has me as the project manager overseeing it, and we have Alexander, Alexsandros, Robert, and potentially a new guy working in this team and sharing the load in between. The proactive side, where we do software audits, like what you can imagine, pen testing audits, we approve the product releases that we do. So basically the release managers come to us and say, is my product secure? We do a final check of the product and give our green. Go ahead. And we do a lot of work before the security shipment during the development phase. And that part of the team consists currently of Johannes, Mild and Matthias, and another potential new hire that we hopefully will be getting this year. So in case you're looking, you're wondering what our names are, these are the names that a US ZUSA developer or ZUSA contributor will likely encounter during your work with the security team. So let me first talk about the reactive security, which is likely the more visible part of our work. We react on any reported security issues on our distribution, as mentioned both on SLEE and on OpenZUSA. To handle this load, I will give you some numbers in a minute. We have clearly specified roles so that everyone knows what to do, everyone knows exactly what he's been assigned to do, so that there is no confusion. We have split this into so-called incident managers and update managers, where we have currently two or three incident managers and one update manager. So the load is distributed evenly between the team. Control rotates monthly so that it doesn't get too boring and too one-sided for anyone. What is our task in the reactive side is we coordinate fixing the security issues from beginning from getting knowledge to it until the end, meaning release of all the updates and doing all the buck closing. We are not doing the fixes ourselves. We do that, we delegate it to the package maintainers, either the SUSE ones or the community maintainer, depending on who is assigned and the build service to the package. We also don't look at the source patches that get submitted, get fixed, that is also done by the OBS or EBS source review teams. The testing is delegated to the QA teams and these days there is a huge part of it done by automated testing just as Oliver told us before, mostly done by OpenQA, especially OpenSUSE is only tested by OpenQA at this time. There is the potential on OpenSUSE to have a testing repository added and used it, but we very strongly rely on OpenQA testing for OpenSUSE maintenance. The next part where we are involved is the release of the update and everything afterwards the documentation. Most of the documentation is automatically generated just because there are so many of it, only for specific incidents we do manual documentation work. So how does it look like? How do we handle the incidents? Where do we interface with the packages? We use the OpenSUSE bugzilla. We have a specific summary line tagging to make queries in bugzilla easier. So we start with a while-0 or while-1 prefix, then put in the CVE number as a prefix. We also put in the package name and then a short summary of the issue. That makes bugzilla searches very easy for us, for you and for everyone who wants to just look at or find such a thing. We fill in the bugzilla report with the description, with links to patches, with the reproteusers so that our QA can verify it or the package can verify it and whatever other information is needed. And we assign it afterwards to the package maintainer itself for fixing. If the package maintainer has questions, he can always come back to us, set need info, ask a question in a bug report and ask us if he needs help for fixing. Internally, we also use a tool called Smash that tracks the effectiveness and the ratings of code streams for the Susie and Enterprise products. Rating we have a simplified rating. It's called low-moderate, important critical. So four steps that kind of easy to understand and to look at. But we also use an industry standard rating called common vulnerability scoring, version free, which gives, which has a lot of indicators which in turn turn into number between zero and ten. That is also used in various industry standards to rate how fast security updates need to be applied or fixed. We evaluate and mark what is affected on which SLE code stream. That to some degree also reflects open SUSE code streams and we are also planning at some point in the future to directly track open SUSE also with this tool. And there's one thing for the minor security issues. We decide occasionally it can wait. It doesn't need to be fixed right away because it's unlikely exploitable or it's really a minor issue. Or if it is a bigger issue then we request an update via our workflow management tool. So first statistics as promised. So I have a query running since 2006. And you notice some trends. So that's the open box per week. And it starts at 2006. And on the right is the 2019 last week. So we started around 11 bucks per week and it really continued until 2014. And then it got very noisy. And knowing from the security point of view that happened because a lot of more automated bug finding was happening. So with fuzzing, with more attention on specific tools with automated bug finding, people were suddenly finding much more bugs in the software. One big example is ImageMagic for instance where over the last years there have been found like 500 security issues or similar. What is interesting is also however that it now starts to decrease a bit. So we had to peak from around 30, 40 bucks open per week. But the last year, the last one and a half years have seen a slight decline. That might mean that some of the means of automatic bug finding are exhausted. Or it might mean that people are focusing themselves more on different topics. Or that someone is not opening enough CVEs or classifying CVEs anymore. So while I was thinking that the trend would only go upwards, I was kind of disproven here. Let's see what the next years will bring. Short look at our incident tracking system that we use internally. We are getting in this ticket system around 100 issues per day. But we open bugs only for around 5 to 10 depending on whatever is being found. So having a good UI where we can track it, you can think of it mostly like a ticketing system that additionally has on top the rating and effectiveness marking that we do. Here's a bit longer list where we really already have specified ratings and affected packages. So how does the update management on our side work? So the package after getting knowledge and after fixing the package, he submits it. The security team, the update manager in this case accepts the update, writes the metadata that is shown to users that gets displayed later on in the automated, generated documentation like the summary, a description, a human readable description, and a list of issues in a machine readable format. We checked if it's, of course, it's building. We have some bots that also catch it if it's not building before these days. But under certain circumstances, it might not build and might need some help. And we then forwarded to QA. So basically QA in that case is set both automated and also manual QA. The automation does these days all the regression testing. So regression testing is no longer done manually, but our QA teams do manual bug fix verification. So they verify that the bug is present before and that it is fixed afterwards. Once QA is finished, we basically only then press the release button. List of the documentation, the emails that you see, the patches that is all automatically generated. But for bigger issues, we do some manual work. So look at our, we have a tooling also for the update management in the enterprise. As you see, we have in that case, we have 138 updates in QA or waiting for a QA engineer with the amount of packages that we release. We release like 50 updates, 60 updates per week on the SLE side, around 20 to 30 on the open source side without a tooling. It's really not manageable anymore. This is tooling that we developed ourselves within the maintenance team with dedicated engineers looking at it that makes our work more efficient and easier. Yes, so what we deliver in the end, as mentioned, of course, the updates themselves get to your system. We send out notifications per email. They are uploaded to a website for this SLE updates. We also send, of course, emails to the open source announcement lists. And we generate advisory data in a standard XML format called CVRF, common vulnerability reporting format. I hope I got it right because that's all acronyms. But that basically encodes the advisory in a very big XML blob and can be then used for importing into ticket systems, et cetera. We also generate other information. So on a per CVE information, we generate for every CVE we find and evaluate and fix. We have one page on our website where you can see description, ratings, effectiveness and what we have released. Also this similar data is generated by us also for machine readable and machine readable format called Oval, open vulnerability and assessment language. Yet again a very big XML format that you can run through an interpreter and see if the system is still affected by a CVE or not and what needs to be updated. So some statistics for this year compared to last year. We are not this year by 2018 just to have a complete year. So the number of CVEs that we looked at is 3,000. So 300 less than last year. Kind of coincidates with the open bug trends a bit. We opened around 2,280 bucks. That's 300 less than in 2017. And if you look at the classification, so I just put in the big numbers, same number of approximately the kernel bugs but way less image magic and graphics magic bugs. So it really 170 bucks less or CVEs less for image magic and graphics magic just because there the bug finding has, that we are using fuzzers has exhausted the bugs basically. It has not completed exhausted them. As you see it's still 90 bucks which is quite an amount but it's not as massive as in 2017. Similar also for Firefox. Firefox 50 bucks less than in 2017. So there's still fixing and finding bugs in those code bases but it gets better over time. The number of updates that we released, firstly it looks much higher but the reason for that is that we have more service packs and more products in 2018 and 2017. I have not equalized it to the number of service packs that would have been a bit too complicated with the general support and LTSS support. For open ZOOZER that is a bit more comparable even though we have 42.3 and 15.0. We have released some more updates for open ZOOZER than in 2017 and I also looked at how much we import from SLEE and we import around 50% from SLEE at this time. Total number of CVEs fixed. We have 100 CVEs less fixed also explained by the image magic decrease mostly and these are the numbers for open ZOOZER and SLEE. They don't add up because they share CVE space. So one thing that the reactive team does is I mentioned it before for some issues we do specific documentation or more documentation and these are the so-called high impact vulnerabilities the things that you read and press about or even on a target show or on CNN. Started all with hard bleed and shell shock and went even into this year with system down and just lately the MDS CPU issue. These issues are special so usually you have to think that our reactive incident work is just it's a pipeline. We are opening bugs, updates get processed, packages get submitted, get released and it just takes a while. But for these high impact vulnerabilities that have a severe technical impact or where people really want to fix right now, we need a quicker reaction. We also need more communication of under what circumstances are we affected, what is the real impact? Is it really that big of a problem? And also documentation for CPU issues we need to document what options there are that customers can use to configure their system. And especially if there are also tricky fixes if you need to do something manually on your system to mitigate it. So we handle this a bit differently, but with much higher attention, with direct attention and we even have a specific team dedicated to the emergency update team that goes and does these fixes themselves, coordinates them, including even weekend work or late night work. So if one or two guys from the emergency update team are also here, Simon from the open Zuzo board is one of these members and Peter, I have not seen yet, but also a member. So last year I talked a bit about Stacklash and what we are planning to do and what we are doing. That was something from 2017. General problem where the stack growing down and MAP areas growing up would collide and could be exploited. And what we had done already was mitigating that in the kernel with bigger stack apps that could not be jumped over. We already had prepared GCC updates that were generating binary code that no longer has this problem. And that's something that only started in effect last year is that we released products open Zuzo Leap 15.0 and sleep 15 that already had this mitigation enabled. And we also started releasing last year's updates for enterprise 12 and also Leap 42.3 that these packages that we released were no longer affected by the Stacklash issue. And our efforts paid themselves off and that's something I want to show off is because in January there was a system down exploit of the system D, where regular user space program could gain root execution by just sending a specific log message to the journal D. But as we had already built system D with the Stacklash protector also within all the updates both Zuzo Enterprise and open Zuzo were already fixed. So doing this proactively, building the packages proactively with this option really paid out as we were not affected by this problem begin of this year. So the big topic is now what to go to in a bit more detail, the CPU issues. So last year I also talked about a bit of mail down and spectra already. But they got some new friends this year and last year the general overview is a side channel information leak attacks that use speculative or out of order execution. To be class spectra is a speculative execution of branches that might not be taken or are not taken, leaks information that happen to be accessed in those non-taken branches. And the other part is the mail down class of CPU side channel issues, basically lack of access control during this kind of out of order execution. Mail down itself was because during out of order execution the protection bit for user, for ring zero protected pages was not checked so everything from the kernel could be read from user space. And various flavors of that like page not present bit, also some high level bits like an MDS that we had. So various other flavors of those were also found. Fortunately none of them as severe as mail down itself but various flavors of both have been found. So the problem with those is that as there are CPU issues, we had a long embargo times. Also for the last issue we had over nine months, nine months of a embargo time. Just to prepare the patches, get every industry, other operating and hypervisor vendor under the same umbrella within timing, get the micro code updates ready. Interest really takes like half a year, three months to half a year to get micro code, fixes ready for their CPUs. So it was kind of long embargo times and we of course need to facilitate it in our kernels test that and be ready on the date of shipment because these issues still attract quite a lot of attention. One issue is of course that they have also almost all of those fixes get some performance losses which is especially in a cloud provider space, not a good thing if suddenly your cloud runs 10% slower or something. I said it become mostly a new family, mail down, 1-4 bound check bypass door, lazy FPU that was possible to leak CPU, FPU registers, L1 terminal fault and net spectrum happened in 2018 and even in 2019 we found, let's misspelled spoiler, smother spectra, port smash and just from 14th of May micro architectural data sampling called also fallout, with or zombie load attack. So the good news is that for a number of them we have mitigations or we have the potential to mitigate them completely, usually the mail down family that is occasionally done with micro code and kernel mitigations acting together and we know how to fix the other ones that are mostly bug classes. And the other good news is that the affected Intel, largely Intel CPUs, Intel has been started shipping new CPUs, Cascade Lake and newer that are no longer affected by the mail down problem. So it's good for Intel that they are selling more CPUs, it's also good for us that we get back our performance and security. So not so good news or the bad news is that lately there have been some issues found that are exploitable when using hyper threading. So hyper threading is where CPU core has running multiple threads of execution and they share various resources like load buffers, store buffers, computational units and so on. And the last attacks at 1TF and the micro architecture data assembling, they were largely exploiting these smaller buffers shared between hyper threads. So the mitigations that we have, they are effective if stuff executes in serial way, like user space to root, to kernel space and back, that is okay but for threads on the same core for a type of thread on the same core, it is hard to solve because we cannot really influence what runs on the other thread. So not in a current setting. So the obvious solution is of course to schedule only processes with the right privileges on the same core. That's called gang scheduling or co-scheduling. Unfortunately that's not really, there's no really working code for that because doing it in the right way that cannot be exploited is very tricky and likely has a similar performance impact than just switching off hyper threading altogether. So there are some draft patches for the Linux kernel and I think also for Xen but these are not yet production ready. We hope that they get to a point where they are production ready and performant and then we will go and deploy them but so far they are sadly not ready yet. So if you are really concerned about on your machine having an attacker running on the same core trying to steal your secrets, the only way that is currently there is to disable hyper threading on your system. So we had like eight different issues, seven different mitigations and they appeared a website called makelinux fast again that put in all the kernel command line options that are needed to get the full performance back. And you see if I now hide it and you try to remember it, even I couldn't remember it. So it needs a simpler front end. And the kernel guys, the Linux kernel developers found yes, it definitely needs a more simple configuration option and they just implemented with the 5.1 or 5.2 kernel, I think 5.1 kernel. They implemented an option, the kernel bootline option called mitigations that controls all of the CPU mitigations at once. And it has three easy settings. It has the setting off, just disabling it, getting back all the performance. It has the setting auto, like it will select based on your CPU type what is needed or what is not needed as mitigation. So if you have a new CPU, it will not switch on Meltdown protection. And it also has the option of disabling hyper threading if you're so, if you're so desire. At SUSE we have discussed it on various levels and decided with going with the default of auto, so leaving hyper threading enabled, leaving that kind of performance still available, but at least enabling all the standard mitigations that are available. And to make it easier also for our users, we have integrated this into the Jaston installer. So if you install the installation overview, we'll have in the security section a CPU mitigations option that you can click on and you can easily change from auto to either off or auto or no SMT depending on your security needs. So there's one last bad thing of that is that some of the issues are bug classes that are not getting fixed in the processor. So Spectre variant one, that is basically branch prediction or speculation over branches that are not taken. This is will not be getting fixed in the processor, but we are supposed to fix that in the program code in the kernel or in user land. It's mostly a bug class. So if you want to fix, there's not a single line of code that needs to be fixed, but it's in a lot of places. And it needs, yeah, it's in a lot of places and there's no generic way to do it. There has been some research into compiler based mitigation of that of Spectre variant one, but unfortunately the performance loss of doing this in a compiler is quite high. So it's really like 50% performance loss if you do it via a compiler. So it's under control in the kernel. The kernel has deployed things like index masking and the respective fences that are needed to control that. Our user land code has also been already supplied with it like some Java script, just in time compilers and so on. But this might be an issue where we are getting, where we still need to take care about in the future. There are two others that were only recently published, Smother Spectre and Portsmash. And these are not leaking direct information, but these look at what other, these can find out what the other CPU thread is doing. So if it's doing a lot of vector stuff, if it's doing a lot of multiplications or additions. By monitoring that you can find out for instance, if there's a cryptographic algorithm running on the other CPU that is not specifically prepared for that, you can find out what kind of keys the other cryptographic algorithm is just processing and find out the content of the keys. So the only way to fix that currently is to really write cryptographic code and any other code that could be snooped on, where information could be gained on from snooping on in a constant time manner. So having constant time compares, constant time multiplications, additions and so on. Fortunately, a lot of cryptographic code is already prepared for that because this kind of computational side channel issue is not a new thing, but it becomes more urgent with this knowledge about this exploitation of issue of via hyper threading. So enough for the reactive security part, what is our proactive team doing? Their general mission statement is making products secure before shipment and implement a secure development lifecycle within Zuse. We are doing this at various stages. We are doing this in automated fashion, in a manual fashion. So the first thing that comes to mind is automated checks during build that, similar to what we do with make check, we also do for security parts. Like if you add a new security binary, your build will fail if it's not white listed by us. If you add a new D-Bus service or policy kit rule and it's not white listed by us, your build will fail and we will tell you to contact a security team for white listing. And that time, you open a bug, we look at the code, we tell you where the bugs are and you fix the bugs and then we white listed. That includes all other things, system network services or whatever security interface you have in your package or in your product. We are not doing this just before the release, but we are doing this during the whole process. So basically, similar to the open-source effector model, all these audits are rolling. Especially as we are also working with factory, the factory developers come to us, say they have a new D-Bus service, can we please review that? We review it, it gets approved. And at some point in time, it gets shipped via open-source-leap or through the end-up price. These are largely based on the packages that we are doing, but we are also doing product audits. Currently, we are doing product audits or the final product, send off audit a month before usually product is getting shipped. So open-source-leap 15-1 was reviewed a month ago. Johannes took a look at it for a week and fortunately found no big issues there. So even if we are the security team, what we are doing is set, we work in factory, a lot of stuff that we are doing is both done in open-source, even in open-source at first and then later on in suzium's enterprise. But our focus is of course getting suzium's enterprise secure, but open-source is our main driver of work there. Just to look at the workload that we had, we had around the same number of audit requests last in 2018 and in 2017. But we were able to process more because our team has grown. I have not gotten those numbers unfortunately. What does the project team also do? We are new hardening methods. Last year I talked a bit about pie support. What we are currently working on is that we had LD flex, we had for a long time already, but now we also want to deploy the minus set now hardening for F binaries. And we want to also move some compiler warning checks that we are currently doing in post-build checks to directly to W error options, which is done by the compiler team. So that your build will not just emit warnings, but it will directly fail if something is wrong. Our colleague Karol, fortunately no longer in the security team, but now in a different role, he is a very big UB key fan and yet he just has now a UB key workshop at the UB key talk yesterday. That is something that we added or improved up in open-source in the last year and also shipped in SLS 12, Sb4 and SLS 15 for two-factor authentication. Last year also saw some Go script security issues. I didn't mention them earlier, but what we did with Go script is that we improved the hardening for Go script. So for image magic and graphics magic that can convert PostScript to images, we now shipped two configuration files. One configuration file has these coders disabled and one configuration has these enabled depending on your use case. And the default is that we are disabled them because Go script is still a very risky tool and might have more security issues than we have already fixed. We also this Sb4 PostScript display in places where it's not really needed or made it optional or tools that you wouldn't even have known that it does that, be disabled that. We also tried confinement using upArmor, a general confinement tool that fortunately has not been very successful yet because Go script is in use of a lot of pipe modes and a lot of settings that upArmor is not fully, where we cannot fully control it with upArmor. We had quite some success, but we had also some kind of angry users that were no longer able to run the preferred tool. We also do audits for non-distribution products like cloud and the storage. We had reviewed the new bigger tools that get added. So a lot of things happening at Zuse is adding stuff on top of distributions. We also planning or occasionally planning on embedding engineers into product teams while we haven't done this in the last year, that is an option that we always consider that especially in bigger agile teams that we occasionally help out and send someone to a system for some weeks. Then we have changing new products like the container management product, CASP, also the cloud application platform that is using Cloud Foundry based on top of CASP, on top of OpenStack, which are very complicated products, quite hard to understand products, quite hard to get an overview over, and of course, in turn, making these also hard to make secure products. But this is something that we will also be looking into in the next years even more closely. One side project that we are doing is certifications. One of this is directly security related. This is the DISA stick. Stick stands for Secure Technical Implementation Guide and is just another formalized word for hardening guide. It's really a formalized thing largely coordinated by the US government and especially the US military. They really describe in a step-by-step formal approach how to secure a system. They have various methods that can be handed out and a textual description form where you have 200, 300 rules that the administrator goes through, checks, makes the changes, gets the next rule, checks, makes the changes, and so on. Or it can be done in an automated format. In a so-called benchmark or SCAP format, SCAP is Secure Content Automation Protocol. And we are also currently working and providing this to our enterprise customers, but the same can only with some small adjustments also likely deployed on open zoos. So it's basically what we had earlier as hardening guides in a very formal textual form and later on in a XML form. We also do cryptographic certifications. Just recently the NIST, also another US government body, released the FIPS 140-3 standard, basically describing crypto parameters that your cryptographic system has to fulfill. And also we plan to work on a common-quarantia certification, which we did for the last enterprise servers. We also plan to do that for the enterprise server 15. The common-quarantia certification describes the whole development process and the product that comes out of it. So how are we publicly visible, accepting of course the service we provide to open zoos. What we do benefits all the upstream open source projects. They get more secure, so even other vendors benefit from our work. We are recently participating in the Nürnberg IT security meetup called 0911.org. And we are hosting every second instance at our offices. And we are presenting occasionally at conferences like this or other security conferences. We are planning on extending our presence more as we are growing. Currently it's not that much, but we are trying to improve on that. So that's it from my side. Good on our landing page and our main contact address where you can address security questions here and that we are looking for some good engineers. Any questions? So I will be hanging around here if you have questions later on. And thank you for listening.
 Yeah.  Welcome to security review of the last year.  My name is Marcus.  I will give you a bit of retrospective of the last year, put on some highlights, but  also tell you about our work that we are doing just in a pretty high level way so it will  not have many technical details.  So first, who are we?  We are the SUSE security team.  We are currently, let me count, eight people.  Our boss has changed recently, so his name is Ivan Teplin.  He's been with us for one month now.  We have one project manager, that's me, and we have several security engineers.  So in alphabetical order, we have Alexander, Alexsandros, Johannes, Malte, Matthias and  Robert, and Alexsandros and Robert are sitting close to the back over there.  The rest is unfortunately not here.  We are still not complete.  We've been still looking for a good security engineer.  We have one very good candidate that we are trying to get and one open position that we  are also planning to fill.  So what are we doing?  The work that we are doing is split into two major parts that we will go about separately.  We have the proactive part and the reactive part.  The reactive part is handling security issues, handling security incidents, reacting on the  security incidents.  We are doing that for all the ZUSA products, for all the open ZUSA products and distributions,  but not for the kind of security issues that address physical security or website security  for ZUSA.  We are only working for the ZUSA Linux and open ZUSA products.  The team is split into two parts.  The reactive part has me as the project manager overseeing it, and we have Alexander, Alexsandros,  Robert, and potentially a new guy working in this team and sharing the load in between.  The proactive side, where we do software audits, like what you can imagine, pen testing audits,  we approve the product releases that we do.  So basically the release managers come to us and say, is my product secure?  We do a final check of the product and give our green.  Go ahead.  And we do a lot of work before the security shipment during the development phase.  And that part of the team consists currently of Johannes, Mild and Matthias, and another  potential new hire that we hopefully will be getting this year.  So in case you're looking, you're wondering what our names are, these are the names that  a US ZUSA developer or ZUSA contributor will likely encounter during your work with the  security team.  So let me first talk about the reactive security, which is likely the more visible part of our  work.  We react on any reported security issues on our distribution, as mentioned both on SLEE  and on OpenZUSA.  To handle this load, I will give you some numbers in a minute.  We have clearly specified roles so that everyone knows what to do, everyone knows exactly what  he's been assigned to do, so that there is no confusion.  We have split this into so-called incident managers and update managers, where we have  currently two or three incident managers and one update manager.  So the load is distributed evenly between the team.  Control rotates monthly so that it doesn't get too boring and too one-sided for anyone.  What is our task in the reactive side is we coordinate fixing the security issues from  beginning from getting knowledge to it until the end, meaning release of all the updates  and doing all the buck closing.  We are not doing the fixes ourselves.  We do that, we delegate it to the package maintainers, either the SUSE ones or the community  maintainer, depending on who is assigned and the build service to the package.  We also don't look at the source patches that get submitted, get fixed, that is also done  by the OBS or EBS source review teams.  The testing is delegated to the QA teams and these days there is a huge part of it done  by automated testing just as Oliver told us before, mostly done by OpenQA, especially  OpenSUSE is only tested by OpenQA at this time.  There is the potential on OpenSUSE to have a testing repository added and used it, but  we very strongly rely on OpenQA testing for OpenSUSE maintenance.  The next part where we are involved is the release of the update and everything afterwards  the documentation.  Most of the documentation is automatically generated just because there are so many of  it, only for specific incidents we do manual documentation work.  So how does it look like?  How do we handle the incidents?  Where do we interface with the packages?  We use the OpenSUSE bugzilla.  We have a specific summary line tagging to make queries in bugzilla easier.  So we start with a while-0 or while-1 prefix, then put in the CVE number as a prefix.  We also put in the package name and then a short summary of the issue.  That makes bugzilla searches very easy for us, for you and for everyone who wants to just  look at or find such a thing.  We fill in the bugzilla report with the description, with links to patches, with the reproteusers  so that our QA can verify it or the package can verify it and whatever other information  is needed.  And we assign it afterwards to the package maintainer itself for fixing.  If the package maintainer has questions, he can always come back to us, set need info,  ask a question in a bug report and ask us if he needs help for fixing.  Internally, we also use a tool called Smash that tracks the effectiveness and the ratings  of code streams for the Susie and Enterprise products.  Rating we have a simplified rating.  It's called low-moderate, important critical.  So four steps that kind of easy to understand and to look at.  But we also use an industry standard rating called common vulnerability scoring, version  free, which gives, which has a lot of indicators which in turn turn into number between zero  and ten.  That is also used in various industry standards to rate how fast security updates need to be  applied or fixed.  We evaluate and mark what is affected on which SLE code stream.  That to some degree also reflects open SUSE code streams and we are also planning at some  point in the future to directly track open SUSE also with this tool.  And there's one thing for the minor security issues.  We decide occasionally it can wait.  It doesn't need to be fixed right away because it's unlikely exploitable or it's really  a minor issue.  Or if it is a bigger issue then we request an update via our workflow management tool.  So first statistics as promised.  So I have a query running since 2006.  And you notice some trends.  So that's the open box per week.  And it starts at 2006.  And on the right is the 2019 last week.  So we started around 11 bucks per week and it really continued until 2014.  And then it got very noisy.  And knowing from the security point of view that happened because a lot of more automated  bug finding was happening.  So with fuzzing, with more attention on specific tools with automated bug finding, people were  suddenly finding much more bugs in the software.  One big example is ImageMagic for instance where over the last years there have been  found like 500 security issues or similar.  What is interesting is also however that it now starts to decrease a bit.  So we had to peak from around 30, 40 bucks open per week.  But the last year, the last one and a half years have seen a slight decline.  That might mean that some of the means of automatic bug finding are exhausted.  Or it might mean that people are focusing themselves more on different topics.  Or that someone is not opening enough CVEs or classifying CVEs anymore.  So while I was thinking that the trend would only go upwards, I was kind of disproven here.  Let's see what the next years will bring.  Short look at our incident tracking system that we use internally.  We are getting in this ticket system around 100 issues per day.  But we open bugs only for around 5 to 10 depending on whatever is being found.  So having a good UI where we can track it, you can think of it mostly like a ticketing  system that additionally has on top the rating and effectiveness marking that we do.  Here's a bit longer list where we really already have specified ratings and affected packages.  So how does the update management on our side work?  So the package after getting knowledge and after fixing the package, he submits it.  The security team, the update manager in this case accepts the update, writes the metadata  that is shown to users that gets displayed later on in the automated, generated documentation  like the summary, a description, a human readable description, and a list of issues in a machine  readable format.  We checked if it's, of course, it's building.  We have some bots that also catch it if it's not building before these days.  But under certain circumstances, it might not build and might need some help.  And we then forwarded to QA.  So basically QA in that case is set both automated and also manual QA.  The automation does these days all the regression testing.  So regression testing is no longer done manually, but our QA teams do manual bug fix verification.  So they verify that the bug is present before and that it is fixed afterwards.  Once QA is finished, we basically only then press the release button.  List of the documentation, the emails that you see, the patches that is all automatically  generated.  But for bigger issues, we do some manual work.  So look at our, we have a tooling also for the update management in the enterprise.  As you see, we have in that case, we have 138 updates in QA or waiting for a QA engineer  with the amount of packages that we release.  We release like 50 updates, 60 updates per week on the SLE side, around 20 to 30 on the  open source side without a tooling.  It's really not manageable anymore.  This is tooling that we developed ourselves within the maintenance team with dedicated  engineers looking at it that makes our work more efficient and easier.  Yes, so what we deliver in the end, as mentioned, of course, the updates themselves get to your  system.  We send out notifications per email.  They are uploaded to a website for this SLE updates.  We also send, of course, emails to the open source announcement lists.  And we generate advisory data in a standard XML format called CVRF, common vulnerability  reporting format.  I hope I got it right because that's all acronyms.  But that basically encodes the advisory in a very big XML blob and can be then used for  importing into ticket systems, et cetera.  We also generate other information.  So on a per CVE information, we generate for every CVE we find and evaluate and fix.  We have one page on our website where you can see description, ratings, effectiveness  and what we have released.  Also this similar data is generated by us also for machine readable and machine readable  format called Oval, open vulnerability and assessment language.  Yet again a very big XML format that you can run through an interpreter and see if the system  is still affected by a CVE or not and what needs to be updated.  So some statistics for this year compared to last year.  We are not this year by 2018 just to have a complete year.  So the number of CVEs that we looked at is 3,000.  So 300 less than last year.  Kind of coincidates with the open bug trends a bit.  We opened around 2,280 bucks.  That's 300 less than in 2017.  And if you look at the classification, so I just put in the big numbers, same number  of approximately the kernel bugs but way less image magic and graphics magic bugs.  So it really 170 bucks less or CVEs less for image magic and graphics magic just because  there the bug finding has, that we are using fuzzers has exhausted the bugs basically.  It has not completed exhausted them.  As you see it's still 90 bucks which is quite an amount but it's not as massive as in 2017.  Similar also for Firefox.  Firefox 50 bucks less than in 2017.  So there's still fixing and finding bugs in those code bases but it gets better over  time.  The number of updates that we released, firstly it looks much higher but the reason for that  is that we have more service packs and more products in 2018 and 2017.  I have not equalized it to the number of service packs that would have been a bit too complicated  with the general support and LTSS support.  For open ZOOZER that is a bit more comparable even though we have 42.3 and 15.0.  We have released some more updates for open ZOOZER than in 2017 and I also looked at how  much we import from SLEE and we import around 50% from SLEE at this time.  Total number of CVEs fixed.  We have 100 CVEs less fixed also explained by the image magic decrease mostly and these  are the numbers for open ZOOZER and SLEE.  They don't add up because they share CVE space.  So one thing that the reactive team does is I mentioned it before for some issues we do  specific documentation or more documentation and these are the so-called high impact vulnerabilities  the things that you read and press about or even on a target show or on CNN.  Started all with hard bleed and shell shock and went even into this year with system down  and just lately the MDS CPU issue.  These issues are special so usually you have to think that our reactive incident work is  just it's a pipeline.  We are opening bugs, updates get processed, packages get submitted, get released and it  just takes a while.  But for these high impact vulnerabilities that have a severe technical impact or where people  really want to fix right now, we need a quicker reaction.  We also need more communication of under what circumstances are we affected, what is the  real impact?  Is it really that big of a problem?  And also documentation for CPU issues we need to document what options there are that customers  can use to configure their system.  And especially if there are also tricky fixes if you need to do something manually on your  system to mitigate it.  So we handle this a bit differently, but with much higher attention, with direct attention  and we even have a specific team dedicated to the emergency update team that goes and  does these fixes themselves, coordinates them, including even weekend work or late night  work.  So if one or two guys from the emergency update team are also here, Simon from the open Zuzo  board is one of these members and Peter, I have not seen yet, but also a member.  So last year I talked a bit about Stacklash and what we are planning to do and what we  are doing.  That was something from 2017.  General problem where the stack growing down and MAP areas growing up would collide and  could be exploited.  And what we had done already was mitigating that in the kernel with bigger stack apps  that could not be jumped over.  We already had prepared GCC updates that were generating binary code that no longer has  this problem.  And that's something that only started in effect last year is that we released products  open Zuzo Leap 15.0 and sleep 15 that already had this mitigation enabled.  And we also started releasing last year's updates for enterprise 12 and also Leap 42.3  that these packages that we released were no longer affected by the Stacklash issue.  And our efforts paid themselves off and that's something I want to show off is because in  January there was a system down exploit of the system D, where regular user space program  could gain root execution by just sending a specific log message to the journal D. But  as we had already built system D with the Stacklash protector also within all the updates  both Zuzo Enterprise and open Zuzo were already fixed.  So doing this proactively, building the packages proactively with this option really paid out  as we were not affected by this problem begin of this year.  So the big topic is now what to go to in a bit more detail, the CPU issues.  So last year I also talked about a bit of mail down and spectra already.  But they got some new friends this year and last year the general overview is a side channel  information leak attacks that use speculative or out of order execution.  To be class spectra is a speculative execution of branches that might not be taken or are  not taken, leaks information that happen to be accessed in those non-taken branches.  And the other part is the mail down class of CPU side channel issues, basically lack  of access control during this kind of out of order execution.  Mail down itself was because during out of order execution the protection bit for user,  for ring zero protected pages was not checked so everything from the kernel could be read  from user space.  And various flavors of that like page not present bit, also some high level bits like  an MDS that we had.  So various other flavors of those were also found.  Fortunately none of them as severe as mail down itself but various flavors of both have  been found.  So the problem with those is that as there are CPU issues, we had a long embargo times.  Also for the last issue we had over nine months, nine months of a embargo time.  Just to prepare the patches, get every industry, other operating and hypervisor vendor under  the same umbrella within timing, get the micro code updates ready.  Interest really takes like half a year, three months to half a year to get micro code, fixes  ready for their CPUs.  So it was kind of long embargo times and we of course need to facilitate it in our kernels  test that and be ready on the date of shipment because these issues still attract quite a  lot of attention.  One issue is of course that they have also almost all of those fixes get some performance  losses which is especially in a cloud provider space, not a good thing if suddenly your cloud  runs 10% slower or something.  I said it become mostly a new family, mail down, 1-4 bound check bypass door, lazy FPU  that was possible to leak CPU, FPU registers, L1 terminal fault and net spectrum happened  in 2018 and even in 2019 we found, let's misspelled spoiler, smother spectra, port smash and just  from 14th of May micro architectural data sampling called also fallout, with or zombie  load attack.  So the good news is that for a number of them we have mitigations or we have the potential  to mitigate them completely, usually the mail down family that is occasionally done with  micro code and kernel mitigations acting together and we know how to fix the other ones that  are mostly bug classes.  And the other good news is that the affected Intel, largely Intel CPUs, Intel has been  started shipping new CPUs, Cascade Lake and newer that are no longer affected by the mail  down problem.  So it's good for Intel that they are selling more CPUs, it's also good for us that we get  back our performance and security.  So not so good news or the bad news is that lately there have been some issues found that  are exploitable when using hyper threading.  So hyper threading is where CPU core has running multiple threads of execution and they share  various resources like load buffers, store buffers, computational units and so on.  And the last attacks at 1TF and the micro architecture data assembling, they were largely  exploiting these smaller buffers shared between hyper threads.  So the mitigations that we have, they are effective if stuff executes in serial way,  like user space to root, to kernel space and back, that is okay but for threads on the  same core for a type of thread on the same core, it is hard to solve because we cannot  really influence what runs on the other thread.  So not in a current setting.  So the obvious solution is of course to schedule only processes with the right privileges on  the same core.  That's called gang scheduling or co-scheduling.  Unfortunately that's not really, there's no really working code for that because doing  it in the right way that cannot be exploited is very tricky and likely has a similar performance  impact than just switching off hyper threading altogether.  So there are some draft patches for the Linux kernel and I think also for Xen but these  are not yet production ready.  We hope that they get to a point where they are production ready and performant and then  we will go and deploy them but so far they are sadly not ready yet.  So if you are really concerned about on your machine having an attacker running on the same  core trying to steal your secrets, the only way that is currently there is to disable  hyper threading on your system.  So we had like eight different issues, seven different mitigations and they appeared a website  called makelinux fast again that put in all the kernel command line options that are needed  to get the full performance back.  And you see if I now hide it and you try to remember it, even I couldn't remember it.  So it needs a simpler front end.  And the kernel guys, the Linux kernel developers found yes, it definitely needs a more simple  configuration option and they just implemented with the 5.1 or 5.2 kernel, I think 5.1 kernel.  They implemented an option, the kernel bootline option called mitigations that controls all  of the CPU mitigations at once.  And it has three easy settings.  It has the setting off, just disabling it, getting back all the performance.  It has the setting auto, like it will select based on your CPU type what is needed or what  is not needed as mitigation.  So if you have a new CPU, it will not switch on Meltdown protection.  And it also has the option of disabling hyper threading if you're so, if you're so desire.  At SUSE we have discussed it on various levels and decided with going with the default of  auto, so leaving hyper threading enabled, leaving that kind of performance still available,  but at least enabling all the standard mitigations that are available.  And to make it easier also for our users, we have integrated this into the Jaston installer.  So if you install the installation overview, we'll have in the security section a CPU mitigations  option that you can click on and you can easily change from auto to either off or auto or  no SMT depending on your security needs.  So there's one last bad thing of that is that some of the issues are bug classes that are  not getting fixed in the processor.  So Spectre variant one, that is basically branch prediction or speculation over branches  that are not taken.  This is will not be getting fixed in the processor, but we are supposed to fix that in the program  code in the kernel or in user land.  It's mostly a bug class.  So if you want to fix, there's not a single line of code that needs to be fixed, but it's  in a lot of places.  And it needs, yeah, it's in a lot of places and there's no generic way to do it.  There has been some research into compiler based mitigation of that of Spectre variant  one, but unfortunately the performance loss of doing this in a compiler is quite high.  So it's really like 50% performance loss if you do it via a compiler.  So it's under control in the kernel.  The kernel has deployed things like index masking and the respective fences that are  needed to control that.  Our user land code has also been already supplied with it like some Java script, just in time  compilers and so on.  But this might be an issue where we are getting, where we still need to take care about in  the future.  There are two others that were only recently published, Smother Spectre and Portsmash.  And these are not leaking direct information, but these look at what other, these can find  out what the other CPU thread is doing.  So if it's doing a lot of vector stuff, if it's doing a lot of multiplications or additions.  By monitoring that you can find out for instance, if there's a cryptographic algorithm running  on the other CPU that is not specifically prepared for that, you can find out what kind  of keys the other cryptographic algorithm is just processing and find out the content  of the keys.  So the only way to fix that currently is to really write cryptographic code and any other  code that could be snooped on, where information could be gained on from snooping on in a constant  time manner.  So having constant time compares, constant time multiplications, additions and so on.  Fortunately, a lot of cryptographic code is already prepared for that because this kind  of computational side channel issue is not a new thing, but it becomes more urgent with  this knowledge about this exploitation of issue of via hyper threading.  So enough for the reactive security part, what is our proactive team doing?  Their general mission statement is making products secure before shipment and implement  a secure development lifecycle within Zuse.  We are doing this at various stages.  We are doing this in automated fashion, in a manual fashion.  So the first thing that comes to mind is automated checks during build that, similar to what we  do with make check, we also do for security parts.  Like if you add a new security binary, your build will fail if it's not white listed by  us.  If you add a new D-Bus service or policy kit rule and it's not white listed by us, your  build will fail and we will tell you to contact a security team for white listing.  And that time, you open a bug, we look at the code, we tell you where the bugs are and  you fix the bugs and then we white listed.  That includes all other things, system network services or whatever security interface you  have in your package or in your product.  We are not doing this just before the release, but we are doing this during the whole process.  So basically, similar to the open-source effector model, all these audits are rolling.  Especially as we are also working with factory, the factory developers come to us, say they  have a new D-Bus service, can we please review that?  We review it, it gets approved.  And at some point in time, it gets shipped via open-source-leap or through the end-up  price.  These are largely based on the packages that we are doing, but we are also doing product  audits.  Currently, we are doing product audits or the final product, send off audit a month before  usually product is getting shipped.  So open-source-leap 15-1 was reviewed a month ago.  Johannes took a look at it for a week and fortunately found no big issues there.  So even if we are the security team, what we are doing is set, we work in factory, a lot  of stuff that we are doing is both done in open-source, even in open-source at first and  then later on in suzium's enterprise.  But our focus is of course getting suzium's enterprise secure, but open-source is our main  driver of work there.  Just to look at the workload that we had, we had around the same number of audit requests  last in 2018 and in 2017.  But we were able to process more because our team has grown.  I have not gotten those numbers unfortunately.  What does the project team also do?  We are new hardening methods.  Last year I talked a bit about pie support.  What we are currently working on is that we had LD flex, we had for a long time already,  but now we also want to deploy the minus set now hardening for F binaries.  And we want to also move some compiler warning checks that we are currently doing in post-build  checks to directly to W error options, which is done by the compiler team.  So that your build will not just emit warnings, but it will directly fail if something is  wrong.  Our colleague Karol, fortunately no longer in the security team, but now in a different  role, he is a very big UB key fan and yet he just has now a UB key workshop at the UB  key talk yesterday.  That is something that we added or improved up in open-source in the last year and also  shipped in SLS 12, Sb4 and SLS 15 for two-factor authentication.  Last year also saw some Go script security issues.  I didn't mention them earlier, but what we did with Go script is that we improved the  hardening for Go script.  So for image magic and graphics magic that can convert PostScript to images, we now shipped  two configuration files.  One configuration file has these coders disabled and one configuration has these enabled depending  on your use case.  And the default is that we are disabled them because Go script is still a very risky tool  and might have more security issues than we have already fixed.  We also this Sb4 PostScript display in places where it's not really needed or made it optional  or tools that you wouldn't even have known that it does that, be disabled that.  We also tried confinement using upArmor, a general confinement tool that fortunately  has not been very successful yet because Go script is in use of a lot of pipe modes and  a lot of settings that upArmor is not fully, where we cannot fully control it with upArmor.  We had quite some success, but we had also some kind of angry users that were no longer  able to run the preferred tool.  We also do audits for non-distribution products like cloud and the storage.  We had reviewed the new bigger tools that get added.  So a lot of things happening at Zuse is adding stuff on top of distributions.  We also planning or occasionally planning on embedding engineers into product teams  while we haven't done this in the last year, that is an option that we always consider  that especially in bigger agile teams that we occasionally help out and send someone to  a system for some weeks.  Then we have changing new products like the container management product, CASP, also the  cloud application platform that is using Cloud Foundry based on top of CASP, on top of OpenStack,  which are very complicated products, quite hard to understand products, quite hard to  get an overview over, and of course, in turn, making these also hard to make secure products.  But this is something that we will also be looking into in the next years even more closely.  One side project that we are doing is certifications.  One of this is directly security related.  This is the DISA stick.  Stick stands for Secure Technical Implementation Guide and is just another formalized word  for hardening guide.  It's really a formalized thing largely coordinated by the US government and especially the US  military.  They really describe in a step-by-step formal approach how to secure a system.  They have various methods that can be handed out and a textual description form where you  have 200, 300 rules that the administrator goes through, checks, makes the changes, gets  the next rule, checks, makes the changes, and so on.  Or it can be done in an automated format.  In a so-called benchmark or SCAP format, SCAP is Secure Content Automation Protocol.  And we are also currently working and providing this to our enterprise customers, but the  same can only with some small adjustments also likely deployed on open zoos.  So it's basically what we had earlier as hardening guides in a very formal textual form and  later on in a XML form.  We also do cryptographic certifications.  Just recently the NIST, also another US government body, released the FIPS 140-3 standard, basically  describing crypto parameters that your cryptographic system has to fulfill.  And also we plan to work on a common-quarantia certification, which we did for the last enterprise  servers.  We also plan to do that for the enterprise server 15.  The common-quarantia certification describes the whole development process and the product  that comes out of it.  So how are we publicly visible, accepting of course the service we provide to open zoos.  What we do benefits all the upstream open source projects.  They get more secure, so even other vendors benefit from our work.  We are recently participating in the Nürnberg IT security meetup called 0911.org.  And we are hosting every second instance at our offices.  And we are presenting occasionally at conferences like this or other security conferences.  We are planning on extending our presence more as we are growing.  Currently it's not that much, but we are trying to improve on that.  So that's it from my side.  Good on our landing page and our main contact address where you can address security questions  here and that we are looking for some good engineers.  Any questions?  So I will be hanging around here if you have questions later on.  And thank you for listening.
openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_0_155.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_156_299.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_300_803.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_804_2267.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_2268_4967.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_4968_5027.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_5028_8699.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_8700_12551.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_12552_15023.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_15024_15827.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_15828_16031.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_16032_18227.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_18228_19319.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_19320_21647.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_21648_25211.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_25212_28331.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_28332_30947.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_30948_34787.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_34788_35711.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_35712_36839.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_36840_40175.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_40176_43247.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_43248_47627.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_47628_47675.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_47676_47735.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_47736_47783.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_47784_49367.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_49368_51899.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_51900_55679.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_55680_57479.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_57480_60623.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_60624_61835.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_61836_61883.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_61884_61943.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_61944_62003.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_62004_62063.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_62064_62123.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_62124_62183.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_62184_62231.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_62232_62279.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_62280_62327.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_62328_62375.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_62376_62447.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_62448_62507.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_62508_62579.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_62580_62639.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_62640_62711.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_62712_62771.png openSUSE_Conference_2019_-_SUSE_Security_retrospec_F90Eggya0dc_62772_62920.png