Trends in Open Source Security
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Weimer, Florian
When dealing with free and open-source software, we have to work together with reporters, upstream developers and other distributions to protect end users from security threats. For distributions, it is a challenge to deal with a huge collection of software packages, both internally and externally developed, employing many different development procedures. This talk looks at best practices which emerged for vulnerability tracking. Tracking already reported vulnerabilities is only one aspect, however. We discuss tool-chain based hardening features (which can sometimes turn vulnerabilities exploitable for code execution into mere crashers), some remaining low-hanging fruits in this area, and more radical approaches for avoiding low-level vulnerabilities related to memory safety. Some of the APIs we provide are difficult to use, and we look at ways to detect API misuse statically, across an entire distribution.
 I'm going to talk about three main areas. Vulnerability tracking, how we can as a large distribution of upstream software track vulnerabilities in the upstream software we ship. Then about two-chain hardening and the final part about distribution-wide defect analysis. Vulnerability tracking these days for all major linux distributions is cv-based. cv-common vulnerabilities and exposures is a project started by mitre and it aims to assign a unique identifier to every publicly disclosed vulnerability. I've put up an example for such an identifier. Basically it's just cv-the year-increasing number within that year. The cv project doesn't create a taxonomy. It's more like bird ringing in the sense that onyphologists put on little rings with numbers on the legs of birds so they can track them. Cv-e is aimed at identifying vulnerabilities in concrete software products, in concrete versions, not about general vulnerability classes like buffer overflow exploits or similar. What happens these days is that cv-assignment alerts are not available in the distribution because anybody can download the cv-e data from mitre's website and you can see, okay, compared to yesterday there have been these identifiers assigned and you can start processing these new vulnerabilities. This works very well for public issues and it's run by op-wall called os-s-security where red hats co-execute, watches and assigns cv-e identifiers for requests that come in for assignment so that the vulnerabilities that haven't been received for cv-e identify yet get one. There are also many identifiers assigned by vendors who have their own pool so they can put on a cv-e identifier when they release a secure advisor without coordinating with anyone. These cv-e identifiers often have less than clear public descriptions so we don't really know what's it about. But for this identifier that is assigned through the open process, code is very diligent in making sure that the vulnerabilities are precisely described so that we actually can use these identifiers later on. So how do we use these identifiers? We have to put the identifier in relation to the software we ship and the first aspect is the package. We have to identify that which package is affected and for each branch we ship, we can note a minimum fixed version when we fix the package. This is an approach that is based on version-based vulnerability tracking. First, i wanted to present it as the ultimate approach to vulnerability tracking. It turns out that i have forgotten that it's really, really complicated in some corner cases to make this work. So Debian is doing this for the unstable and stable distributions and we have learned over the years that it is really complicated to cover all eventualities. But it works very well to some extent. Obviously the version-based vulnerability tracking is tied to version numbering scheme and branch models. So you cannot reuse Debian version numbers on Ubuntu or even on open-suzo or Fedora that's not going to work. So here's an example from the same cv identifier as before. That's a vulnerability in Ruby on Rails which is related to YAML parsing. YAML is yet another market language and it has the odd aspect that it's a superset of jason and xml at the same time. The Ruby program has used the YAML parser to pass xml and jason, i think, and it's comparable to php's unserialized or Python's fickle or both storable in the sense that once you serialize an untrusted object, you're almost guaranteed to have arbitrary code execution because YAML deserialization instantiates arbitrary code execution. So here's an example from the Debian version tracking database. And i'm just concentrating on the annotations for the unstable branch and for the squeeze, the stable release. The squeeze release is a very simple and very easy to do. It's a simple branch and for the squeeze, the stable release. The squeeze release had received a security update. So there was a subsequent security update. This one. And it's on the same branch, so we know that it's fixed. So by comparing it with the branch-specific fixed version, for the unfixed versions, we have an older version from testing that was in testing or unstable before the bug was fixed. And here's an older version from the squeeze release branch which predates the fixed version. And the interesting thing is that this data can be used to rate installed packages on the system. So you can use this data to see if you have got vulnerabilities on your system. The other approach for vulnerability tracking that's probably used by most other distributions is just file tracker bugs in your bug tracker. That's what redhead is doing with Fedora and the supported products. Basically, the security response team creates a tracker bug in backzilla.redact.com, which is all the way to the CVE. This tracker bug will be made public after disclosure. And there's also extensive metadata in the whiteboard field of that bug, which is also publicly accessible. And this tracker bug depends on product-specific bugs. Usually, you can only see those for Fedora, not for the internal tracker bugs for the enterprise products, usually not publicly accessible because there's QA information in them. And that's usually not interesting. So what happens when you upload to Fedora using a URL and the tracker bug for Fedora is that the Fedora update system actually feeds back the fixed version information into backzilla. So you have that in a single player. All this is quite different from version-based tracking, but the advantage is it's conceptually much simpler. And you can cover any case because you never actually have to compare version numbers. So even if your version numbering scheme is totally strange, explicit tracker bugs still cover it. So the same vulnerability, and if you pull up the slides later from the web page, you can look at the references there. The URL at the top is the CVE tracker bug. You could actually put the CVE name behind the equals sign, and it would still work because it's an alias. And the Fedora tracking bugs are public, and they are internal bugs for the enterprise products, which are also affected by this vulnerability. And these enterprise products were eventually fixed through the security advisers that are mentioned there. And obviously there's a lot of automation for this to work smoothly. So I believe that you want to gen2, openSUSE, etc. use similar schemes for vulnerability tracking. And the good thing is that most upstream products provide critical information to power this sort of vulnerability tracking. They have the impact analysis and the concrete description of the vulnerability in the security advisory or in the bug tracker. And they are also linked to individual patches or commits in gate or mercurial repositories that fix specific vulnerabilities. So we need, as a Linux distribution, we need these individual patches to be able to prepare security updates because we don't want to bundle months of unrelated development work with the security update because too much could break. So if we don't have this information, we have to reverse engineer it from TAR boards we could download from your website. And this is time consuming. You already have this work. And we would like to spend them on patch review or testing. And as a Linux distribution, we will always distribute isolated security patches and make it quite visible what we are fixing because that's just how we prepare the updates. And some of the analysis is also on the public OSS security list. So by hiding this information, you have not much security advantages. So the important thing here is that we really want to have, we really want to encourage upstreams to make available individual commits in public version control repositories so that the security fixes can be prepared in a timely manner. So one thing that comes up at this point is how can we improve cross distribution information sharing at this point. One thing that's annoying or just a fact of life is the versioning schemes and package names differ greatly between different distributions even though we all ship firefox under some name, we ship gcc, we ship bash, we ship a patchy httpd. Yet the names of the packages are all a bit different and we also encode the upstream versions in different ways into our own versioning systems. But I think in an abstract sense, the CVE to packages mapping could be shared. There's a separate project for that, it's called common platform enumeration that's also from Mitre. This might act as a meet in the middle point where both the distributions can see okay, this CVE applies to this CPE and our own packages to this CPE are such and such and this should make this initial mapping step easier. But apart from that, it's really difficult to automate the versioning part of this, so what is the fixed upstream version look like and this information is really hard to automate unfortunately. But again, please publish your security patches in a publicly accessible version control repository as separate commits. This is something we, as LinkStips distributions, we value it tremendously and we would like to encourage to keep it that way. It's not a trend that vendors go away from publishing individual patches, but there are still a few projects that have always operated by releasing complete tarboards which can obtain unrelated development and not just security fixes and of course there are some projects that go from a separate page point of view to a tarboard based distribution and we don't hope that this trend turns into a trend or causes a problem for us for preparing security updates. So that's it for vulnerability tracking. The next part is toolchain hardening. Basically, that's about changing GCC, bin year deals, the kernel, let's see everything that lies below most programs running on GNU linux and change them to reduce the impact of security vulnerabilities. Most of these countermeasures are probabilistic, countermeasures against code execution. That means that there's always a nonzero probability that an attacker goes through because there's run-in-ness involved and if the attacker guesses it's still going to result in code execution. The countermeasures basically fall into two different areas. One aspect is we try to make it more difficult for the attacker to write at a specific memory location that is that affects the program's control flow. It should not be easy to redirect execution because the memory location is not either not writable at all or it's difficult to locate in the program. The other aspect is that we want to make it more difficult to write a value that has a meaningful impact for the program being attacked and not so that it doesn't result in a crash but something more interesting for the attacker. It's important to keep in mind that these are not complete solutions. What we do is we make the program crash if the countermeasure is effective and code is not executed but it's still a crash and these bugs obviously still need fixing. They are just something that might be lower priority but it's still something we actually want to fix at some point. So these features mentioned on this page are mostly enabled by default just if you use gcc on the tool chain. The first point is address space layout randomization that reorders the shared libraries so they are positioned in the address space. The start address of the stack is randomized. The start address of the heap is randomized. This makes it more difficult to find spaces to write to in a predictable fashion and it also makes it more difficult to get a valid function pointer to code in a shared library because the address has been randomized. There's another aspect about program start addresses. I'm going to cover that in a minute. Then the next thing that you get basically for free is non executable stack and heap. That means that when the attacker supplies code to your program and it's stored on the heap, it is not directly executable because the heap hasn't got the execute bit in the page protection set. That relies on hardware support for x86 that hasn't been in the initial architecture but all chips and cum would use have it so it's basically universally available. Then also some time ago, it must have been several years by now, we have fixed gLib C so that double free bugs where you deallocate the same pointer twice do not resolve in a bug that's directly executable for code execution or heap manipulation that checks in gLib C that detects such direct exploitation attempts. Then there's a stack protector that's about protecting the return address on the stack so that the stack-based stuff overflow that overrides the return address also overrides a stack canary and the canary then is checked before returning from the function if it has the expected value and if it hasn't, we assume it has been overwritten by an attacker and abort the program. This kind of measure you have to enable explicitly but most distributions have tweaked the gcc defaults in a way that this is always active when compiling programs. Then there are additional compiler warnings for format string misuse. Some distributions turn them into errors either during build or in a separate step by examining the build logs and the warnings gcc has written. There are also warnings about some cases of integer overflow that trigger undefined behavior. What's new, what that landed in gcc in last fall is operator harrowing new hardening for c++ programs. They used to be an really old bug disclosed in 2002 which basically means that if you allocate a large array using gcc and operator new, it could happen that the value of wrapped around internally and what you get back from the c++ runtime library is just a very small heap log and then you start writing into the array and you get a heap-based buffer overflow. This has been fixed in gcc 4.8 which is currently under development. It's always active. There's no way to turn it off. The patch has already been back-botted to Fedora 18. In Fedora 18 most critical packages have been rebuilt with gcc and the fix applied. The unfortunate aspect of this vulnerability is that it is encode generated by gcc. You have to recompile everything that might use this c++ functionality and before that you still have a latent exposure to this vulnerability. But it's going to be fixed and gcc 4.8 is going to be released in April or May I think. Or we could just pick up the back-botted patch from Fedora. Another thing about 2-chain hiring, this is completely optional but many distributions in Avalet is called 45 source. The very neat aspect is the first item that's a cooperation between gcc and gLipc. Gcc provides a compiler built in where you can put in a pointer, a bare point of without any length information and gcc uses context from the program source code it sees to infer the length of the array. And nullipc contains wrappers that pass this length information if it's available. It's not always available, of course, to wrapper functions which perform length checks. So what you get there is you have stack-based buffer overflows which reliable result in crashes because the length check kicks in and terminates the program before the buffer overflow actually occurs. And nullipc disables a person's N in writable format strings in 45 source mode. They used to be a back-but this has been fixed. And the second point is disables disarms format string attacks that use person N for writing data in a controlled fetch. So not everything is, we don't enable all hardening opportunities or functionality that is readily available in the through chain right now. For instance, one thing that would help which is obviously on the way is to get rid of 32-bit architectures because 32-bit are just too little, contain too little entropy to get full randomization and full unpredictable heap and addresses or shared library addresses. That's also another aspect because size T is 64-bits in 64-bit architectures. You often have integer overflows that result in buffer overflows which are only exploitable on 32-bit architectures because on 64-bit architectures the modification is in 64-bits and just doesn't overflow there. So getting rid of 32-bits is one step that's going to help me eventually and will help us somewhat in this area. Another thing that's actually pretty easy to do but Fedora, for example, hasn't done is disable prilling. Prilling disables address space layout and address randomization because it picks a single fixed value that is valid until you run the tool the next time on this system. So it's not entirely predictable. It still probably works quite well against network-based attacks where the attacker doesn't see the addresses on the local system but it doesn't work against local attackers who can just check what are the addresses used on this system by prilling and use disinformation for further attacks. What we should do is randomization of program start addresses. There is one point where we do not fully randomize executable pages in the process image. Basically all processes have a start address which is four and a couple of zeros. That's a constant and the fear is that the actual program itself contains enough machine code to provide targets for return-oriented programming. So there is the potential of address-based layout randomization bypass there. This hasn't been implemented because there is a concern for overhead but Grant Murphy has recently done some analysis and it looks like we should really do this and enable it for everything and not just critical network services like OpenSSH. Then another thing we can do which is readily available is bind now. That's a feature of the LDSO dynamic linker. It instructs the linker to perform all symbol resolution at program start and then make the global offset table read only. This means that the global offset table cannot be overwritten by attackers and because it contains function pointers among other things, it is really important to get this protection. What we also can do but which doesn't seem to be much on the radar is we could compile that the package will have a wrap v. This makes integer overflows in C deterministic using two compliments arithmetic but this only helps broken programs which use incorrect ways for overflow checking in C or C++. So it's not really compelling to enable that because it interferes with loop optimization but maybe we can provide the bulletin to enable that as well. The next point that's also already in GCC has been for a while. That's stack checking. That's about stack-based buffer overflows or stack-point manipulation resulting from un-loaded allocator or variable length array allocation on the stack. Basically what happens is that I call to allocate with a large argument, moves the stack pointer from within the allocated stack, completely outside the stack where anything can happen. Most of the time there's going to be a crash but it's conceivable that this is exploitable and I think it has been demonstrated recently that these things can result in arbitrary code execution. The state right now is that fstack-tech has considerable code size impact because every call is instrumented even though the stack frame is small and doesn't contain allocator so we really need to work on that and get the code size impact down and then we can start looking at performance impact in more detail. There's a backup plan if the current approach about which writes probe values on a sequence on the stack and hopes to reach the guard page on a large allocator which will kill the process. If this approach with the loop and the writing is too expensive for allocator, we have as a support back plan we probably could stack boundaries provided by split stacks without actually enabling split stacks. We would only use the stack boundary that is used by split stacks to detect that we are too close to the top of the stack to fulfill the allocation request. I hope that we will be able to tackle this problem this year because there's so much code out there that uses allocator and so far attempts to discourage its use have not been received by programmers because they like it. It's an easy interface. You don't have to call free unlike malloc but it obviously has the security risk and so we have to fix it in the tool chain. One thing that I'd like to see fixed as well is the JavaScript checking for operator brackets in c++ that affects standard vectors, standard string and standard array. If you use this shorthand notation, it's basically currently with the glipsy, with the gcc, it's as safe as c programs because there's no range checking even though the runtime library knows how long the array is and could insert the size checks as it is done for java or python or many other programming languages. The c++ standard actually gives us permission to do this check but we try to make the change to the library and implemented it and it seemed to have a performance impact on tidal loops so we had to put it on the backbone of a now and I think that's first the research needed how we can address this and in the meantime if you're a c++ programmer and want to sub-script check, you have to use the add member function which is the same as the sub-script operator but it's required to perform the sub-script check. So this is something where we still have to do some work, I think. So one aspect that makes it difficult for us to talk about these hardening features is that we have to decide how much performance impact we want to take in order to improve security and what I'm trying to show here is we have a countermeasure that we implement and the damage just stays unchanged and now we deploy an effective countermeasure and the damage from attacks we measure goes way down and then comes a new attack and goes up. The problem with that is we lack data on detected attacks at a large scale because we currently are on the target in the sense that others are. It would be different if Oracle had open-source the java plugin, but fortunately for us that's kind of autoscoped. Then we would have this data available, what changes we make and what changes we see in attack success. But lack of this data makes it really difficult to judge security improvements in the tool chain in an objective way. So here's a quick list of things we could do if we feel that we want to make things take it to a new level. There are some things we can do to cc++ that probably change the language quite a lot and require probably standard works. So it's a very long-term project. I'm not sure if that's going to happen anytime soon. Equally, unrealistic is library consolidation. That's a thing that should happen, but it's unlikely to materialize. I think most linux distributions ship four or five different implementations of ssh. That's something that for a complex crypto protocol that's not going to help to bring down security bugs. One thing that might help is Floss specific security coding guidelines. There have been a couple of books about secure coding, but they usually do not address new linux needs and are pretty much out of date by now. So we probably have to revive that. And eventually all this experience we gained from fix and run abilities and documenting risky library calls should probably lead to better apis at one point, but that's also a rather long-term. And of course we can go completely crazy and rewrite everything in a different programming language. And curiously, that's not going to, it's completely unpractical right now because most of the new system programming languages are just not deeply embedded in the sense that we can replace code in a library and the calling program doesn't notice that it's no longer C but Rust. For example, the Netscape security services library could eventually use Rust for implementing critical functionality, but right now neither of the languages except Ada and maybe LuaJit qualify for this kind of deep embedding. And Ada is probably not to everyone's taste, but there are tremendous, there's a tremendous amount of tools available for Ada which help to build secure software and perhaps we should look into that more. And I understand Ada has a huge falling of programmers here at FOSTEM, so maybe that's going to be interesting. So my final topic is distribution-wide defect analysis. This is about using tools to find probabilities across a whole distribution. And what I did for preparing this slide is we ran a proprietary static analyzer on all change packages in reted enterprise linux 6 which were released in 2012. And we looked at the changes reported by the static analyzers report a huge number of potential issues. And we looked at the before-after state for each reported bug and found interesting, we found actual matches for vulnerabilities and bugs we have fixed in erata during 2012. Only the first one is a real security bug, the other ones were regular bugs. And the first one actually was caught by the 45 source buffer checking logic because GCC has its own static analyzer implemented. So this is something we should gather more data on it and there's a FEDOR project, yes, the URL which tries to gather all kinds of information about open source static analysis tools so we can use them in Fedora and actually tweak them so that they match closely to the apis we use and provide us with good information about our code. One thing that I find very interesting is matching machines and humans and coupling their different skill sets. And one thing that you can do is you can take a search engine what actually is happening right now a lot and search for certain patterns and then manually go through the hits and identify potential security issues that are related to the search string. For example, the ruby on race vulnerability i mentioned earlier could be located in using this string and this code point to code copies that have similar vulnerabilities. But code search engines are sort of difficult to set up so Vincent Danin and later myself have written an indexer for an L symbol database. I think what we do is we unpack RPM packages, pipe them through an L parser and load the results into a PostgreSQL database. That has really interesting applications. You can use the full power of PostgreSQL joins and anti-joins to use and use them to find library codes that are highly likely to be vulnerable to specific attacks like... The first example is from xpad where you call it xml parser create but do not restrict the entity expansion using xml set entity. This means that a small xml file you download from the net can blow up in your process into multi-gigabyte xml structure. This one is pretty complicated but can be implemented in SQL. You can implement the dynamic linker or model the dynamic linker using recursive queries in PostgreSQL. Then you can find insecure users of getend from PAM and nss namesales switch modules. This is something that could lead to privilege escalation because the module, the PAM module, the nss module could run in a privilege process such as su or parseweedy. I think for global analysis we could do much more if we had better tools. We could use debugging information and something we could do is we could actually parse the compiled code and extract further information from disassembly. It would be nice to cover additional languages and for dynamic languages like Python or PHP we probably need Horistix to gather good information about the functions which are being called by the programs. What I hope to have eventually is to have combined the f symbol database with something that goes through the binaries and extracts function arguments for example and uses that to validate, to discover certain types of vulnerabilities like unsecured umask and demons or accidentally disabled ssl certificate validation or the last point is about path max bugs because path max actually isn't a hard limit on linux. We could use the f symbol databases and index to locate interesting points and then use disassembly to extract the function arguments and actually find all these codes in a reliable manner. So that is for me. So I think the two things you should take home is we should try to fix allocate this here and static analysis and code search engines are exciting and please share your version control repository so that we can use that to prepare security updates. Okay. That's it for me. If there are any questions. Thank you. Is there any question? One, two, three. Okay then. Thank you.
 I'm going to talk about three main areas.  Vulnerability tracking, how we can as a large distribution of  upstream software track vulnerabilities in the upstream  software we ship. Then about two-chain hardening and  the final part about distribution-wide defect analysis.  Vulnerability tracking these days for all major linux  distributions is cv-based. cv-common vulnerabilities and  exposures is a project started by mitre and it aims to  assign a unique identifier to every publicly disclosed  vulnerability. I've put up an example for  such an identifier. Basically it's just cv-the year-increasing  number within that year. The cv project doesn't create a  taxonomy. It's more like bird ringing in the sense that  onyphologists put on little rings with numbers on the  legs of birds so they can track them. Cv-e is aimed at  identifying vulnerabilities in concrete software products,  in concrete versions, not about general vulnerability classes  like buffer overflow exploits or similar.  What happens these days is that cv-assignment alerts  are not available in the distribution because anybody  can download the cv-e data from mitre's website and you can  see, okay, compared to yesterday there have been these  identifiers assigned and you can start processing these new  vulnerabilities. This works very well for public issues and  it's run by op-wall called os-s-security where red hats  co-execute, watches and assigns cv-e identifiers for  requests that come in for assignment so that the vulnerabilities  that haven't been received for cv-e identify yet get one.  There are also many identifiers assigned by vendors who have  their own pool so they can put on a cv-e identifier when they  release a secure advisor without coordinating with anyone.  These cv-e identifiers often have less than clear public  descriptions so we don't really know what's it about.  But for this identifier that is assigned through the open  process, code is very diligent in making sure that the  vulnerabilities are precisely described so that we actually  can use these identifiers later on.  So how do we use these identifiers?  We have to put the identifier in relation to the software we  ship and the first aspect is the package.  We have to identify that which package is affected and for  each branch we ship, we can note a minimum fixed version when  we fix the package.  This is an approach that is based on version-based vulnerability  tracking. First, i wanted to present it as the ultimate  approach to vulnerability tracking.  It turns out that i have forgotten that it's really,  really complicated in some corner cases to make this work.  So Debian is doing this for the unstable and  stable distributions and we have learned over the years that  it is really complicated to cover all eventualities.  But it works very well to some extent.  Obviously the version-based vulnerability tracking is tied to  version numbering scheme and branch models.  So you cannot reuse Debian version numbers on Ubuntu or  even on open-suzo or Fedora that's not going to work.  So here's an example from the same cv identifier as before.  That's a vulnerability in Ruby on Rails which is related to  YAML parsing. YAML is yet another market language and it has  the odd aspect that it's a superset of jason and xml at the  same time. The Ruby program has used the YAML  parser to pass xml and jason, i think, and it's  comparable to php's unserialized or Python's fickle or  both storable in the sense that once you serialize an  untrusted object, you're almost guaranteed to have arbitrary code  execution because YAML deserialization  instantiates arbitrary code execution.  So here's an example from the Debian version tracking database.  And i'm just concentrating on the annotations for the  unstable branch and for the squeeze, the stable release.  The squeeze release is a very simple and very easy to do.  It's a simple branch and for the squeeze, the stable release.  The squeeze release had received a security update.  So there was a subsequent security update.  This one. And it's on the same branch, so we know that it's fixed.  So by comparing it with the branch-specific fixed version,  for the unfixed versions, we have an older version from  testing that was in testing or unstable before the bug was  fixed. And here's an older version from the squeeze  release branch which predates the fixed version.  And the interesting thing is that this data can be used to  rate installed packages on the system.  So you can use this data to see if you have got vulnerabilities  on your system.  The other approach for vulnerability tracking that's  probably used by most other distributions is just file  tracker bugs in your bug tracker.  That's what redhead is doing with Fedora and the supported  products. Basically, the security response team  creates a tracker bug in backzilla.redact.com, which is  all the way to the CVE. This tracker bug will be made public  after disclosure. And there's also extensive metadata in the  whiteboard field of that bug, which is also publicly accessible.  And this tracker bug depends on product-specific bugs.  Usually, you can only see those for Fedora, not for the  internal tracker bugs for the enterprise products,  usually not publicly accessible because there's QA  information in them. And that's usually not interesting.  So what happens when you upload to Fedora using a URL and  the tracker bug for Fedora is that the Fedora update system  actually feeds back the fixed version information into  backzilla. So you have that in a single player.  All this is quite different from version-based tracking, but  the advantage is it's conceptually much simpler.  And you can cover any case because you never actually have  to compare version numbers. So even if your version numbering  scheme is totally strange, explicit tracker bugs still  cover it. So the same vulnerability, and if you  pull up the slides later from the web page, you can look at  the references there. The URL at the top is the CVE  tracker bug. You could actually put the CVE name behind the  equals sign, and it would still work because it's an alias.  And the Fedora tracking bugs are public, and they are  internal bugs for the enterprise products, which are also  affected by this vulnerability. And these enterprise products  were eventually fixed through the security advisers that are  mentioned there. And obviously there's a lot of automation  for this to work smoothly. So I believe that you want to  gen2, openSUSE, etc. use similar schemes for vulnerability  tracking. And the good thing is that most upstream  products provide critical information to power this sort  of vulnerability tracking. They have the impact analysis and  the concrete description of the vulnerability in the security  advisory or in the bug tracker. And they are also linked to  individual patches or commits in gate or mercurial repositories  that fix specific vulnerabilities. So we need, as a Linux  distribution, we need these individual patches to be able to  prepare security updates because we don't want to bundle months  of unrelated development work with the security update because  too much could break. So if we don't have this information, we  have to reverse engineer it from TAR boards we could download  from your website. And this is time consuming. You already have  this work. And we would like to spend them on patch review or  testing. And as a Linux distribution, we will always  distribute isolated security patches and make it quite  visible what we are fixing because that's just how we  prepare the updates. And some of the analysis is also on the  public OSS security list. So by hiding this information, you  have not much security advantages. So the important  thing here is that we really want to have, we really want to  encourage upstreams to make available individual commits in  public version control repositories so that the security fixes  can be prepared in a timely manner. So one thing that comes up  at this point is how can we improve cross distribution  information sharing at this point. One thing that's annoying or  just a fact of life is the versioning schemes and package  names differ greatly between different distributions even  though we all ship firefox under some name, we ship gcc,  we ship bash, we ship a patchy httpd. Yet the names of the  packages are all a bit different and we also encode the  upstream versions in different ways into our own versioning  systems. But I think in an abstract sense, the CVE to  packages mapping could be shared. There's a separate project  for that, it's called common platform enumeration that's  also from Mitre. This might act as a meet in the middle point  where both the distributions can see okay, this CVE  applies to this CPE and our own packages to this CPE are such  and such and this should make this initial mapping step  easier. But apart from that, it's really difficult to automate  the versioning part of this, so what is the fixed upstream  version look like and this information is really hard to  automate unfortunately. But again, please publish your  security patches in a publicly accessible version control  repository as separate commits. This is something we, as  LinkStips distributions, we value it tremendously and we would  like to encourage to keep it that way. It's not a trend that  vendors go away from publishing individual patches, but there  are still a few projects that have always operated by  releasing complete tarboards which can obtain unrelated  development and not just security fixes and of course there are  some projects that go from a separate page point of view to  a tarboard based distribution and we don't hope that this trend  turns into a trend or causes a problem for us for preparing  security updates. So that's it for vulnerability tracking.  The next part is toolchain hardening. Basically, that's about  changing GCC, bin year deals, the kernel, let's see everything  that lies below most programs running on GNU linux and  change them to reduce the impact of security vulnerabilities.  Most of these countermeasures are probabilistic, countermeasures  against code execution. That means that there's always a  nonzero probability that an attacker goes through because  there's run-in-ness involved and if the attacker guesses  it's still going to result in code execution.  The countermeasures basically fall into two different areas.  One aspect is we try to make it more difficult for the attacker  to write at a specific memory location that is  that affects the program's control flow.  It should not be easy to redirect execution because the memory  location is not either not writable at all or it's difficult to  locate in the program. The other aspect is that we want to  make it more difficult to write a value that has a meaningful  impact for the program being attacked and not so that it  doesn't result in a crash but something more interesting for  the attacker. It's important to keep in mind that these are  not complete solutions. What we do is we make the program  crash if the countermeasure is effective and code is not  executed but it's still a crash and these bugs obviously still  need fixing. They are just something that might be  lower priority but it's still something we actually want to fix  at some point. So these features mentioned on this page are  mostly enabled by default just if you use gcc on the tool  chain. The first point is address space  layout randomization that reorders the shared libraries  so they are positioned in the address space. The start address  of the stack is randomized. The start address of the heap is  randomized. This makes it more difficult to find spaces to write  to in a predictable fashion and it also makes it more  difficult to get a valid function pointer to code in a  shared library because the address has been randomized.  There's another aspect about program start addresses. I'm  going to cover that in a minute. Then the next thing that you  get basically for free is non executable stack and heap.  That means that when the attacker supplies code to your  program and it's stored on the heap, it is not directly  executable because the heap hasn't got the execute bit in the  page protection set. That relies on hardware support for  x86 that hasn't been in the initial architecture but all  chips and cum would use have it so it's basically universally  available. Then also some time ago, it must have been  several years by now, we have fixed gLib C so that double  free bugs where you deallocate the same pointer twice do not  resolve in a bug that's directly executable for code  execution or heap manipulation that checks in gLib C that  detects such direct exploitation attempts. Then there's a stack  protector that's about protecting the return address on the stack  so that the stack-based stuff overflow that overrides the  return address also overrides a stack canary and the canary  then is checked before returning from the function if it has  the expected value and if it hasn't, we assume it has been  overwritten by an attacker and abort the program. This kind of  measure you have to enable explicitly but most distributions  have tweaked the gcc defaults in a way that this is always  active when compiling programs. Then there are additional  compiler warnings for format string misuse. Some distributions  turn them into errors either during build or in a separate  step by examining the build logs and the warnings gcc has  written. There are also warnings about some cases of  integer overflow that trigger undefined behavior. What's  new, what that landed in gcc in last fall is operator  harrowing new hardening for c++ programs. They used to be  an really old bug disclosed in 2002 which basically means that  if you allocate a large array using gcc and operator  new, it could happen that the value of wrapped around  internally and what you get back from the c++ runtime library  is just a very small heap log and then you start writing  into the array and you get a heap-based buffer overflow.  This has been fixed in gcc 4.8 which is currently under  development. It's always active. There's no way to turn it  off. The patch has already been back-botted to Fedora 18.  In Fedora 18 most critical packages have been rebuilt  with gcc and the fix applied. The unfortunate aspect of  this vulnerability is that it is encode generated by gcc.  You have to recompile everything that might use this  c++ functionality and before that you still have  a latent exposure to this vulnerability. But it's going to  be fixed and gcc 4.8 is going to be released in April or  May I think. Or we could just pick up the  back-botted patch from Fedora. Another thing about 2-chain  hiring, this is completely optional but many distributions  in Avalet is called 45 source. The very neat aspect is the  first item that's a cooperation between gcc and  gLipc. Gcc provides a compiler built in where you can put in  a pointer, a bare point of without any length information  and gcc uses context from the program source code it sees to  infer the length of the array. And nullipc contains  wrappers that pass this length information if it's  available. It's not always available, of course, to  wrapper functions which perform length checks. So what you  get there is you have stack-based buffer  overflows which reliable result in crashes because the  length check kicks in and terminates the program before  the buffer overflow actually occurs. And nullipc  disables a person's N in writable format strings in  45 source mode. They used to be a back-but this has been  fixed. And the second point is disables disarms format  string attacks that use person N for writing data in a  controlled fetch. So not everything is, we don't  enable all hardening opportunities or functionality that  is readily available in the through chain right now.  For instance, one thing that would help which is obviously  on the way is to get rid of 32-bit architectures because  32-bit are just too little, contain too little  entropy to get full randomization and full  unpredictable heap and addresses or shared library  addresses. That's also another aspect because size T is  64-bits in 64-bit architectures. You often have  integer overflows that result in buffer overflows which  are only exploitable on 32-bit architectures because on 64-bit  architectures the modification is in 64-bits and just  doesn't overflow there. So getting rid of 32-bits is one  step that's going to help me eventually and will help us  somewhat in this area. Another thing that's actually pretty  easy to do but Fedora, for example, hasn't done is disable  prilling. Prilling disables address space layout  and address randomization because it picks a single fixed  value that is valid until you run the tool the next time on  this system. So it's not entirely predictable. It still  probably works quite well against network-based attacks  where the attacker doesn't see the addresses on the local  system but it doesn't work against local attackers who  can just check what are the addresses used on this system  by prilling and use disinformation for further attacks.  What we should do is randomization of program start  addresses. There is one point where we do not fully  randomize executable pages in the process image.  Basically all processes have a start address which is  four and a couple of zeros. That's a constant and the fear  is that the actual program itself contains enough  machine code to provide targets for return-oriented  programming. So there is the potential of address-based  layout randomization bypass there. This hasn't been  implemented because there is a concern for overhead but  Grant Murphy has recently done some analysis and it looks  like we should really do this and enable it for everything  and not just critical network services like OpenSSH.  Then another thing we can do which is readily available  is bind now. That's a feature of the LDSO dynamic  linker. It instructs the linker to perform all symbol  resolution at program start and then make the global offset  table read only. This means that the global offset  table cannot be overwritten by attackers and because it  contains function pointers among other things, it is  really important to get this protection. What we also can do  but which doesn't seem to be much on the radar is we could  compile that the package will have a wrap v. This makes  integer overflows in C deterministic using two  compliments arithmetic but this only helps broken  programs which use incorrect ways for overflow checking  in C or C++. So it's not really compelling to enable that  because it interferes with loop optimization but maybe we  can provide the bulletin to enable that as well. The next  point that's also already in GCC has been for a while.  That's stack checking. That's about stack-based  buffer overflows or stack-point manipulation resulting  from un-loaded allocator or variable length array  allocation on the stack. Basically what happens is that  I call to allocate with a large argument, moves the stack  pointer from within the allocated stack, completely  outside the stack where anything can happen. Most of the time  there's going to be a crash but it's conceivable that this is  exploitable and I think it has been demonstrated recently  that these things can result in arbitrary code execution.  The state right now is that fstack-tech has considerable  code size impact because every call is instrumented even  though the stack frame is small and doesn't contain allocator  so we really need to work on that and get the code size  impact down and then we can start looking at performance  impact in more detail. There's a backup plan if the  current approach about which writes probe values on  a sequence on the stack and hopes to reach the guard page  on a large allocator which will kill the process.  If this approach with the loop and the writing is too  expensive for allocator, we have as a support back plan  we probably could stack boundaries provided by split stacks  without actually enabling split stacks. We would only use the  stack boundary that is used by split stacks to detect  that we are too close to the top of the stack to fulfill  the allocation request. I hope that we will be able to  tackle this problem this year because there's so much code  out there that uses allocator and so far attempts to  discourage its use have not been received by programmers  because they like it. It's an easy interface. You don't have to call  free unlike malloc but it obviously has the security  risk and so we have to fix it in the tool chain.  One thing that I'd like to see fixed as well is  the JavaScript checking for operator brackets in c++  that affects standard vectors, standard string and standard  array. If you use this shorthand notation, it's basically  currently with the glipsy, with the gcc, it's as safe as  c programs because there's no range checking even though the  runtime library knows how long the array is and could insert  the size checks as it is done for java or python or many  other programming languages. The c++ standard actually gives  us permission to do this check but we try to make the change  to the library and implemented it and it seemed to have a  performance impact on tidal loops so we had to put it on the  backbone of a now and I think that's first the research needed  how we can address this and in the meantime if you're a c++  programmer and want to sub-script check, you have to use the  add member function which is the same as the sub-script  operator but it's required to perform the sub-script check.  So this is something where we still have to do some work, I  think. So one aspect that makes it difficult for us to talk  about these hardening features is that we have to decide how  much performance impact we want to take in order to improve  security and what I'm trying to show here is we have a  countermeasure that we implement and the damage just stays  unchanged and now we deploy an effective countermeasure and the  damage from attacks we measure goes way down and then comes a  new attack and goes up. The problem with that is we lack data  on detected attacks at a large scale because we currently are  on the target in the sense that others are.  It would be different if Oracle had open-source the  java plugin, but fortunately for us that's kind of  autoscoped. Then we would have this data available, what  changes we make and what changes we see in attack success.  But lack of this data makes it really difficult to judge  security improvements in the tool chain in an objective way.  So here's a quick list of things we could do if we feel that  we want to make things take it to a new level.  There are some things we can do to cc++ that probably change  the language quite a lot and require probably standard  works. So it's a very long-term project.  I'm not sure if that's going to happen anytime soon.  Equally, unrealistic is library consolidation.  That's a thing that should happen, but it's unlikely to  materialize. I think most linux distributions  ship four or five different implementations of ssh.  That's something that for a complex crypto protocol that's  not going to help to bring down security bugs.  One thing that might help is Floss specific security  coding guidelines. There have been a couple of books about  secure coding, but they usually do not address  new linux needs and are pretty much out of date by now.  So we probably have to revive that.  And eventually all this experience we gained from fix and  run abilities and documenting risky library calls should  probably lead to better apis at one point, but that's also  a rather long-term. And of course we can go completely  crazy and rewrite everything in a different programming  language. And curiously, that's not going to,  it's completely unpractical right now because most of the new  system programming languages are just not deeply  embedded in the sense that we can replace code in a library  and the calling program doesn't notice that it's no longer  C but Rust. For example, the Netscape security services  library could eventually use Rust for implementing critical  functionality, but right now neither of the languages except  Ada and maybe LuaJit qualify for this kind of deep embedding.  And Ada is probably not to everyone's taste, but there are  tremendous, there's a tremendous amount of tools available  for Ada which help to build secure software and perhaps we  should look into that more. And I understand Ada has a huge  falling of programmers here at FOSTEM, so maybe that's going  to be interesting. So my final topic is distribution-wide  defect analysis. This is about using tools to find  probabilities across a whole distribution. And what I did for  preparing this slide is we ran a proprietary static analyzer  on all change packages in reted enterprise linux 6 which were  released in 2012. And we looked at the changes reported by the  static analyzers report a huge number of potential issues.  And we looked at the before-after state for each reported  bug and found interesting, we found actual matches for  vulnerabilities and bugs we have fixed in erata during 2012.  Only the first one is a real security bug, the other ones were  regular bugs. And the first one actually was caught by the  45 source buffer checking logic because GCC has its own static  analyzer implemented. So this is something we should gather  more data on it and there's a FEDOR project, yes, the URL which  tries to gather all kinds of information about open source  static analysis tools so we can use them in Fedora and actually  tweak them so that they match closely to the apis we use and  provide us with good information about our code.  One thing that I find very interesting is matching machines  and humans and coupling their different skill sets.  And one thing that you can do is you can take a search engine  what actually is happening right now a lot and search for  certain patterns and then manually go through the hits and  identify potential security issues that are related to the  search string. For example, the ruby on race vulnerability  i mentioned earlier could be located in using this string  and this code point to code copies that have similar  vulnerabilities. But code search engines are sort of  difficult to set up so Vincent Danin and later myself have  written an indexer for an L symbol database.  I think what we do is we unpack RPM packages, pipe them through  an L parser and load the results into a PostgreSQL database.  That has really interesting applications.  You can use the full power of PostgreSQL joins and anti-joins  to use and use them to find library codes that are highly  likely to be vulnerable to specific attacks like...  The first example is from xpad where you call it xml  parser create but do not restrict the entity expansion using  xml set entity. This means that a small xml file you  download from the net can blow up in your process into  multi-gigabyte xml structure.  This one is pretty complicated but can be implemented in  SQL. You can implement the dynamic linker or model the  dynamic linker using recursive queries in PostgreSQL.  Then you can find insecure users of getend from PAM and  nss namesales switch modules. This is something that could  lead to privilege escalation because the module, the PAM  module, the nss module could run in a privilege process such  as su or parseweedy.  I think for global analysis we could do much more if we  had better tools. We could use debugging information and  something we could do is we could actually parse the  compiled code and extract further information from  disassembly. It would be nice to cover additional languages  and for dynamic languages like Python or PHP we probably  need Horistix to gather good information about the  functions which are being called by the programs.  What I hope to have eventually is to have combined the f  symbol database with something that goes through the  binaries and extracts function arguments for example and  uses that to validate, to discover certain types of  vulnerabilities like unsecured umask and demons or  accidentally disabled ssl certificate validation or  the last point is about path max bugs because path max  actually isn't a hard limit on linux. We could use the f  symbol databases and index to locate interesting points and  then use disassembly to extract the function arguments and  actually find all these codes in a reliable manner.  So that is for me.  So I think the two things you should take home is we should  try to fix allocate this here and static analysis and code  search engines are exciting and please share your version control  repository so that we can use that to prepare security  updates. Okay. That's it for me.  If there are any questions.  Thank you.  Is there any question?  One, two, three.  Okay then. Thank you.
Trends_in_Open_Source_Security_98TmFU6Lpyk_2_0_4727.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_4728_5615.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_5616_5999.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_6000_6587.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_6588_7811.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_7812_8327.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_8328_11087.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_11088_12539.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_12540_13223.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_13224_13463.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_13464_14663.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_14664_14759.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_14760_15803.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_15804_16199.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_16200_16895.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_16896_17003.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_17004_17063.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_17064_17111.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_17112_17195.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_17196_17543.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_17544_17903.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_17904_17987.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_17988_18263.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_18264_18443.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_18444_20951.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_20952_21119.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_21120_23687.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_23688_24815.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_24816_25127.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_25128_25307.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_25308_25379.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_25380_34355.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_34356_35387.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_35388_36491.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_36492_37331.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_37332_39935.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_39936_48551.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_48552_48671.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_48672_49223.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_49224_49427.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_49428_49511.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_49512_49799.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_49800_59651.png Trends_in_Open_Source_Security_98TmFU6Lpyk_2_59652_71491.png