IoT Village - IoT under the Microscope
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Wiksell, Parker
IoT device manufacturers have no idea what's running on their devices - they really don't. In 2002 then-US Secretary of Defense, Donald Rumsfield, brought public attention to a notion that information can be divided into three categories: known knowns, known unknowns, and unknown unknowns. As hackers, how can we apply this formulation to IoT vulnerabilities? The known knowns: Vulnerabilities that have been explicitly discovered through scanning and testing. The known unknowns: Newly created software that has yet to undergo any application security testing. The unknown unknowns: Systems that the defender does not know about. There is, in fact, a fourth dimension: unknown knowns, which comprise “that which we intentionally refuse to acknowledge that we know” or “do not like to know.” The unknown knowns: Vulnerabilities that are known to exist, but that have not been associated with all the systems they actually affect. In this talk, we report on IoT device vulnerability findings at massive scale, as a result of our firmware collection and analysis. For this research we have selected approximately 50k firmware images, representing over 7M files, 10k products, and 150 vendors, spanning many different architectures and operating systems. We will highlight some of the trends we've uncovered in supply chain vulnerabilities, and reveal specific examples of device backdoors, botnets, and vulnerabilities discovered in medical, home, and commercial device firmware.
 All right. All right, all right, all right. What's up, y'all? All right, welcome to my talk IOT under the microscope here virtually. It's kind of disappointing. I'm looking at a screen here instead of all you in the audience, but hopefully we'll get through this and we'll have a good time doing it. I want to talk about vulnerability trends in the supply chain. I've got some very interesting things I think that we found in our data set size. So hopefully you guys will learn something here and we'll have a little fun while doing it. All right. Okay. So who am I? I'm Parker Wixel. I'm born and raised here in Columbus, Ohio. I've got 25 years industry experience on cybersecurity, on software development, full stack development, last 10 years of which have really been focused on cybersecurity research and product development. I was a contributor and developer of open source security projects like AFL Unicorn. It's a fuzzing framework for emulated binaries and then Patchwork, which is a statics compilation of Linux kernels or patching of Linux kernels for debugging purposes. Like it was mentioned, I'm a senior engineer at Finite State. We're an IoT cybersecurity firm dealing with a lot of the topics that we're talking about here and all my data sets and stuff like that come from our Finite State repos and some of the products that we're working on here. I'm also a database lecturer over at the Ohio State University looking to kick off yet another fall there and then I'm a composer and a musician. Don't hold that against me. I realize early on that there's not a lot of money in music. Here I am on just another passion of mine, computers. Why is this talk relevant to your interests? We're going to be talking about supply chain trends, vulnerable and not vulnerable, vulnerability standards and reporting, and then some firmware statistics and observations. Probably about the first half of the talk is going to be delving into the background of what supply chains are, some of the ways that we have to talk about vulnerabilities, what the supply chain introduces as far as vulnerabilities and visibility into those. Then probably the last half will be delving into the fund numbers, probably why you came to see this talk. Nevertheless, hopefully we learned something in both parts. Our dataset that I'm pulling from right now, we do have partnerships with some private industry partners. We do not include all of our private repos and stuff like that, but for this particular talk I've got about 7 million files, represents about 50,000 firmware images, 10,000 distinct product lines, and 150 different vendors. This is different architectures, different operating systems. A lot of them are Linux based, some of them are RTOS, obviously. We're hitting all the different verticals that usually hear about in these talks, medical devices, critical infrastructure, security devices, home routers, Alexa, whatever. We've got a bunch of different types of products in our dataset. Hopefully the statistics that we talk about here on the second half, keep this in mind, it's fun to be able to troll a dataset of this size. Let's take a step back. Let's talk about the supply chain. Let's talk about the problems that are introduced as a part of the supply chain and what the problems are, maybe even go into some of the solutions. If you are a manufacturer XYZ of a security camera, that security camera is running some sort of firmware on it. There's hardware and there's firmware. The firmware is actually software that's written for that hardware device. We like to think of it as firmware because it's kind of baked in. It's usually not as fluid or as dynamic. The software tends to be in, say, a PC or whatever. The camera still has a full processor memory architecture. It's a full computer running in that thing. If you can take advantage of that or take over that product from a vulnerability standpoint, you've accessed a whole computer's worth of resources. You have hardware components that go in the thing. You have drivers that talk to those hardware, operating systems, libraries, apps, you name it. They're all just the same except they're called firmware. The problem is on a security camera like this or any kind of IoT device is you're going to have multiple vendors. If you're a company XYZ, you're making this product, those hardware components may come from various different vendors and then there may be other vendors like vendor A who talks to your underlying camera optics and can put some image recognition on top of it or whatever else like that. Vendor B may be some support libraries or whatever. The real problem comes in is not only do you have to track vendor A, vendor V, vendor C and what all they're putting onto your device, which quite frankly is not always the case. There's not always full disclosure there, but then vendor A and vendor B may rely on an open source library somewhere, vendor X, that you don't even know about. They may or may not even know that they're using it depending on if their developers have reported on it. The thing that makes that even worse is that vendor X library, say it's the same low-level image processing library. It may be version 1.1 in vendor A and vendor B in their libraries that they're including on your device has 1.5. Maybe vendor A's 1.1 version of vendor X has a vulnerability in it and the 1.5 version doesn't. How do you know what all component libraries you have in the simple camera? This is a full computer running on your network, right? So Donald Rumsfeld, when he was Secretary of Defense, brought this notion together that information can be divided into three categories he said at the time, known, known, unknown, unknown, unknowns. We take that kind of approach towards IoT vulnerabilities. Our known, knowns are vulnerabilities that have explicitly been discovered through scanning and testing on our devices. We've tested it, we know that there's a vulnerability, we patch it or whatever like that. So that's our known knowns. Our known unknowns are newly created software versions or even just upgraded versions or whatever that we've pulled in libraries or whatever that we don't have any kind of application testing behind yet. So who knows what is going on under the hood there? So we know the device, we just don't know if there's any vulnerabilities there. And then in the last categorization of the three, unknown, unknowns are vulnerabilities that are in your camera or your device that you don't know and that nobody else knows. And these are what we're calling zero days, all right? So these zero days or not zero days because we haven't even discovered them like say Ripple 20. Ripple 20 was just discovered a month or two ago. Before then, it was an unknown unknown. It was in all these different devices, but nobody knew it was there. But the weakness was still there waiting to be discovered. So there's an awful lot of work to be done in discovering zero days. As a security researcher, we know that trying to protect yourself and to get through all that is really a challenge trying to find out what is vulnerable or not. So for this talk, we're going to talk about known or unknown knowns. This is a fourth dimension that we like to talk about. And it's comprising that which we intentionally refuse to acknowledge that we know or we don't like to know, okay? So that their vulnerabilities that are known to exist that have not been associated with all the systems that are actually affected. So we know all these CVs are against this open SSL library, but we don't know if that SSL library is in our device. So we're just going to kind of ignore it for now. So this unknown knowns is where we can do an awful lot on our part as manufacturers or security researchers to ferret out and to discover and to patch before other actors out there find the same vulnerabilities and test them against your same device. So all of that can be done through a software bill of materials. So a software bill of materials or an SBOM is there's a bill of materials that usually comes in the manufacturing world that they're very used to of all the components that make up certain devices. So if you're getting big printing presses or whatever else like that, you know all the pieces that make up that printing press so you can plan maintenance, you know, all, you know what the cost is going to be up front. So as a industry software industry, IOT industry, we should have the same thing, the software bill of materials, but we don't. All right. So manufacturers don't know all their components, all the different chips, system on chips that are running inside their devices, all kinds of vendors don't know all their suppliers and their suppliers suppliers because a lot of times vendors will make a product, but then they'll ship it off to a manufacturer to actually make for them. And so how do you validate that, you know, that things are exactly as you designed them or whatever. And then consumers, those of us who put these devices in our critical infrastructure into our security systems into our monitoring software or monitoring networks, even in our homes, we put these devices into our networks, but we don't know all the software that's running on those devices. So the analogy is like, if you found a laptop out in the parking lot of your company or whatever, would you bring that laptop in, fire it up, boot it up and plug it into your critical infrastructure security system network, just to poke around on the laptop and see what's running on it. And I would hope that most of you or all of you would say no, there's not a chance that you would ever do that because we know that those are full computer systems with operating systems that can be compromised with viruses, malicious software, et cetera. But yet with that exact same mentality, we don't apply to IoT devices. We'll take a camera that we do not know all the supply, all the software that's running on it and all the weaknesses that might be inherent on the software. And we'll take that camera and we'll plug that same camera into and talk to it on our critical infrastructure network. So we don't have any way to enumerate this. So how do we generate one of these software build materials reliably? How do we keep track of all those? Say it was even possible to keep track of all the components that are in there, how do we validate one of our devices against one? So we as manufacturers may even develop a device and we might know exactly what we want on it, but if we send it away and it comes back, who's to say that whoever made that device for us put our firmware on there exactly as it was intended and they didn't slip something else in? The other thing would be, and this has happened, is if you as a consumer have a device and you want to update it to the latest update patch, say there's a security vulnerability and you want to patch it, you go to the manufacturer or the vendor's site and you download the update for that firmware and you flash your device with that firmware, how do you know that software wasn't compromised? We have seen in the industry places where upload servers have been compromised by malicious actors and custom firmware has been placed in there as updates and customers have downloaded malicious updates to their devices, which might have been perfectly fine in the first place, but now we're running malicious software. So not only generating a software bill of materials, but validating against the bill of materials is critical and being able to do that. So let's shift away. One thing I'd like to mention about company commitment and stuff like that is I just read Microsoft has a product Azure sphere that they're developing that's a secure IoT chip and it's one way of approaching it, hey, let's control the ecosystem from the beginning and secure it down and they've made commitments to the software bill of materials. So it would be nice if more companies could be able to control their environment such as Microsoft has the luxury to do. And I think these kind of commitments are going to be our way forward is following practices like this generating our own software bill of materials, taking a hard look at what's running on that. So let's switch to today. We have these devices. We don't know what's running on them. Let's look at the CVE CPE reporting mechanisms that we have for finding vulnerabilities in our software. So CVE is common vulnerability exposures. It's a system that provides reference methods for publicly known information security vulnerabilities and exposures. So it's been around a while, MITRE is the one who came up with that and helped maintain that. So these are all the vulnerabilities that are discovered and reported for public knowledge. So it's great we have a central place to report vulnerabilities. As well we have a national vulnerability database with the US government that keeps track of CPEs or common platform enumeration. So this is a structure naming schemes for naming products. This is system, software, software packages, et cetera. So there's a common way to put all this together. The only problem is that we have these frameworks for enumerating these kinds of vulnerabilities or these products, but there's not a lot of adherence or sorry, hard regulations around how we use this. It's a very flexible system and it works fairly well when treated well. But there's a lot of inconsistency across the whole space about how we use CPEs, how we report them, how we link them to products that they apply to, et cetera. So let's look at some of these. Our CPEs, just complete products is at your whole camera system is that whole device, one product, a CPE. Are the component systems in there, your optics and stuff like that? Are the system on chips that are running on there? Is that a CPE? Should that be have its own product entry into that database? Are there libraries within? You're using OpenSSL. That's great. But what about libcrypto that lives within it? Could that be separate from OpenSSL? Could that be its own product? So there's an awful lot of questions that we need to answer with all that. So if we look at something like OpenSSL, it's a commercial-grade product. It's a commercial-grade, secure sockets layer. It also has cryptographic libraries in it. If you go to the NVD, the database, and look up OpenSSL, because you want to find out what product it relates to, because maybe you found a vulnerability, OpenSSL brings back 405 results. Now, every single version of that software is going to be another result. So 405 is not necessarily all the different types of OpenSSL. But there are several. So here's a few examples that come back with OpenSSL. So the very top one is usually what we think of as OpenSSL. It's a CC++ library that is compiled into a lot of Linux systems and stuff with OpenSSL included. 0.9.7 was particularly vulnerable. There's a lot of different CVEs on there. 0.9.8, 1.0, 0.1, et cetera. So there's a lot of different versions of that. The Starfields, we'll just kind of gloss over for now, but those are further ways that you can enumerate or specify what specific product this is if there's a lot of different versions, betas, alphas, different platforms, et cetera. But what are all these other CVEs or CPEs that we see out there? We see this Calderon Pi OpenSSL. So the first part is our vendor. The second part is the actual product. So we have this Pi OpenSSL. I guess we can make a guess that that would be a Python binding for OpenSSL or a Python library. Then we have Lua OpenSSL on the next line. So we guess maybe that's Lua script. But then look at the next two. We have Node OpenSSL. And if you go all the way down to the node.js, that's the target software that it's targeting, not the language, but the target software, which is it's written for Node.js, right? So it's a JavaScript module written for Node. But the very next line, OpenSSL.js project, OpenSSL.js. So you have Node OpenSSL, and then you have OpenSSL.js, and you have Node.js at the end. So now you have someone who's writing for Node that has their own way of specifying, but you have two different competing libraries, which one is which. And then you go down to the last one, OpenSSL project, OpenSSL. Now that looks an awful lot like the first one, OpenSSL. So which one is that? Well, the hard part is, is we really don't know. And if you dig into the metadata about this, and you actually go to the web page, the title doesn't help you very much, OpenSSL project, OpenSSL. So we go into the references, and in the change log, we see a GitHub reference there. And in there we have a Rust OpenSSL. And if you go in there and you look at these different things, you have these keywords of Rust, that's for the Rust language. So finding qualified platform, sorry, CPEs, can be a real challenge. The other thing that's a real challenge here is we don't know between CPEs what relates to each other. Because this Rust language depend on OpenSSL of a certain version, the CC++ version of the first line, to be a certain version, right? Does 092 map to 097 or 097A or 1.01? We don't have these kinds of interrelations. So not only is generating an S-bomb difficult, but developing a body of ground truth around an S-bomb is extremely difficult. And it's not a problem that you think of because you think it should be obvious, but who owns that? Is it the company's responsibility to add their platform to the CPE database so that vulnerabilities are found against it, that there's reliable data there? Or is it the CVEs who find the vulnerabilities? Is it their job to correctly find and identify the platforms if they're not there? So that's just a little background here on the CPE system. They're good systems to start off with, but we need more on top of that. We need some ground truth. We need some ways to relate this. By the way, there are some projects that are starting to put these together, but it's hard. Do you scrape the apt get repose for what components are in OpenSSL or HTTP servers? And HTTP servers rely on OpenSSL, which versions go with which? Well, if you're installing it, you know, because you look at the readme for the HTTP installer, right? And it'll tell you which versions you have, but we don't have any way of systematically obtaining any of that information. So let's go to a specific example of vulnerability and let's look at the supply chain here. We'll go with an old example. This is from four years ago. I'm not going to try to butcher Robert's last name, but Darkonius, he released a write-up on a router backdoor that he had discovered originating from a version of OpenWRT, an open source router operating system, from at that point was 10 years ago, was 10 years in the past. So there was this backdoor hash that's down in the pseudocode below, 1D, 680, whatever. If you entered in that on the command line, you immediately got a root show and you can see the relevant code there. The problem is, is this was just in an OpenWRT version that was 10 years old from, I guess, 2006, right? But the write-up that he had was this was not on an OpenWRT router. This was on a commercial router that had included OpenWRT and its libraries and were using components such as this component script called logon.sh as a part of their operating system. So Darkonius found the backdoor in one or two sets of devices. And if you read through the comments, different people were chiming in, oh, I just tested against this and I found it on this and this. So we found in the comments, three, four, maybe five models of devices listed, but how is Darkonius supposed to know and go out and find, is it his responsibility to go find every single device that this relates to? Is it even his obligation to go to the manufacturer of this device and tell him, hey, I found this in there? If you go and you look for this CVE, you don't find this CVE in the database. If you go even further to the CPE, the platform for this device that he looked for, you don't even find CPEs that are related to this specific device. And this is not a small device. This is a device that made its rounds and you can find it different places. So whose responsibility is it? In our dataset, when we've looked at all of our firmware just sitting around, we just did a string search for that magic hash. And lo and behold, we found 3,810 files that had this hash in it. And it wasn't just the hash, it was the full login script. But notice the file name, COI2, factoryboot.sh, login.sh. So the original login.sh was a part of the original package. But we found the actual login.sh encapsulated in a binary called a command line interface, a CLI, or even two that was used by web servers and stuff on IoT devices that still ran the same code from 2006 that nobody, I'm sure nobody knows, is in there. The manufacturer doesn't even know it's there anymore. And the thing is, is that three major vendor companies have this. 44 different product models and 281 versions of this firmware. So this open WRT login.sh, this hash, has made its way into many, many, many different vendors. This isn't just one person accidentally downloading it and putting it in. This is the supply chain of people taking from A who takes from X, who takes from Y, and it multiplies. So this is the heart of the problem. This is why we're in the bind that we're in is the supply chain. We'll do one more example before we get into the more juicy stuff, the statistics that we found. This latest CVE that we're talking about is Ripple 20. I'm sure you guys have heard of it. 19-0 day vulnerabilities amplified by the supply chain. Its title says it all. And this was two months ago that this was or so was reported. And according to their white paper on it, this affects hundreds of millions of devices or more, includes multiple remote code execution vulnerabilities. So that's the worst kind of vulnerability that you'd want to have. Many other major international vendors suspected of being vulnerable in medical transportation, industrial control systems, et cetera. So my previous example kind of laid the groundwork for how something like Ripple 20 exists. So let's just take one of the CVEs as an example that they reported. So this was published two months ago, June 17, 2020. The description of it, the Trekk TCP IP stack before this version allows remote code execution related to IPv4 tunneling. So this is a vulnerability. They said it's bad. There's a base score of 10 critical on here because the remote code execution and how easy it is to theoretically get access to this TCP IP stack. If we look at the CPE that is related to all CVEs, have all the CPEs that are related to them, we find these four Trekk versions are all rolled up underneath this Trekk stack CPE. The problem is, is when we look for this Trekk stack and we go back to or we click on the actual record in the CPE database, we find the quick info is that the CPE for this Trekk stack was created almost a month after the vulnerability was discovered. So there was no product for Trekk TCP. We had to invent, not we, but the vulnerability researchers had to invent the CPE that even belonged to the Trekk stack. And then we go back to our original problem. If you have this Trekk library and you're trying to figure out where all it's at in the world, you've got to kind of do some detective work. And they did some detective work because we don't have any CPE to CPE relationships. And even if we did, the CPE didn't even exist. Nobody even knew that Trekk TCP exists. So I kind of imagine reading through the paper about this. I almost imagine Luis from Ant-Man 2 when he's being interviewed, where is Scott? And I kind of imagine him talking about, hey, Luis, where is Trekk? And he goes, hey, man, see, that's complicated. See Trekk, there's some smart guys, but they need some help, right? So they go to Elmick Systems and they're like, hey, we need some developers. And Elmick Systems is like, yeah, we think you got it going on. So let's get together. And so they decided to work it together for some. But then something happened. And like, nah, homie, we're split. We're out of here. So Trekk goes off on their own way and Elmick Systems go off on their own way. And Trekk be like, we got the Trekk TCP IP stack. We're going to market all this in the United States. And Elmick Systems are like, no, homie, we're no longer in Elmick Systems. We're Zook and Elmick. And we got the Casago TCP IP. I know it sounds like yours, but it's better because it's been renamed. And you think you got USA and you think you're all hot? Hmm, homie. Dang, we got all of Asia that we're going through. Meanwhile, you got security researchers in the middle of the light going, hey, can you just tell me where Trekk is? And Trekk is like, hmm, see, it's complicated. And they go to Zook and Elmick and see, they're just like, like, where's this Casago at? And we can't even figure it out because their legal system is cutting us out. Whereas Trekk is back here going, let's see, I got an uncle's girlfriend, boyfriend who had a relative named HP. And it's kind of like the Rona virus, right? Like, where's the contact on this? Like tracing. And their uncle's second cousin once removed is named Aruba. And I think we were at a party to get, and meanwhile, Zook and Elmick is like, and Jsoft is just like, where is this? And Zook and Elmick and Trekk and all these people can't find out. And they're like, that's what I'm trying to say. It's complicated. Right? I'm sorry, I probably put you there. But I can imagine industrial control systems and industrial plants that have SIP 13 investigators or people coming in, I can see their first slide to the SIP 13 auditors are going to be this picture of Luis and two words, it's complicated. Right? Because it is trying to find out where this Trekk TCP stack was according to this paper, pretty nuts. Okay. So anyways, it was developed somewhere in the 90s to 2000. It was included in a large quantity of firmware for devices. There's been various patches to it. Some of the Ripple 20 CBEs that were even disclosed were patched and versions as far back as 2009. So you're a vendor, what products of yours contain this TCP IP stack? How do you know? How do you update a device when a patch is present or even know what version of the Trekk stack you used to have in your device? How serious is the threat to your device? Right? Maybe it's not that problematic. Maybe it was just included but never used. So reproducing CBEs are problematic. The actual attack vectors are really hard to classify in these individual products. You potentially waste time from vendors on trivial vulnerabilities that really aren't practical. So really the bottom line for vendors are how much money do you spend trying to find out whether you have to develop firmware patch for software that you don't even know is in your device or how likely is the software that it's even going to be exploitable. Right? So it's a hard, hard question. So I'll come back to the software build materials here in a little bit. But just for right now, let's make a little shift over into our database and let's look at some of the vulnerability statistics that we have here. Hopefully you guys find this enlightening. I certainly did. This is just kind of represents a slice of the industry. We have medical devices. We have information control systems gear. We have all sorts of different sectors represented and verticals in our data sector. So here we go. All right. So Etsy Resolve popular name servers 127001. Not surprising is one of the top ones. We got some Google addresses there, 8844. And then the next three down there that you see 168 are all Asian DNS's. And then you've got some strange noise in there, 192168. Someone.com, a couple others, corporate. So if you have all these devices and they're going to the US and they have Asian domain servers, you're going to have a performance hit or vice versa. How do you know where these products are going and which name servers are hitting? I saw some forum posts about 1921680.7. Maybe the people who made Resolve or this Etsy Resolve had it on test and they were just copy and pasting from a forum into this file. Someone.com, that actually was one vendor, but it's on a whole slew of firmware for all sorts of different products in their product line. So you can see not only is that illustrative that someone has just put that in as an example and they found that example somewhere, but it's in all their Etsy Resolve and they just copy and paste that file into all their other firmware. So where's the checks and balances on what's actually running in there? Even stuff like corp.ubnt.com, corp.ubnt.com doesn't exist, but it's running on a device as a Resolve. So if someone decides to turn up that host name and then start serving requests on there, you could have some interesting results. Here's another one, TTY count in Etsy secure TTY. So if you look at the bottom, Etsy secure TTY file allows you to specify which TTY devices a root user is allowed to log in on. So let's just make one thing clear. Once you have an IoT device in production, you probably shouldn't be logging into that device at all. All right. So most normal operation shouldn't allow anybody from the outside to ever be tunneling into that device remotely. Not a root user. So I want to highlight this 196148. So what that means is that these counts are the number of TTYs listed as approved TTYs that root can log into. So 1486 firmware have file has an Etsy secure TTY with 196 approved TTY locations for root to log in on. All right. So I tracked this one down because this file is just like, where is this anomaly from? I found a Yachto project, which is a way to programmatically or to easily generate a Linux operating system custom build for your project. And this particular file that I found was from 2013. So here is a supply chain. Someone who's building and wants a secure TTY, they pull down this file that's pretty permissive probably for development purposes and that really supposed to be for production. And yet here it makes its way into all these devices. So here's the supply chain in action. All right. Number of valid login shells. So this is an Etsy shell. This is the number of acceptable different types of shells that you can log in from. And most have one. So again, whether you should have a shell that people can log into or not, that's questionable, but at least a lot of them, majority of them, have only one. But then we have this 10 login shells. And I guess what they want by this, a little tongue in cheek, is if their hackers are coming in and are used to have all their key bindings in their environment files for bash, they want to make it easier for them. So they're just going to throw in bash in there. And if you're Ash or ZSH, we got you covered. You can log in however you want. Again, a little tongue in cheek, but that's what that number represents. An additional thing here that was kind of concerning is when I got delved into the actual types of devices at this source. There was over 60 of these or models of patient monitoring health systems. So over 60 different models, which represents however many thousands or hundreds of thousands of health systems, patient monitors that are sitting bedside have 10 or so login shells that somebody could log in with. These are just configuration examples in the supply chain. I'm not saying these are actual vulnerabilities. They're just interesting anomalies from the supply chain. So here's a question we'll configure. We're about out of time, so we're just going to go through these quick firmware where the root user has a login shell, 15,345. So 30% of all of our firmware that we looked at has some sort of login shell. Whereas if you don't want the root to log in, use SBIN login, guess how many? Over under one. We found one firmware that actually had SBIN no longer in it. Okay. Firmware with keys and authorized keys. Now this is a bad backdoor because this basically says if someone SSH is into your box, if your key matches their key in the authorized keys, you get to log in without any password. So this is a classic backdoor, whether it was purposefully put there or not, what's the over under? 175. 175 firmware. Remember, this is firmware, however many devices this represents. This is 12 different vendors with these keys. And over 29, these were really interesting known hosts to see where these came from and the different medical facilities and different places that had known hosts coming from them. Clam AV, only for firmware had any kind of an open antivirus software. Firmware running HTTPD with Mod Auto Index, so this auto index is a way to auto index files on your web browser or your web server when there isn't any kind of match for the HTML name. So it's a great way to kind of troll through your directories. So these are, again, things that are questionable that are in our configs. Firmware starting TTY and init tab. So quite a few firmware are starting TTY, but not that many distinct files. So here we're seeing amplification in the supply chain of here's 63 files, but these 63 files are found in 4,729 devices. Here's a particularly bad one. Firmware with PHP to default to display errors, 332 firmware. So using SQL injection or any kind of command injection, I love when developers leave on their display errors because when I made a mistake, it tells me exactly what's there. The problem is that PHP to by default allows display errors. You have to explicitly turn it off. So maybe we should be making software that doesn't have defaults that are vulnerable like this in their default configuration. All right, next part. Firmware with insecure default DES encryption. So if you don't specify the type of encryption for your passwords, you're going to use DES, which is a bit weaker. 318 firmware, 29 files. But even worse, there were firmware that specified the flag to turn on MD5 as their password crypt. It was 12 firmware. Fortunately, it's small, but they're still 12 firmware. Have no idea where those are at, but they're out there. Firmware with a master.password with no password on root. So in the password file, there is no over, under, on 100 devices, 10 devices, a million devices, not only seven, but still one file made its way in with no password on the root. MySQL binding to 00000. So listen to the world and allow people to go right to your MySQL, 26 devices. Firmware with Redis, the same way, default world bind. So if you don't specify a bind by default, it will allow you in three. It's only one file, but still three firmware allow you into their Redis by default. Average number of unsafe function calls per firmware. So this one is any unsafe function calls is stuff like stir copy, mem copy, all those unbounded ones like Ripple 20 found and stuff like malicious ways to use standard allocation functions without checking lengths and stuff like that. How many do you think in per firmware? 1500 and some average unsafe function calls per firmware. Okay. Number of firmware with unsafe function calls? 53,000. That's over 50%. And those are the ones that we've named. We haven't even trolled through those with function hashing and stuff to find the unlabeled ones that also call these. All right. Firmware, here's the last one. Firmware exporting NFS mounts to the world. All right. Over and around this one. 42. There are four files that contribute to 42 firmware mount to the world such as slash root, mount a star, read write. So you can as root write to root as user ID zero. So these are all questionable configs that have made it their way just fun things that I found trolling through. And finally, what it would have talked like this be without talking about the common passwords and all that. So you see here amplified here again is the supply chain where you have the file counts of all those are the light orange. And then the dark orange is actually how many firmware use those files in them. And the number one being admin, it's only in five files in the password file. But those five password files are found in 470 firmware, which may represent hundreds of thousands of devices where we've cracked the password admin. I'm going to give you 10 to one guess what the password is going to be for that admin user. All right. Anyways, in conclusion, it's all about the software bill of materials drive towards generating and validating these. This is the path for manufacturing's implementing software inventory systems being developers being diligent and reporting all the components used in their systems, consumers having a high demand or having a high standard in transparency and favorite companies and buying from companies with that info as well. These are the team makers who help guide these unified standards and reporting in formats and our security community here developing tools to assist in the automation of the inventory. If we're doing the same thing now in five years, we're going behind. We should be using machine learning. We should be using tools and other things to help us inventory and validate these things not just on the front and on the back end to validate patches and updates are what they say they are and they contain what they think they contain. Thanks to all my team at Fine State and a lot of good research that's been in the industry. Finally, obviously the question and answer on this session are going to the Discord, but I've had a fun time talking to you all. Peace out. Hope you enjoy the rest of DEF CON and IoT Village. Thank you so much for having me.
 All right.  All right, all right, all right.  What's up, y'all?  All right, welcome to my talk IOT under the microscope here virtually.  It's kind of disappointing.  I'm looking at a screen here instead of all you in the audience, but hopefully we'll get  through this and we'll have a good time doing it.  I want to talk about vulnerability trends in the supply chain.  I've got some very interesting things I think that we found in our data set size.  So hopefully you guys will learn something here and we'll have a little fun while doing  it.  All right.  Okay.  So who am I?  I'm Parker Wixel.  I'm born and raised here in Columbus, Ohio.  I've got 25 years industry experience on cybersecurity, on software development, full  stack development, last 10 years of which have really been focused on cybersecurity research  and product development.  I was a contributor and developer of open source security projects like AFL Unicorn.  It's a fuzzing framework for emulated binaries and then Patchwork, which is a statics compilation  of Linux kernels or patching of Linux kernels for debugging purposes.  Like it was mentioned, I'm a senior engineer at Finite State.  We're an IoT cybersecurity firm dealing with a lot of the topics that we're talking about  here and all my data sets and stuff like that come from our Finite State repos and some  of the products that we're working on here.  I'm also a database lecturer over at the Ohio State University looking to kick off yet  another fall there and then I'm a composer and a musician.  Don't hold that against me.  I realize early on that there's not a lot of money in music.  Here I am on just another passion of mine, computers.  Why is this talk relevant to your interests?  We're going to be talking about supply chain trends, vulnerable and not vulnerable, vulnerability  standards and reporting, and then some firmware statistics and observations.  Probably about the first half of the talk is going to be delving into the background  of what supply chains are, some of the ways that we have to talk about vulnerabilities,  what the supply chain introduces as far as vulnerabilities and visibility into those.  Then probably the last half will be delving into the fund numbers, probably why you came  to see this talk.  Nevertheless, hopefully we learned something in both parts.  Our dataset that I'm pulling from right now, we do have partnerships with some private  industry partners.  We do not include all of our private repos and stuff like that, but for this particular  talk I've got about 7 million files, represents about 50,000 firmware images, 10,000 distinct  product lines, and 150 different vendors.  This is different architectures, different operating systems.  A lot of them are Linux based, some of them are RTOS, obviously.  We're hitting all the different verticals that usually hear about in these talks, medical  devices, critical infrastructure, security devices, home routers, Alexa, whatever.  We've got a bunch of different types of products in our dataset.  Hopefully the statistics that we talk about here on the second half, keep this in mind,  it's fun to be able to troll a dataset of this size.  Let's take a step back.  Let's talk about the supply chain.  Let's talk about the problems that are introduced as a part of the supply chain and what the  problems are, maybe even go into some of the solutions.  If you are a manufacturer XYZ of a security camera, that security camera is running some  sort of firmware on it.  There's hardware and there's firmware.  The firmware is actually software that's written for that hardware device.  We like to think of it as firmware because it's kind of baked in.  It's usually not as fluid or as dynamic.  The software tends to be in, say, a PC or whatever.  The camera still has a full processor memory architecture.  It's a full computer running in that thing.  If you can take advantage of that or take over that product from a vulnerability standpoint,  you've accessed a whole computer's worth of resources.  You have hardware components that go in the thing.  You have drivers that talk to those hardware, operating systems, libraries, apps, you name  it.  They're all just the same except they're called firmware.  The problem is on a security camera like this or any kind of IoT device is you're going  to have multiple vendors.  If you're a company XYZ, you're making this product, those hardware components may come  from various different vendors and then there may be other vendors like vendor A who talks  to your underlying camera optics and can put some image recognition on top of it or whatever  else like that.  Vendor B may be some support libraries or whatever.  The real problem comes in is not only do you have to track vendor A, vendor V, vendor C  and what all they're putting onto your device, which quite frankly is not always the case.  There's not always full disclosure there, but then vendor A and vendor B may rely on  an open source library somewhere, vendor X, that you don't even know about.  They may or may not even know that they're using it depending on if their developers  have reported on it.  The thing that makes that even worse is that vendor X library, say it's the same low-level  image processing library.  It may be version 1.1 in vendor A and vendor B in their libraries that they're including  on your device has 1.5.  Maybe vendor A's 1.1 version of vendor X has a vulnerability in it and the 1.5 version  doesn't.  How do you know what all component libraries you have in the simple camera?  This is a full computer running on your network, right?  So Donald Rumsfeld, when he was Secretary of Defense, brought this notion together that  information can be divided into three categories he said at the time, known, known, unknown,  unknown, unknowns.  We take that kind of approach towards IoT vulnerabilities.  Our known, knowns are vulnerabilities that have explicitly been discovered through scanning  and testing on our devices.  We've tested it, we know that there's a vulnerability, we patch it or whatever like that.  So that's our known knowns.  Our known unknowns are newly created software versions or even just upgraded versions or  whatever that we've pulled in libraries or whatever that we don't have any kind of application  testing behind yet.  So who knows what is going on under the hood there?  So we know the device, we just don't know if there's any vulnerabilities there.  And then in the last categorization of the three, unknown, unknowns are vulnerabilities  that are in your camera or your device that you don't know and that nobody else knows.  And these are what we're calling zero days, all right?  So these zero days or not zero days because we haven't even discovered them like say Ripple  20.  Ripple 20 was just discovered a month or two ago.  Before then, it was an unknown unknown.  It was in all these different devices, but nobody knew it was there.  But the weakness was still there waiting to be discovered.  So there's an awful lot of work to be done in discovering zero days.  As a security researcher, we know that trying to protect yourself and to get through all  that is really a challenge trying to find out what is vulnerable or not.  So for this talk, we're going to talk about known or unknown knowns.  This is a fourth dimension that we like to talk about.  And it's comprising that which we intentionally refuse to acknowledge that we know or we don't  like to know, okay?  So that their vulnerabilities that are known to exist that have not been associated with  all the systems that are actually affected.  So we know all these CVs are against this open SSL library, but we don't know if that  SSL library is in our device.  So we're just going to kind of ignore it for now.  So this unknown knowns is where we can do an awful lot on our part as manufacturers or  security researchers to ferret out and to discover and to patch before other actors  out there find the same vulnerabilities and test them against your same device.  So all of that can be done through a software bill of materials.  So a software bill of materials or an SBOM is there's a bill of materials that usually  comes in the manufacturing world that they're very used to of all the components that make  up certain devices.  So if you're getting big printing presses or whatever else like that, you know all the  pieces that make up that printing press so you can plan maintenance, you know, all, you  know what the cost is going to be up front.  So as a industry software industry, IOT industry, we should have the same thing, the software  bill of materials, but we don't.  All right.  So manufacturers don't know all their components, all the different chips, system on chips that  are running inside their devices, all kinds of vendors don't know all their suppliers  and their suppliers suppliers because a lot of times vendors will make a product, but  then they'll ship it off to a manufacturer to actually make for them.  And so how do you validate that, you know, that things are exactly as you designed them  or whatever.  And then consumers, those of us who put these devices in our critical infrastructure into  our security systems into our monitoring software or monitoring networks, even in our homes,  we put these devices into our networks, but we don't know all the software that's running  on those devices.  So the analogy is like, if you found a laptop out in the parking lot of your company or whatever,  would you bring that laptop in, fire it up, boot it up and plug it into your critical  infrastructure security system network, just to poke around on the laptop and see what's  running on it.  And I would hope that most of you or all of you would say no, there's not a chance that  you would ever do that because we know that those are full computer systems with operating  systems that can be compromised with viruses, malicious software, et cetera.  But yet with that exact same mentality, we don't apply to IoT devices.  We'll take a camera that we do not know all the supply, all the software that's running  on it and all the weaknesses that might be inherent on the software.  And we'll take that camera and we'll plug that same camera into and talk to it on our  critical infrastructure network.  So we don't have any way to enumerate this.  So how do we generate one of these software build materials reliably?  How do we keep track of all those?  Say it was even possible to keep track of all the components that are in there, how  do we validate one of our devices against one?  So we as manufacturers may even develop a device and we might know exactly what we want on  it, but if we send it away and it comes back, who's to say that whoever made that device  for us put our firmware on there exactly as it was intended and they didn't slip something  else in?  The other thing would be, and this has happened, is if you as a consumer have a device and  you want to update it to the latest update patch, say there's a security vulnerability  and you want to patch it, you go to the manufacturer or the vendor's site and you download the  update for that firmware and you flash your device with that firmware, how do you know  that software wasn't compromised?  We have seen in the industry places where upload servers have been compromised by malicious  actors and custom firmware has been placed in there as updates and customers have downloaded  malicious updates to their devices, which might have been perfectly fine in the first  place, but now we're running malicious software.  So not only generating a software bill of materials, but validating against the bill  of materials is critical and being able to do that.  So let's shift away.  One thing I'd like to mention about company commitment and stuff like that is I just read  Microsoft has a product Azure sphere that they're developing that's a secure IoT chip  and it's one way of approaching it, hey, let's control the ecosystem from the beginning  and secure it down and they've made commitments to the software bill of materials.  So it would be nice if more companies could be able to control their environment such  as Microsoft has the luxury to do.  And I think these kind of commitments are going to be our way forward is following practices  like this generating our own software bill of materials, taking a hard look at what's  running on that.  So let's switch to today.  We have these devices.  We don't know what's running on them.  Let's look at the CVE CPE reporting mechanisms that we have for finding vulnerabilities in  our software.  So CVE is common vulnerability exposures.  It's a system that provides reference methods for publicly known information security vulnerabilities  and exposures.  So it's been around a while, MITRE is the one who came up with that and helped maintain  that.  So these are all the vulnerabilities that are discovered and reported for public knowledge.  So it's great we have a central place to report vulnerabilities.  As well we have a national vulnerability database with the US government that keeps track of  CPEs or common platform enumeration.  So this is a structure naming schemes for naming products.  This is system, software, software packages, et cetera.  So there's a common way to put all this together.  The only problem is that we have these frameworks for enumerating these kinds of vulnerabilities  or these products, but there's not a lot of adherence or sorry, hard regulations around  how we use this.  It's a very flexible system and it works fairly well when treated well.  But there's a lot of inconsistency across the whole space about how we use CPEs, how  we report them, how we link them to products that they apply to, et cetera.  So let's look at some of these.  Our CPEs, just complete products is at your whole camera system is that whole device,  one product, a CPE.  Are the component systems in there, your optics and stuff like that?  Are the system on chips that are running on there?  Is that a CPE?  Should that be have its own product entry into that database?  Are there libraries within?  You're using OpenSSL.  That's great.  But what about libcrypto that lives within it?  Could that be separate from OpenSSL?  Could that be its own product?  So there's an awful lot of questions that we need to answer with all that.  So if we look at something like OpenSSL, it's a commercial-grade product.  It's a commercial-grade, secure sockets layer.  It also has cryptographic libraries in it.  If you go to the NVD, the database, and look up OpenSSL, because you want to find out what  product it relates to, because maybe you found a vulnerability, OpenSSL brings back 405  results.  Now, every single version of that software is going to be another result.  So 405 is not necessarily all the different types of OpenSSL.  But there are several.  So here's a few examples that come back with OpenSSL.  So the very top one is usually what we think of as OpenSSL.  It's a CC++ library that is compiled into a lot of Linux systems and stuff with OpenSSL  included.  0.9.7 was particularly vulnerable.  There's a lot of different CVEs on there.  0.9.8, 1.0, 0.1, et cetera.  So there's a lot of different versions of that.  The Starfields, we'll just kind of gloss over for now, but those are further ways that you  can enumerate or specify what specific product this is if there's a lot of different versions,  betas, alphas, different platforms, et cetera.  But what are all these other CVEs or CPEs that we see out there?  We see this Calderon Pi OpenSSL.  So the first part is our vendor.  The second part is the actual product.  So we have this Pi OpenSSL.  I guess we can make a guess that that would be a Python binding for OpenSSL or a Python  library.  Then we have Lua OpenSSL on the next line.  So we guess maybe that's Lua script.  But then look at the next two.  We have Node OpenSSL.  And if you go all the way down to the node.js, that's the target software that it's targeting,  not the language, but the target software, which is it's written for Node.js, right?  So it's a JavaScript module written for Node.  But the very next line, OpenSSL.js project, OpenSSL.js.  So you have Node OpenSSL, and then you have OpenSSL.js, and you have Node.js at the end.  So now you have someone who's writing for Node that has their own way of specifying,  but you have two different competing libraries, which one is which.  And then you go down to the last one, OpenSSL project, OpenSSL.  Now that looks an awful lot like the first one, OpenSSL.  So which one is that?  Well, the hard part is, is we really don't know.  And if you dig into the metadata about this, and you actually go to the web page, the title  doesn't help you very much, OpenSSL project, OpenSSL.  So we go into the references, and in the change log, we see a GitHub reference there.  And in there we have a Rust OpenSSL.  And if you go in there and you look at these different things, you have these keywords  of Rust, that's for the Rust language.  So finding qualified platform, sorry, CPEs, can be a real challenge.  The other thing that's a real challenge here is we don't know between CPEs what relates  to each other.  Because this Rust language depend on OpenSSL of a certain version, the CC++ version of  the first line, to be a certain version, right?  Does 092 map to 097 or 097A or 1.01?  We don't have these kinds of interrelations.  So not only is generating an S-bomb difficult, but developing a body of ground truth around  an S-bomb is extremely difficult.  And it's not a problem that you think of because you think it should be obvious, but who owns  that?  Is it the company's responsibility to add their platform to the CPE database so that  vulnerabilities are found against it, that there's reliable data there?  Or is it the CVEs who find the vulnerabilities?  Is it their job to correctly find and identify the platforms if they're not there?  So that's just a little background here on the CPE system.  They're good systems to start off with, but we need more on top of that.  We need some ground truth.  We need some ways to relate this.  By the way, there are some projects that are starting to put these together, but it's hard.  Do you scrape the apt get repose for what components are in OpenSSL or HTTP servers?  And HTTP servers rely on OpenSSL, which versions go with which?  Well, if you're installing it, you know, because you look at the readme for the HTTP installer,  right?  And it'll tell you which versions you have, but we don't have any way of systematically  obtaining any of that information.  So let's go to a specific example of vulnerability and let's look at the supply chain here.  We'll go with an old example.  This is from four years ago.  I'm not going to try to butcher Robert's last name, but Darkonius, he released a write-up  on a router backdoor that he had discovered originating from a version of OpenWRT, an  open source router operating system, from at that point was 10 years ago, was 10 years  in the past.  So there was this backdoor hash that's down in the pseudocode below, 1D, 680, whatever.  If you entered in that on the command line, you immediately got a root show and you can  see the relevant code there.  The problem is, is this was just in an OpenWRT version that was 10 years old from, I guess,  2006, right?  But the write-up that he had was this was not on an OpenWRT router.  This was on a commercial router that had included OpenWRT and its libraries and were using components  such as this component script called logon.sh as a part of their operating system.  So Darkonius found the backdoor in one or two sets of devices.  And if you read through the comments, different people were chiming in, oh, I just tested  against this and I found it on this and this.  So we found in the comments, three, four, maybe five models of devices listed, but how  is Darkonius supposed to know and go out and find, is it his responsibility to go find  every single device that this relates to?  Is it even his obligation to go to the manufacturer of this device and tell him, hey, I found  this in there?  If you go and you look for this CVE, you don't find this CVE in the database.  If you go even further to the CPE, the platform for this device that he looked for, you don't  even find CPEs that are related to this specific device.  And this is not a small device.  This is a device that made its rounds and you can find it different places.  So whose responsibility is it?  In our dataset, when we've looked at all of our firmware just sitting around, we just  did a string search for that magic hash.  And lo and behold, we found 3,810 files that had this hash in it.  And it wasn't just the hash, it was the full login script.  But notice the file name, COI2, factoryboot.sh, login.sh.  So the original login.sh was a part of the original package.  But we found the actual login.sh encapsulated in a binary called a command line interface,  a CLI, or even two that was used by web servers and stuff on IoT devices that still ran the  same code from 2006 that nobody, I'm sure nobody knows, is in there.  The manufacturer doesn't even know it's there anymore.  And the thing is, is that three major vendor companies have this.  44 different product models and 281 versions of this firmware.  So this open WRT login.sh, this hash, has made its way into many, many, many different vendors.  This isn't just one person accidentally downloading it and putting it in.  This is the supply chain of people taking from A who takes from X, who takes from Y,  and it multiplies.  So this is the heart of the problem.  This is why we're in the bind that we're in is the supply chain.  We'll do one more example before we get into the more juicy stuff, the statistics that  we found.  This latest CVE that we're talking about is Ripple 20.  I'm sure you guys have heard of it.  19-0 day vulnerabilities amplified by the supply chain.  Its title says it all.  And this was two months ago that this was or so was reported.  And according to their white paper on it, this affects hundreds of millions of devices  or more, includes multiple remote code execution vulnerabilities.  So that's the worst kind of vulnerability that you'd want to have.  Many other major international vendors suspected of being vulnerable in medical transportation,  industrial control systems, et cetera.  So my previous example kind of laid the groundwork for how something like Ripple 20 exists.  So let's just take one of the CVEs as an example that they reported.  So this was published two months ago, June 17, 2020.  The description of it, the Trekk TCP IP stack before this version allows remote code execution  related to IPv4 tunneling.  So this is a vulnerability.  They said it's bad.  There's a base score of 10 critical on here because the remote code execution and how  easy it is to theoretically get access to this TCP IP stack.  If we look at the CPE that is related to all CVEs, have all the CPEs that are related to  them, we find these four Trekk versions are all rolled up underneath this Trekk stack  CPE.  The problem is, is when we look for this Trekk stack and we go back to or we click on the  actual record in the CPE database, we find the quick info is that the CPE for this Trekk  stack was created almost a month after the vulnerability was discovered.  So there was no product for Trekk TCP.  We had to invent, not we, but the vulnerability researchers had to invent the CPE that even  belonged to the Trekk stack.  And then we go back to our original problem.  If you have this Trekk library and you're trying to figure out where all it's at in  the world, you've got to kind of do some detective work.  And they did some detective work because we don't have any CPE to CPE relationships.  And even if we did, the CPE didn't even exist.  Nobody even knew that Trekk TCP exists.  So I kind of imagine reading through the paper about this.  I almost imagine Luis from Ant-Man 2 when he's being interviewed, where is Scott?  And I kind of imagine him talking about, hey, Luis, where is Trekk?  And he goes, hey, man, see, that's complicated.  See Trekk, there's some smart guys, but they need some help, right?  So they go to Elmick Systems and they're like, hey, we need some developers.  And Elmick Systems is like, yeah, we think you got it going on.  So let's get together.  And so they decided to work it together for some.  But then something happened.  And like, nah, homie, we're split.  We're out of here.  So Trekk goes off on their own way and Elmick Systems go off on their own way.  And Trekk be like, we got the Trekk TCP IP stack.  We're going to market all this in the United States.  And Elmick Systems are like, no, homie, we're no longer in Elmick Systems.  We're Zook and Elmick.  And we got the Casago TCP IP.  I know it sounds like yours, but it's better because it's been renamed.  And you think you got USA and you think you're all hot?  Hmm, homie.  Dang, we got all of Asia that we're going through.  Meanwhile, you got security researchers in the middle of the light going, hey, can you  just tell me where Trekk is?  And Trekk is like, hmm, see, it's complicated.  And they go to Zook and Elmick and see, they're just like, like, where's this Casago at?  And we can't even figure it out because their legal system is cutting us out.  Whereas Trekk is back here going, let's see, I got an uncle's girlfriend, boyfriend who  had a relative named HP.  And it's kind of like the Rona virus, right?  Like, where's the contact on this?  Like tracing.  And their uncle's second cousin once removed is named Aruba.  And I think we were at a party to get, and meanwhile, Zook and Elmick is like, and Jsoft  is just like, where is this?  And Zook and Elmick and Trekk and all these people can't find out.  And they're like, that's what I'm trying to say.  It's complicated.  Right?  I'm sorry, I probably put you there.  But I can imagine industrial control systems and industrial plants that have SIP 13 investigators  or people coming in, I can see their first slide to the SIP 13 auditors are going to  be this picture of Luis and two words, it's complicated.  Right?  Because it is trying to find out where this Trekk TCP stack was according to this paper,  pretty nuts.  Okay.  So anyways, it was developed somewhere in the 90s to 2000.  It was included in a large quantity of firmware for devices.  There's been various patches to it.  Some of the Ripple 20 CBEs that were even disclosed were patched and versions as far  back as 2009.  So you're a vendor, what products of yours contain this TCP IP stack?  How do you know?  How do you update a device when a patch is present or even know what version of the Trekk  stack you used to have in your device?  How serious is the threat to your device?  Right?  Maybe it's not that problematic.  Maybe it was just included but never used.  So reproducing CBEs are problematic.  The actual attack vectors are really hard to classify in these individual products.  You potentially waste time from vendors on trivial vulnerabilities that really aren't  practical.  So really the bottom line for vendors are how much money do you spend trying to find  out whether you have to develop firmware patch for software that you don't even know  is in your device or how likely is the software that it's even going to be exploitable.  Right?  So it's a hard, hard question.  So I'll come back to the software build materials here in a little bit.  But just for right now, let's make a little shift over into our database and let's look  at some of the vulnerability statistics that we have here.  Hopefully you guys find this enlightening.  I certainly did.  This is just kind of represents a slice of the industry.  We have medical devices.  We have information control systems gear.  We have all sorts of different sectors represented and verticals in our data sector.  So here we go.  All right.  So Etsy Resolve popular name servers 127001.  Not surprising is one of the top ones.  We got some Google addresses there, 8844.  And then the next three down there that you see 168 are all Asian DNS's.  And then you've got some strange noise in there, 192168.  Someone.com, a couple others, corporate.  So if you have all these devices and they're going to the US and they have Asian domain  servers, you're going to have a performance hit or vice versa.  How do you know where these products are going and which name servers are hitting?  I saw some forum posts about 1921680.7.  Maybe the people who made Resolve or this Etsy Resolve had it on test and they were  just copy and pasting from a forum into this file.  Someone.com, that actually was one vendor, but it's on a whole slew of firmware for all  sorts of different products in their product line.  So you can see not only is that illustrative that someone has just put that in as an example  and they found that example somewhere, but it's in all their Etsy Resolve and they just  copy and paste that file into all their other firmware.  So where's the checks and balances on what's actually running in there?  Even stuff like corp.ubnt.com, corp.ubnt.com doesn't exist, but it's running on a device  as a Resolve.  So if someone decides to turn up that host name and then start serving requests on there,  you could have some interesting results.  Here's another one, TTY count in Etsy secure TTY.  So if you look at the bottom, Etsy secure TTY file allows you to specify which TTY devices  a root user is allowed to log in on.  So let's just make one thing clear.  Once you have an IoT device in production, you probably shouldn't be logging into that  device at all.  All right.  So most normal operation shouldn't allow anybody from the outside to ever be tunneling into  that device remotely.  Not a root user.  So I want to highlight this 196148.  So what that means is that these counts are the number of TTYs listed as approved TTYs  that root can log into.  So 1486 firmware have file has an Etsy secure TTY with 196 approved TTY locations for root  to log in on.  All right.  So I tracked this one down because this file is just like, where is this anomaly from?  I found a Yachto project, which is a way to programmatically or to easily generate a  Linux operating system custom build for your project.  And this particular file that I found was from 2013.  So here is a supply chain.  Someone who's building and wants a secure TTY, they pull down this file that's pretty  permissive probably for development purposes and that really supposed to be for production.  And yet here it makes its way into all these devices.  So here's the supply chain in action.  All right.  Number of valid login shells.  So this is an Etsy shell.  This is the number of acceptable different types of shells that you can log in from.  And most have one.  So again, whether you should have a shell that people can log into or not, that's questionable,  but at least a lot of them, majority of them, have only one.  But then we have this 10 login shells.  And I guess what they want by this, a little tongue in cheek, is if their hackers are coming  in and are used to have all their key bindings in their environment files for bash, they  want to make it easier for them.  So they're just going to throw in bash in there.  And if you're Ash or ZSH, we got you covered.  You can log in however you want.  Again, a little tongue in cheek, but that's what that number represents.  An additional thing here that was kind of concerning is when I got delved into the  actual types of devices at this source.  There was over 60 of these or models of patient monitoring health systems.  So over 60 different models, which represents however many thousands or hundreds of thousands  of health systems, patient monitors that are sitting bedside have 10 or so login shells  that somebody could log in with.  These are just configuration examples in the supply chain.  I'm not saying these are actual vulnerabilities.  They're just interesting anomalies from the supply chain.  So here's a question we'll configure.  We're about out of time, so we're just going to go through these quick firmware where the  root user has a login shell, 15,345.  So 30% of all of our firmware that we looked at has some sort of login shell.  Whereas if you don't want the root to log in, use SBIN login, guess how many?  Over under one.  We found one firmware that actually had SBIN no longer in it.  Okay.  Firmware with keys and authorized keys.  Now this is a bad backdoor because this basically says if someone SSH is into your box, if your  key matches their key in the authorized keys, you get to log in without any password.  So this is a classic backdoor, whether it was purposefully put there or not, what's  the over under?  175.  175 firmware.  Remember, this is firmware, however many devices this represents.  This is 12 different vendors with these keys.  And over 29, these were really interesting known hosts to see where these came from and  the different medical facilities and different places that had known hosts coming from them.  Clam AV, only for firmware had any kind of an open antivirus software.  Firmware running HTTPD with Mod Auto Index, so this auto index is a way to auto index  files on your web browser or your web server when there isn't any kind of match for the  HTML name.  So it's a great way to kind of troll through your directories.  So these are, again, things that are questionable that are in our configs.  Firmware starting TTY and init tab.  So quite a few firmware are starting TTY, but not that many distinct files.  So here we're seeing amplification in the supply chain of here's 63 files, but these  63 files are found in 4,729 devices.  Here's a particularly bad one.  Firmware with PHP to default to display errors, 332 firmware.  So using SQL injection or any kind of command injection, I love when developers leave on  their display errors because when I made a mistake, it tells me exactly what's there.  The problem is that PHP to by default allows display errors.  You have to explicitly turn it off.  So maybe we should be making software that doesn't have defaults that are vulnerable  like this in their default configuration.  All right, next part.  Firmware with insecure default DES encryption.  So if you don't specify the type of encryption for your passwords, you're going to use DES,  which is a bit weaker.  318 firmware, 29 files.  But even worse, there were firmware that specified the flag to turn on MD5 as their password  crypt.  It was 12 firmware.  Fortunately, it's small, but they're still 12 firmware.  Have no idea where those are at, but they're out there.  Firmware with a master.password with no password on root.  So in the password file, there is no over, under, on 100 devices, 10 devices, a million  devices, not only seven, but still one file made its way in with no password on the root.  MySQL binding to 00000.  So listen to the world and allow people to go right to your MySQL, 26 devices.  Firmware with Redis, the same way, default world bind.  So if you don't specify a bind by default, it will allow you in three.  It's only one file, but still three firmware allow you into their Redis by default.  Average number of unsafe function calls per firmware.  So this one is any unsafe function calls is stuff like stir copy, mem copy, all those  unbounded ones like Ripple 20 found and stuff like malicious ways to use standard allocation  functions without checking lengths and stuff like that.  How many do you think in per firmware?  1500 and some average unsafe function calls per firmware.  Okay.  Number of firmware with unsafe function calls?  53,000.  That's over 50%.  And those are the ones that we've named.  We haven't even trolled through those with function hashing and stuff to find the unlabeled  ones that also call these.  All right.  Firmware, here's the last one.  Firmware exporting NFS mounts to the world.  All right.  Over and around this one.  42.  There are four files that contribute to 42 firmware mount to the world such as slash  root, mount a star, read write.  So you can as root write to root as user ID zero.  So these are all questionable configs that have made it their way just fun things that  I found trolling through.  And finally, what it would have talked like this be without talking about the common passwords  and all that.  So you see here amplified here again is the supply chain where you have the file counts  of all those are the light orange.  And then the dark orange is actually how many firmware use those files in them.  And the number one being admin, it's only in five files in the password file.  But those five password files are found in 470 firmware, which may represent hundreds  of thousands of devices where we've cracked the password admin.  I'm going to give you 10 to one guess what the password is going to be for that admin  user.  All right.  Anyways, in conclusion, it's all about the software bill of materials drive towards generating  and validating these.  This is the path for manufacturing's implementing software inventory systems being developers  being diligent and reporting all the components used in their systems, consumers having a  high demand or having a high standard in transparency and favorite companies and buying from companies  with that info as well.  These are the team makers who help guide these unified standards and reporting in formats  and our security community here developing tools to assist in the automation of the inventory.  If we're doing the same thing now in five years, we're going behind.  We should be using machine learning.  We should be using tools and other things to help us inventory and validate these things  not just on the front and on the back end to validate patches and updates are what they  say they are and they contain what they think they contain.  Thanks to all my team at Fine State and a lot of good research that's been in the industry.  Finally, obviously the question and answer on this session are going to the Discord,  but I've had a fun time talking to you all.  Peace out.  Hope you enjoy the rest of DEF CON and IoT Village.  Thank you so much for having me.
DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_0_1139.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_1140_3119.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_3120_4163.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_4164_5783.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_5784_6107.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_6108_9611.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_9612_10151.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_10152_10475.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_10476_12059.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_12060_13895.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_13896_19691.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_19692_20723.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_20724_21035.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_21036_23147.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_23148_23555.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_23556_23759.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_23760_24083.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_24084_24899.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_24900_27971.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_27972_31583.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_31584_33551.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_33552_37391.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_37392_38735.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_38736_39599.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_39600_40223.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_40224_40931.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_40932_41087.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_41088_41615.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_41616_41879.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_41880_41963.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_41964_42491.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_42492_42635.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_42636_42707.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_42708_42935.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_42936_43055.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_43056_45227.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_45228_46607.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_46608_47123.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_47124_47819.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_47820_50303.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_50304_52883.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_52884_55151.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_55152_58919.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_58920_59843.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_59844_60143.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_60144_61787.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_61788_62243.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_62244_63359.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_63360_64751.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_64752_64931.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_64932_65351.png DEF_CON_Safe_Mode_IoT_Village_-_Parker_Wiksell_-VulnerabilityTrendsintheSupplyChain_65352_65371.png