Sol[IDA]rity
CC Attribution 4.0 International:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Gaasedelen, Markus
Burnett, Nick
Reverse engineering is an exercise of exploration and digital cartography. Researchers slowly unearth bits and pieces of a puzzle, putting them together to better understand the bigger picture. Binaries, like puzzles, can be put together much faster in collaboration with others. And with services such as Google Docs, Office 365, or Etherpad, it is easy to recognize the power and effectiveness of real-time collaboration in the digital space. Unfortunately, reverse engineering as many know it today is almost exclusively an individual experience. Our present reversing tools offer little in the way of collaboration among multiple users. This can make reverse engineering tedious and wasteful in a fast-paced team setting. In this talk we’ll be publicly unveiling Sol[IDA]rity, the newest collaborative solution for the popular disassembler IDA Pro. What started as a simple plugin to sync IDA databases between users in real-time, soon evolved into an interconnectivity platform for IDA with endless potential. Join us for a glimpse at the latest generation of collaborative reverse engineering.
 This morning we have a talk on collaborative solution for Ida to start things off. So thanks everyone for making out here last year recon. So solidarity. Who are we? My name is Marcus Gadsdalen. I'm a security software engineer at Microsoft. And I'm Nick Burnett and I am currently a student at RPI. So if you don't know already, this talk is almost purely about Ida. Ida Pro, if you don't know, it's a disassembler as developed by Hexraiser here somewhere. So I'm sure all of you know what it is. You probably all have a license, hopefully. So just to give some background on our project and the history of collaborative reverse engineering in the context of Ida Pro, we have a little timeline here. I don't want to go into the specifics of the individual projects too much because this is only a 30 minute talk. So way back in 2005, we first saw Ida Sync by Pedram. That was a very simple solution. You guys can Google some of these if you want. It's pretty interesting to look back on history. Collaborates one that everyone probably knows and talks most about. They're all like, hey, you know that one Ida syncing solution that was really crappy and crash all the time? That was Collaborate. That was by Chris Eagle and Tim Beatus, I think, right? So what's interesting about these two projects though is that this was way back and these projects were attempting real time syncing. They were kind of what we call a push-based model. But then we started seeing other solutions coming through such as Bincrowd and CrowdRE, Ida Toolbag, Ida Syncerty. And all of these solutions were pull slash version control, SVN, Git-based. You would commit or you would push your changes up by hand and other people would have to pull them down by hand. And it was very strange, you know, kind of what happens here. And I like to say we stopped dreaming, right? But it also might have been the fact that Ida wasn't quite ready. Ida API wasn't in a place where it could support real time collaboration. Well, real time event capture to the degree that was needed for accurate event syncing or how developers would need it. So version control, you know, now talking about why version control was bad is, version control is a medium to document the process of creation, right, for source control. You're developing code, you're writing code. But reverse engineering is an exercise in discovery and digital cartography, right? You're trying to understand what this binary is, what this blob is. You're trying to document it. You're not writing code. You're not developing a system. So, you know, pull and version control SVN get that seemed very strange for a medium of, you know, storing and maintaining notes and whatnot. So here we are at recon 2016 and we are unveiling Solidarity, a personal project that we've been working on for about a year. Year and a half. Yeah, about a year since about last June. So, Solidarity. We started off with these initial goals right here. You know, four key kind of central goals was we want to sync out of databases in real time, right? We play lots of CTFs. We're very fast paced. We, you know, we need to be able to share information instantaneously. And we also want to say, hey, can we sync X-rays in real time too? You know, that was something that collaborate. There is no way collaborate would be able to do that. CrowdRE kind of touched on some of that stuff a little bit. But then again, there also was more of a kind of a pull based model. And then I said, I want user cursors to dance on Ida's nav band. So, Ida's nav band is the thing on the top of Ida. If you've seen it, it's always usually blue or whatever. Most people just ignore it. But I was like, I want to see multiple users working on the same database and I want to see where they are in the database on that nav band. And then the last point was it has to work, right? I don't want another tool that, you know, kind of half works and doesn't and whatnot. So, we started out, you know, we took the first goal. This is June last year. And this is a little gif of it. You see very quick demo. It worked then. It was just a very simple demo, a perfect concept. You know, we sat down really in a couple hours like, oh man, like, you know, this is like syncing atomic user actions between databases. We're just like renaming a function call here. And we're like, wow, that's so easy. We'll be done by Defcon finals, I said. That was June 2015 and we are still working on this. So, you know, the first attempt was very simple. You have two Ida Pro instances or solidarity clients as we'll call them for now. This was like the one of solidarity. We've gone through probably about three huge architecture overhauls. And so syncing atomic user actions between databases that is actually really easy. If you put in some time and effort. But then, you know, and this is just between two databases, two users, right? You just both load the same database and both are like, all right, let's type in each other's IPs and connect. And, you know, that wasn't hard. But there is a lot of added complexities. You start to realize that, okay, you don't actually want to sync databases. There's a lot, a lot more. Like, how do you ensure that everyone starts out in sync, right? So, you just have this very simple idea of if we just sync every event, everything will be okay. But how do you even make sure that everyone is on the same page before you start syncing events, right? So, people are going to be going to sleep waking up like during CTS, like having different states of their database, you know, you need the same base to start from. And so, that's just one thing. Coordinating and routing multiple sessions at once. What if you want multiple people working on different challenges at the same time across the same server? What if they're, you know, how do you have like five people working on the same binary? You can't really do that with the, you know, that single pipeline we had before. And then capturing only the correct database event. So, Ida's notorious for its auto analyzer, at least in this space of... Collaborative. Yeah, of collaboration. Almost every project is struggled with the auto analyzer because the auto analyzer is kind of like a one-way hash function. Like, you say, hey, define a function here and it will go through your database and just start modifying the database and doing all this crazy stuff and there's no way to go back. And that's why there's no undo. You only go forwards. And so, a lot of people struggle to actually differentiate between what the user is doing and what the auto analyzer is doing to the database. So, you know, in our naive initial attempts, we had no idea about the auto analyzer and all that crazy stuff. And so, that was one of the big issues that we had to work through and that everyone else that has worked in this space has had to work through. We've come up with some good solutions, but we'll get to that in a little bit. So, what happened is this project ended up evolving into more of a... We recognized that we needed a platform to build this collaborative environment that we were seeking for CTFs. And instead of just trying to sync information and data and events between databases, we wanted a platform that you could really work together on. And so, this became a platform to enable Ida interconnectivity. And so, we're going to give a brief kind of high-level overview of the architecture that then zooms down. So, you know, you just have a basic central server and client kind of model going on, right? All your Ida clients phone into the server and the server bounces everything and routes traffic accordingly based on different sessions that are going on. And all that, you know, there's nothing too crazy there. But then we start to get to kind of what we actually call the platform. So, you have two major components to Solidarity. You have the client and the server halves. And so, this is kind of the relationship we see between the two halves. So, you have your client operating system, you know, you could be Windows, Mac, whatever, running Ida Pro. And then you have a server OS. We've mostly been using Ubuntu. But it's just essentially we have a twisted, a Python twisted... I have for networking. Yeah, we'll talk about it in a second. Yeah, on the server. We'll get there. And then... And then often on top of these, the server and the client were able to extend them with modules. So, yeah, this is, you know, so we have in the plugin, which is built on top of Ida Pro, which is what we call the Solidarity client itself, and then the Solidarity server. And this whole get up right here is the platform. But then what we do is we actually build modules on top of the platform. So, it's... we actually made an Ida plugin that then has plugins. So, it's kind of funny. And the server has equivalent plugins on its side. So, it's a very interesting modular framework that, you know, you'll see some cool stuff here. So, going into a little bit more detail about the client here, it's written in Python entirely. We actually had to extend Ida Python and flush it out a lot early on, because the event handling stuff was incomplete when we started last year. Now, I think as of 6.8 and 6.9, it is more complete, or if not finished at this point. So, the plugin, the client plugin is entirely in Python, which is different from Collaborate to the past projects, which were built in C. And we actually use... the client is using the Actor programming pattern, if you guys know about that. And it's specifically using the Python library. And so, the Actor programming pattern essentially is... it takes up things into all these different little nodes or modules, and these modules communicate with each other. They pass messages between each other and act on these messages. And so, you know, we have a core kind of module, and this kind of helps load and organize all the other modules, and all these modules can pass messages to each other based on their needs and what they're interested in doing. And so, it's pretty sweet. It's all asynchronous, nothing blocks anything else. And then there's more modules. So, on the server side of the deal, we have... also, it's written in Python, but it's using Twisted, async.io through Twisted to do the networking event loop and to keep things from blocking that way. And so, we can handle all the different clients coming in and route the events, basically. And for the database model, we're using SQL Alchemy to do the... like make the actual ORM modeling. And then, so the purpose of the server to route between clients, obviously, but then we also implemented a user management system and a web server to help you sort of like manage the entire system better, and we'll see that a little bit later. Yep. And also, just something that's kind of funny to note is that all of the network communication is over TLS 1.2, and we're like, oh, we should, you know, put a little option there to like make it optional. We're like, oh, no, in the spirit of good security, let's actually force that on so you can't turn that off. So, yeah, we actually, you know, try and instill some good security standards here. So, let's start talking. Let's get into... this is... we don't have a live demo. We have lots of really cool GIFs for you guys, which is basically our demo. So, let's start talking about the modules that extend the platform, you know, that really make up the core of what you guys care about, you know, the syncing and all the really cool stuff. So, the first thing we really needed once we had, you know, kind of established the platform was some way to correlate, you know, these users in these sessions and binaries. So, if you have a binary, right, you need... you want to have like five people working on this binary. So, you know, we basically kind of call this a project. And this is, you know, what becomes the foundation for collaboration. And so, the projects module offers a few features. So, even if you don't want to necessarily work in real time with other people, you might want to share databases with other people. So, the projects module offers minimal functionality of... you can easily upload databases, download databases to the server, and it provides a very easy way within Ida to access and push and like pull databases super conveniently. It's no longer like, hey, what's your IP? Let me netcat my IDB to you. You can just do all this directly from within Ida. So, this is a little demo of it. And so, we're able to, from within Ida, actually go and create a project through this GUI. And we can actually... there's this entire directory structure that we can go in and create new directories and projects in. And then from there, actually upload the database that we're working on right now so that other people can then use it once it's been uploaded. Yeah. And so, you can specify some details about the project or about the IDB so that other people can see it. And then we just upload it right to the server. And it will become instantly available to everyone else who's using the platform. And then, yep, and then to show the analogous of this, the next one we are downloading and opening. So, we have a blank Ida instance and we're going to open from the server directly. Yep. So, here's, you know, a user. I just uploaded something. Someone else is like, all right, hey, I'm going to grab it and, you know, join you. And so, this downloads. And so, now, you could very easily access someone else's IDB that they wanted you to get without having to do anything too complicated and directly from inside Ida. Yep. So, I mean, you know, even without talking about syncing events and, you know, all this and that, we're already, you know, starting to see some great solid, you know, collaboration opportunities right here. But just, you know, the ease of being able to share databases across, you know, anyone that has access to the server, so, like, you know, a CTF team or an instant response team inside some company. Yep. Super convenient. Okay. And I quickly just mentioned users. Yeah. Let's see. Yeah. Okay. Yeah. So, because you might have sensitive IDBs and stuff, we've implemented a system so that you can either have open projects or public private projects or private projects so you can add users to private projects and only they can access it. Just as a protection in case you have sensitive stuff inside your company that only certain people can access and so on. Yep. So, just some basic permissions. All right. So, now, we'll talk a little bit about real time, you know, what a lot of you probably care about. Database event syncing, right? Yeah. So, this is just some more trivial syncing. So, we have two different ID instances here, Bulford connected. You can imagine that they were on different computers, but it was easier this way. Yeah. And so, we're able to do all kinds of different operations in IDA and they will be directly synced between both users in real time. So, yeah, we're just doing some basic renames in this one. Here, we're going to change an operand. So, we change that to 20. You can see these on both sides. So, this is what we call more of the easy syncing JIF. Yeah. Because these are pretty simple operations. They don't have any side effects really. Yeah. They don't tap IDA too heavily. So, this is, you know, stepping up a little bit more. This is like structure creation and changing the types of the different fields and structures. This is a little bit more interesting, a little bit more out of the norm. Very useful though, because structures are often very useful to have for reverse engineering. So, it's great to be able to share them in real time with people working on the project with you. Yep. So, this is just structure creation, you know, defining some fields, renaming some stuff. It's all very quick. We'll just let this play through. How are we doing on time? Oh, we're doing good. We're doing good. So, yeah, pretty cool, pretty convenient. Changing some types on the fields. And I think we end up actually delineating it entirely just for fun. And it's gone. Yeah. Cool. All right. And then something a little bit more advanced here is we're actually going to engage the auto analyzer a little bit by undefining functions. You see both sides have undefined them the same way. So now we can go back and start redefining code and let the auto analyzer do its stuff. But you see they're still able to sync the same events even though Ida is creating a ton of actions that it's doing to its own database. So now we're starting to undefine and define selection of bytes, which is also a little interesting. It's different. Undefining and defining selection is actually different from just hitting U and like undefining, you know, from where you are downwards. So, yeah, it's cool. You know, we're screwing up all the code here, but it's in sync, right? That's what we care. We're trying to minimize divergences as much as possible. So, yep. So that was a little bit more advanced. So, anyway, some of the secrets for the event syncing that a lot of people have circled on. I wanted to put more slides in here, but I ran out of time. So you only get one slide of some of the secret sauce is you have to really respect Ida's auto analyzer. So it might not have been the best design decision way back. I know I've seen Ilfalk talk about how way back in the day they were hoping to have an AI built into Ida to help, you know, assist you with reverse engineering and whatnot. And so it's, you really have to respect it and, you know, don't mess with it. Like when it needs to do stuff, you let it do stuff and you back off. And so the other, you know, the other super secret hint is hook and utilize the cute subsystem. Like it is there for the taking and you can do almost anything you want to Ida. Few people realize that but Ida is so, so customizable. You could change just about any aspect. If you guys have seen some of the crazy Ali, like Ali, you know, customizations people have done for Ali debug and whatnot, like crazy menus and whatnot. Like you could do the same exact thing with Ida. Few people actually utilize Ida's API or it's, you know, the power of the QT subsystem. So also the last point is providing context to the signals and hooks and events that you use are that are made available in Ida. Collaborate in some of the other projects just captured them and broadcasted them. But the thing is some of those individual signals don't really mean a whole lot. You need to understand them in context. And I wish I had time to put together a diagram to better explain how we handle this. But our event syncing stuff, how we kind of deal with and watch and understand the auto analyzer and what the user is doing is we actually use a finite state machine to walk through. We walk through this finite state machine based on the signals we're receiving. So one signal could be interpreted by different ways. It could mean five different things, but it really depends on the context that you capture it. And that's how we try and interpret things. And we're being bitten by the auto analyzer. Yeah. So, all right, let's keep moving through some of the modules. So replay. Very useful if your friend is doing a bunch of reversing and you went to sleep or something and now he's done a bunch of changes. And you want to come back online and get those changes. So basically what we're showing here is we're going to make changes to one of the databases that's connected. And the other one is currently not open or connected. So then once we've made some changes, we will connect the other one and it will sync up to the state of the first one. And we will be back in sync with the other one. You can see the changes happening in the background and now we're back in sync again. And so if you think, you might think, oh, well, what if there's a ton of states? Well, we have a system where snapshots of the IDB get uploaded. You download that and then rebase from there. So it's a very few amount of events that you actually have to sync. And so it really keeps you able to, or you're really able to keep up with that. Yeah. So we're already running low on time. So we got to keep moving. So, status bar. So in IDA, you have, I saw the status bar on the bottom and I'm like, wow, there's so much potential. So I started doing crazy things with QT and I was like, I want to put stuff down there. And so I did. So I put like your server status. You can easily select which server you want to connect to. In case you have multiple, maybe you play with multiple CTF teams or companies or whatever. So you could just right click this nice little thing, pick your server, what not, and go. But we got to keep moving. So invites. So I gave art a toast, right? Like we love toast notifications. So imagine if you can right click somewhere in the database and be like, hey, like here I am chatting across the room. Hey, Nick, I found the blog. It's in this function. And he's like, oh, what function is that? I'm not going to say 0x41bf7 3f anymore. I just say I right click and I say invite Nick here and he just gets a notification. Boom. You can just click that notification and his database gets flown over there. Alternatively, you can also, you know, if you were busy reversing something, you have your nice little Q of the notifications that came in. So you're a little in box. This is kind of what the colors mean. But we got to keep going. So cursors. I said I want to put stuff on the nav band. So the text stories gives you two APIs. One of them refreshes the nav band. The other lets you draw a single line of pixels to the nav band. The refresh API doesn't actually work. So I had one. I had one API and the QT subsystem to do this with. So you can see these are the different users we're working with. If you hover over them, you actually can see their names, which is fun. You can kind of see where people are working in the binary and spread out that way. You know, practically nobody will probably ever use this, but this was part of the vision. We're trying to establish a user experience, right? Yeah. Hex rays. So we said we want to do hex rays. Started working on it. I haven't worked on it in a little while. It is definitely feasible. This kind of demos some of it. This just a little bit old. Yeah. So we are doing some different like type changes and structural manipulation in the actual hex ray window and it's syncing up to the other person who's also within a hex rays window. And so that way you don't have to work at assembly level if you don't want. Yeah. So I mean, we all use hex rays so much nowadays. It's a skill to be good at it. A lot of people just press F5 and are like, oh, this is unreadable. But, you know, there's, you know, we're working on silently syncing hex rays. It's definitely very feasible. If you guys could open up the microcode, that would be great. Or some more SDK or APIs for it. That'd be awesome. I've had to reverse hex rays and, you know, the hex rays plug itself in IDA at some times to understand how some of the APIs work. Anyway, yeah. So now I'm very quick. Just want to show the web server. This is because IDA GUIs can be a little tricky. Maybe you don't want to open IDA to like add someone to a project or something. So, yeah. So what we have is it's basically a very simple web server here. You can see like all the projects that you are on. You can see recent actions that have happened. Here's like an overview of the projects. You can see information like, oh, these all have sync enabled. That one's private. So on you can get like a view of the directories so that you can kind of visualize that better. And then further, you can go to a project and like see very detailed information about what's been happening in it. So you can like get an idea of how the reversing is going and as well as be able to see like all the IDBs that have been uploaded in case you want to like take one out of the system and use it personally or something. Yeah. And then so the other thing is we really want like fine grain control through the web interface because it's much easier to implement that kind of like control through all the different forms and everything. So you're able, this is the easiest way to add users to projects, change project details and everything so you don't have to go into IDA to those kind of tasks that don't actually involve reverse engineering. So yeah, that's pretty much the web server. Yeah. So yeah, there's a, you know, there's actually a lot of, there's a lot more that we did not cover. We only have 30 minutes. So we, you know, we're out of time basically. We showed a bunch. There's a lot more planned. We also like to release as a beta. We submitted for a black hat and we're hoping to release it or beta it out then, but that didn't pan out. So it'll be betaed soon. That's all we can promise. There's a lot of other really cool stuff that we wish we could have shown you guys, but hopefully soon. So, you know, follow us on Twitter. The website we're going to push up soon just, you know, so there's some central resource. And then we'll have all the demos that we showed today if you want to look at them. Yeah. And we'll, you know, we'll put these slides up and yeah. So that's all we've got. Yeah. Yeah. Yeah. So, yeah. So the, we were as mentioned earlier, how the question. Yes. Yeah. Sorry. He was wondering if there was some sort of like version control for the actual like databases as you're going along case somebody like completely messes up a database and you can't undo it. So as I mentioned earlier, we have a system that will like make snapshots of IDBs as you're going on so that you don't have to sync up as many events. So you could easily tell solidarity to use one of those as the most recent one and then just move on from there. And the other ones will just be like put aside and you can download them and do whatever if you want, but they won't be used for anything anymore. Yep. Yep. So, you know, we maintain a history of all the events that are flying over the server that, you know, are more detailed, precise events, not just the stuff that was spewed kind of like collaborate. We, there, we maintain this whole log of them as well as various snapshots of databases that automatically get uploaded from a user in that session. So every 500 events, let's say a database gets uploaded. And so anytime someone else joins that session now that database gets pulled down and they, you know, they join the session that database will get pulled down and then any events they have missed that have been, you know, occurred since then will be played, you know, which is going to be. Yeah, and you can do the same thing with any of the databases pretty much that have been in like the snapshots and that way you could basically go back in time if you really need to. Yep. So any other questions? Yeah. So divergences offline is just difficult, right? Like there is no good solution to that. So, so one thing if you have to make a lot of changes offline, you can basically create that new snapshot then and people who want to work from that kind of like fork of the thing can then go on from there. But if you're the idea is you're doing most of the stuff online. So ideally you would be able to connect and send like record the events that you've been making. Otherwise, the server doesn't know what you've been doing. Yeah, the simple answer is basically get forked. So, so yeah, I don't, I don't think we really have any more time. Unfortunately, we're already over. But, you know, hit us up will be around for the rest of the day. Or otherwise, yeah, hit us on Twitter. Yeah. Yeah.
 This morning we have a talk on collaborative solution for Ida to start things off.  So thanks everyone for making out here last year recon.  So solidarity.  Who are we?  My name is Marcus Gadsdalen.  I'm a security software engineer at Microsoft.  And I'm Nick Burnett and I am currently a student at RPI.  So if you don't know already, this talk is almost purely about Ida.  Ida Pro, if you don't know, it's a disassembler as developed by Hexraiser here somewhere.  So I'm sure all of you know what it is.  You probably all have a license, hopefully.  So just to give some background on our project and the history of collaborative reverse engineering in the context of Ida Pro, we have a little timeline here.  I don't want to go into the specifics of the individual projects too much because this is only a 30 minute talk.  So way back in 2005, we first saw Ida Sync by Pedram.  That was a very simple solution.  You guys can Google some of these if you want.  It's pretty interesting to look back on history.  Collaborates one that everyone probably knows and talks most about.  They're all like, hey, you know that one Ida syncing solution that was really crappy and crash all the time?  That was Collaborate.  That was by Chris Eagle and Tim Beatus, I think, right?  So what's interesting about these two projects though is that this was way back and these projects were attempting real time syncing.  They were kind of what we call a push-based model.  But then we started seeing other solutions coming through such as Bincrowd and CrowdRE, Ida Toolbag, Ida Syncerty.  And all of these solutions were pull slash version control, SVN, Git-based.  You would commit or you would push your changes up by hand and other people would have to pull them down by hand.  And it was very strange, you know, kind of what happens here.  And I like to say we stopped dreaming, right?  But it also might have been the fact that Ida wasn't quite ready.  Ida API wasn't in a place where it could support real time collaboration.  Well, real time event capture to the degree that was needed for accurate event syncing or how developers would need it.  So version control, you know, now talking about why version control was bad is,  version control is a medium to document the process of creation, right, for source control.  You're developing code, you're writing code.  But reverse engineering is an exercise in discovery and digital cartography, right?  You're trying to understand what this binary is, what this blob is.  You're trying to document it.  You're not writing code.  You're not developing a system.  So, you know, pull and version control SVN get that seemed very strange for a medium of, you know, storing and maintaining notes and whatnot.  So here we are at recon 2016 and we are unveiling Solidarity, a personal project that we've been working on for about a year.  Year and a half.  Yeah, about a year since about last June.  So, Solidarity.  We started off with these initial goals right here.  You know, four key kind of central goals was we want to sync out of databases in real time, right?  We play lots of CTFs.  We're very fast paced.  We, you know, we need to be able to share information instantaneously.  And we also want to say, hey, can we sync X-rays in real time too?  You know, that was something that collaborate.  There is no way collaborate would be able to do that.  CrowdRE kind of touched on some of that stuff a little bit.  But then again, there also was more of a kind of a pull based model.  And then I said, I want user cursors to dance on Ida's nav band.  So, Ida's nav band is the thing on the top of Ida.  If you've seen it, it's always usually blue or whatever.  Most people just ignore it.  But I was like, I want to see multiple users working on the same database and I want to see where they are in the database on that nav band.  And then the last point was it has to work, right?  I don't want another tool that, you know, kind of half works and doesn't and whatnot.  So, we started out, you know, we took the first goal.  This is June last year.  And this is a little gif of it.  You see very quick demo.  It worked then.  It was just a very simple demo, a perfect concept.  You know, we sat down really in a couple hours like, oh man, like, you know, this is like syncing atomic user actions between databases.  We're just like renaming a function call here.  And we're like, wow, that's so easy.  We'll be done by Defcon finals, I said.  That was June 2015 and we are still working on this.  So, you know, the first attempt was very simple.  You have two Ida Pro instances or solidarity clients as we'll call them for now.  This was like the one of solidarity.  We've gone through probably about three huge architecture overhauls.  And so syncing atomic user actions between databases that is actually really easy.  If you put in some time and effort.  But then, you know, and this is just between two databases, two users, right?  You just both load the same database and both are like, all right, let's type in each other's IPs and connect.  And, you know, that wasn't hard.  But there is a lot of added complexities.  You start to realize that, okay, you don't actually want to sync databases.  There's a lot, a lot more.  Like, how do you ensure that everyone starts out in sync, right?  So, you just have this very simple idea of if we just sync every event, everything will be okay.  But how do you even make sure that everyone is on the same page before you start syncing events, right?  So, people are going to be going to sleep waking up like during CTS, like having different states of their database, you know, you need the same base to start from.  And so, that's just one thing.  Coordinating and routing multiple sessions at once.  What if you want multiple people working on different challenges at the same time across the same server?  What if they're, you know, how do you have like five people working on the same binary?  You can't really do that with the, you know, that single pipeline we had before.  And then capturing only the correct database event.  So, Ida's notorious for its auto analyzer, at least in this space of...  Collaborative.  Yeah, of collaboration.  Almost every project is struggled with the auto analyzer because the auto analyzer is kind of like a one-way hash function.  Like, you say, hey, define a function here and it will go through your database and just start modifying the database and doing all this crazy stuff and there's no way to go back.  And that's why there's no undo. You only go forwards.  And so, a lot of people struggle to actually differentiate between what the user is doing and what the auto analyzer is doing to the database.  So, you know, in our naive initial attempts, we had no idea about the auto analyzer and all that crazy stuff.  And so, that was one of the big issues that we had to work through and that everyone else that has worked in this space has had to work through.  We've come up with some good solutions, but we'll get to that in a little bit.  So, what happened is this project ended up evolving into more of a...  We recognized that we needed a platform to build this collaborative environment that we were seeking for CTFs.  And instead of just trying to sync information and data and events between databases,  we wanted a platform that you could really work together on.  And so, this became a platform to enable Ida interconnectivity.  And so, we're going to give a brief kind of high-level overview of the architecture that then zooms down.  So, you know, you just have a basic central server and client kind of model going on, right?  All your Ida clients phone into the server and the server bounces everything and routes traffic accordingly based on different sessions that are going on.  And all that, you know, there's nothing too crazy there.  But then we start to get to kind of what we actually call the platform.  So, you have two major components to Solidarity.  You have the client and the server halves.  And so, this is kind of the relationship we see between the two halves.  So, you have your client operating system, you know, you could be Windows, Mac, whatever, running Ida Pro.  And then you have a server OS.  We've mostly been using Ubuntu.  But it's just essentially we have a twisted, a Python twisted...  I have for networking.  Yeah, we'll talk about it in a second.  Yeah, on the server. We'll get there.  And then...  And then often on top of these, the server and the client were able to extend them with modules.  So, yeah, this is, you know, so we have in the plugin, which is built on top of Ida Pro,  which is what we call the Solidarity client itself, and then the Solidarity server.  And this whole get up right here is the platform.  But then what we do is we actually build modules on top of the platform.  So, it's... we actually made an Ida plugin that then has plugins.  So, it's kind of funny.  And the server has equivalent plugins on its side.  So, it's a very interesting modular framework that, you know, you'll see some cool stuff here.  So, going into a little bit more detail about the client here, it's written in Python entirely.  We actually had to extend Ida Python and flush it out a lot early on,  because the event handling stuff was incomplete when we started last year.  Now, I think as of 6.8 and 6.9, it is more complete, or if not finished at this point.  So, the plugin, the client plugin is entirely in Python, which is different from Collaborate  to the past projects, which were built in C.  And we actually use... the client is using the Actor programming pattern, if you guys know about that.  And it's specifically using the Python library.  And so, the Actor programming pattern essentially is...  it takes up things into all these different little nodes or modules,  and these modules communicate with each other.  They pass messages between each other and act on these messages.  And so, you know, we have a core kind of module,  and this kind of helps load and organize all the other modules,  and all these modules can pass messages to each other based on their needs and what they're interested in doing.  And so, it's pretty sweet. It's all asynchronous, nothing blocks anything else.  And then there's more modules.  So, on the server side of the deal, we have... also, it's written in Python,  but it's using Twisted, async.io through Twisted to do the networking event loop  and to keep things from blocking that way.  And so, we can handle all the different clients coming in and route the events, basically.  And for the database model, we're using SQL Alchemy to do the...  like make the actual ORM modeling.  And then, so the purpose of the server to route between clients, obviously,  but then we also implemented a user management system and a web server  to help you sort of like manage the entire system better,  and we'll see that a little bit later.  Yep. And also, just something that's kind of funny to note is that  all of the network communication is over TLS 1.2,  and we're like, oh, we should, you know, put a little option there to like make it optional.  We're like, oh, no, in the spirit of good security, let's actually force that on so you can't turn that off.  So, yeah, we actually, you know, try and instill some good security standards here.  So, let's start talking. Let's get into... this is... we don't have a live demo.  We have lots of really cool GIFs for you guys, which is basically our demo.  So, let's start talking about the modules that extend the platform,  you know, that really make up the core of what you guys care about,  you know, the syncing and all the really cool stuff.  So, the first thing we really needed once we had, you know, kind of established the platform  was some way to correlate, you know, these users in these sessions and binaries.  So, if you have a binary, right, you need... you want to have like five people working on this binary.  So, you know, we basically kind of call this a project.  And this is, you know, what becomes the foundation for collaboration.  And so, the projects module offers a few features.  So, even if you don't want to necessarily work in real time with other people,  you might want to share databases with other people.  So, the projects module offers minimal functionality of...  you can easily upload databases, download databases to the server,  and it provides a very easy way within Ida to access and push and like pull databases super conveniently.  It's no longer like, hey, what's your IP? Let me netcat my IDB to you.  You can just do all this directly from within Ida.  So, this is a little demo of it.  And so, we're able to, from within Ida, actually go and create a project through this GUI.  And we can actually... there's this entire directory structure that we can go in  and create new directories and projects in.  And then from there, actually upload the database that we're working on right now  so that other people can then use it once it's been uploaded.  Yeah. And so, you can specify some details about the project or about the IDB  so that other people can see it.  And then we just upload it right to the server.  And it will become instantly available to everyone else who's using the platform.  And then, yep, and then to show the analogous of this, the next one we are downloading  and opening.  So, we have a blank Ida instance and we're going to open from the server directly.  Yep. So, here's, you know, a user. I just uploaded something.  Someone else is like, all right, hey, I'm going to grab it and, you know, join you.  And so, this downloads.  And so, now, you could very easily access someone else's IDB that they wanted you to get  without having to do anything too complicated and directly from inside Ida.  Yep. So, I mean, you know, even without talking about syncing events and, you know,  all this and that, we're already, you know, starting to see some great solid, you know,  collaboration opportunities right here. But just, you know, the ease of being able to  share databases across, you know, anyone that has access to the server, so, like, you know,  a CTF team or an instant response team inside some company.  Yep. Super convenient.  Okay. And I quickly just mentioned users. Yeah.  Let's see.  Yeah. Okay. Yeah. So, because you might have sensitive IDBs and stuff, we've implemented a system  so that you can either have open projects or public private projects or private projects  so you can add users to private projects and only they can access it.  Just as a protection in case you have sensitive stuff inside your company that only certain people  can access and so on.  Yep. So, just some basic permissions.  All right. So, now, we'll talk a little bit about real time, you know, what a lot of you probably  care about. Database event syncing, right?  Yeah. So, this is just some more trivial syncing.  So, we have two different ID instances here, Bulford connected.  You can imagine that they were on different computers, but it was easier this way.  Yeah.  And so, we're able to do all kinds of different operations in IDA and they will be directly  synced between both users in real time.  So, yeah, we're just doing some basic renames in this one.  Here, we're going to change an operand.  So, we change that to 20. You can see these on both sides.  So, this is what we call more of the easy syncing JIF.  Yeah. Because these are pretty simple operations. They don't have any side effects really.  Yeah. They don't tap IDA too heavily.  So, this is, you know, stepping up a little bit more.  This is like structure creation and changing the types of the different fields and structures.  This is a little bit more interesting, a little bit more out of the norm.  Very useful though, because structures are often very useful to have for reverse engineering.  So, it's great to be able to share them in real time with people working on the project with you.  Yep. So, this is just structure creation, you know, defining some fields, renaming some stuff.  It's all very quick.  We'll just let this play through. How are we doing on time?  Oh, we're doing good. We're doing good.  So, yeah, pretty cool, pretty convenient.  Changing some types on the fields.  And I think we end up actually delineating it entirely just for fun.  And it's gone.  Yeah. Cool. All right.  And then something a little bit more advanced here is we're actually going to engage the auto analyzer a little bit by undefining functions.  You see both sides have undefined them the same way. So now we can go back and start redefining code and let the auto analyzer do its stuff.  But you see they're still able to sync the same events even though Ida is creating a ton of actions that it's doing to its own database.  So now we're starting to undefine and define selection of bytes, which is also a little interesting.  It's different. Undefining and defining selection is actually different from just hitting U and like undefining, you know, from where you are downwards.  So, yeah, it's cool. You know, we're screwing up all the code here, but it's in sync, right?  That's what we care. We're trying to minimize divergences as much as possible.  So, yep. So that was a little bit more advanced.  So, anyway, some of the secrets for the event syncing that a lot of people have circled on.  I wanted to put more slides in here, but I ran out of time.  So you only get one slide of some of the secret sauce is you have to really respect Ida's auto analyzer.  So it might not have been the best design decision way back.  I know I've seen Ilfalk talk about how way back in the day they were hoping to have an AI built into Ida to help, you know, assist you with reverse engineering and whatnot.  And so it's, you really have to respect it and, you know, don't mess with it.  Like when it needs to do stuff, you let it do stuff and you back off.  And so the other, you know, the other super secret hint is hook and utilize the cute subsystem.  Like it is there for the taking and you can do almost anything you want to Ida.  Few people realize that but Ida is so, so customizable.  You could change just about any aspect.  If you guys have seen some of the crazy Ali, like Ali, you know, customizations people have done for Ali debug and whatnot, like crazy menus and whatnot.  Like you could do the same exact thing with Ida.  Few people actually utilize Ida's API or it's, you know, the power of the QT subsystem.  So also the last point is providing context to the signals and hooks and events that you use are that are made available in Ida.  Collaborate in some of the other projects just captured them and broadcasted them.  But the thing is some of those individual signals don't really mean a whole lot.  You need to understand them in context.  And I wish I had time to put together a diagram to better explain how we handle this.  But our event syncing stuff, how we kind of deal with and watch and understand the auto analyzer and what the user is doing is we actually use a finite state machine to walk through.  We walk through this finite state machine based on the signals we're receiving.  So one signal could be interpreted by different ways.  It could mean five different things, but it really depends on the context that you capture it.  And that's how we try and interpret things.  And we're being bitten by the auto analyzer.  Yeah. So, all right, let's keep moving through some of the modules.  So replay.  Very useful if your friend is doing a bunch of reversing and you went to sleep or something and now he's done a bunch of changes.  And you want to come back online and get those changes.  So basically what we're showing here is we're going to make changes to one of the databases that's connected.  And the other one is currently not open or connected.  So then once we've made some changes, we will connect the other one and it will sync up to the state of the first one.  And we will be back in sync with the other one.  You can see the changes happening in the background and now we're back in sync again.  And so if you think, you might think, oh, well, what if there's a ton of states?  Well, we have a system where snapshots of the IDB get uploaded.  You download that and then rebase from there.  So it's a very few amount of events that you actually have to sync.  And so it really keeps you able to, or you're really able to keep up with that.  Yeah. So we're already running low on time.  So we got to keep moving. So, status bar.  So in IDA, you have, I saw the status bar on the bottom and I'm like, wow, there's so much potential.  So I started doing crazy things with QT and I was like, I want to put stuff down there.  And so I did. So I put like your server status.  You can easily select which server you want to connect to.  In case you have multiple, maybe you play with multiple CTF teams or companies or whatever.  So you could just right click this nice little thing, pick your server, what not, and go.  But we got to keep moving.  So invites. So I gave art a toast, right?  Like we love toast notifications.  So imagine if you can right click somewhere in the database and be like, hey, like here I am chatting across the room.  Hey, Nick, I found the blog. It's in this function.  And he's like, oh, what function is that?  I'm not going to say 0x41bf7 3f anymore.  I just say I right click and I say invite Nick here and he just gets a notification. Boom.  You can just click that notification and his database gets flown over there.  Alternatively, you can also, you know, if you were busy reversing something,  you have your nice little Q of the notifications that came in.  So you're a little in box. This is kind of what the colors mean.  But we got to keep going.  So cursors. I said I want to put stuff on the nav band.  So the text stories gives you two APIs. One of them refreshes the nav band.  The other lets you draw a single line of pixels to the nav band.  The refresh API doesn't actually work.  So I had one. I had one API and the QT subsystem to do this with.  So you can see these are the different users we're working with.  If you hover over them, you actually can see their names, which is fun.  You can kind of see where people are working in the binary and spread out that way.  You know, practically nobody will probably ever use this, but this was part of the vision.  We're trying to establish a user experience, right?  Yeah.  Hex rays. So we said we want to do hex rays.  Started working on it. I haven't worked on it in a little while.  It is definitely feasible.  This kind of demos some of it. This just a little bit old.  Yeah. So we are doing some different like type changes and structural manipulation  in the actual hex ray window and it's syncing up to the other person who's also within a hex rays window.  And so that way you don't have to work at assembly level if you don't want.  Yeah. So I mean, we all use hex rays so much nowadays.  It's a skill to be good at it. A lot of people just press F5 and are like, oh, this is unreadable.  But, you know, there's, you know, we're working on silently syncing hex rays.  It's definitely very feasible.  If you guys could open up the microcode, that would be great.  Or some more SDK or APIs for it. That'd be awesome.  I've had to reverse hex rays and, you know, the hex rays plug itself in IDA at some times to understand how some of the APIs work.  Anyway, yeah. So now I'm very quick.  Just want to show the web server.  This is because IDA GUIs can be a little tricky.  Maybe you don't want to open IDA to like add someone to a project or something.  So, yeah. So what we have is it's basically a very simple web server here.  You can see like all the projects that you are on.  You can see recent actions that have happened.  Here's like an overview of the projects. You can see information like, oh, these all have sync enabled.  That one's private.  So on you can get like a view of the directories so that you can kind of visualize that better.  And then further, you can go to a project and like see very detailed information about what's been happening in it.  So you can like get an idea of how the reversing is going and as well as be able to see like all the IDBs that have been uploaded in case you want to like take one out of the system and use it personally or something.  Yeah. And then so the other thing is we really want like fine grain control through the web interface because it's much easier to implement that kind of like control through all the different forms and everything.  So you're able, this is the easiest way to add users to projects, change project details and everything so you don't have to go into IDA to those kind of tasks that don't actually involve reverse engineering.  So yeah, that's pretty much the web server.  Yeah.  So yeah, there's a, you know, there's actually a lot of, there's a lot more that we did not cover. We only have 30 minutes.  So we, you know, we're out of time basically.  We showed a bunch.  There's a lot more planned.  We also like to release as a beta. We submitted for a black hat and we're hoping to release it or beta it out then, but that didn't pan out.  So it'll be betaed soon.  That's all we can promise.  There's a lot of other really cool stuff that we wish we could have shown you guys, but hopefully soon.  So, you know, follow us on Twitter.  The website we're going to push up soon just, you know, so there's some central resource.  And then we'll have all the demos that we showed today if you want to look at them.  Yeah. And we'll, you know, we'll put these slides up and yeah.  So that's all we've got.  Yeah.  Yeah.  Yeah. So, yeah. So the, we were as mentioned earlier, how the question.  Yes. Yeah. Sorry.  He was wondering if there was some sort of like version control for the actual like databases as you're going along case somebody like completely messes up a database and you can't undo it.  So as I mentioned earlier, we have a system that will like make snapshots of IDBs as you're going on so that you don't have to sync up as many events.  So you could easily tell solidarity to use one of those as the most recent one and then just move on from there.  And the other ones will just be like put aside and you can download them and do whatever if you want, but they won't be used for anything anymore.  Yep. Yep. So, you know, we maintain a history of all the events that are flying over the server that, you know, are more detailed, precise events, not just the stuff that was spewed kind of like collaborate.  We, there, we maintain this whole log of them as well as various snapshots of databases that automatically get uploaded from a user in that session.  So every 500 events, let's say a database gets uploaded.  And so anytime someone else joins that session now that database gets pulled down and they, you know, they join the session that database will get pulled down and then any events they have missed that have been, you know, occurred since then will be played, you know, which is going to be.  Yeah, and you can do the same thing with any of the databases pretty much that have been in like the snapshots and that way you could basically go back in time if you really need to.  Yep. So any other questions? Yeah.  So divergences offline is just difficult, right? Like there is no good solution to that.  So, so one thing if you have to make a lot of changes offline, you can basically create that new snapshot then and people who want to work from that kind of like fork of the thing can then go on from there.  But if you're the idea is you're doing most of the stuff online.  So ideally you would be able to connect and send like record the events that you've been making.  Otherwise, the server doesn't know what you've been doing.  Yeah, the simple answer is basically get forked. So, so yeah, I don't, I don't think we really have any more time.  Unfortunately, we're already over. But, you know, hit us up will be around for the rest of the day.  Or otherwise, yeah, hit us on Twitter.  Yeah.  Yeah.
recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_0_959.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_960_5495.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_5496_6131.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_6132_17807.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_17808_24347.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_24348_25355.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_25356_26087.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_26088_26891.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_26892_26951.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_26952_27071.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_27072_29195.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_29196_30815.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_30816_32495.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_32496_37643.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_37644_38519.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_38520_39647.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_39648_44183.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_44184_49991.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_49992_50051.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_50052_50099.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_50100_50147.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_50148_50207.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_50208_50255.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_50256_50495.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_50496_50555.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_50556_50603.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_50604_50663.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_50664_50879.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_50880_51011.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_51012_51167.png recon2016-15-markus-gaasedelen-nick-burnett-SolIDArity_51168_51638.png