Web-App-Encryption
CC Attribution - NonCommercial 2.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Hoffmann, Didi
Web-App-Encryption Is your data secure by default? How Django can be used to make you sleep at night. This talk will detail the different threats a web application faces today and how different types of encryption can solve many of these problems. We will discuss the whole web-stack and show various technologies to deploy secure encryption. The main focus will be on using Django as a web-frontent in a highly distributed and load optimised environment. More than ever websites have to deploy encryption to protect their users. First it has to be defined what threats the data faces and how these can be mitigated. It is vital, that a lot of though is put into what is sensible for what use case. We will describe different strategies based on a little piece of software (written in Django) we use to showcase where encryption can happen (client-browser-server-cgi-database). We will back these steps up by real life examples, numbers and benchmarks we have collected from a productive environment. Finally we will discuss some problems that arise, when hosting is out of the house, your backups are encrypted, you have a fail-over distributed environment and you as a service provider can't see the data you are hosting. ······························ Speaker: Didi Hoffmann Event: FrOSCon 2014 by the Free and Open Source Software Conference (FrOSCon) e.V.
 So good to see some faces that I saw in the previous talk. Can't have been that bad then. So coming from the same project, we're going to extend on security today. A little bit about myself. You don't know me yet. I'm born in London. I work for quite a few companies. Now I live in Berlin. I'm in the startup scene, probably everybody in Berlin is. Studied in England. I'm originally a C programmer that sort of moved up the ranks to do a lot of project management now. I tried to do some coding. I tried to do most of my coding actually. And so anyway, currently I'm working on a new project that's called D-Book, which is a document creation tool. And so the talk I held today in the morning or the early afternoon was about how we do data management. And another thing we did is security. And so security is if you get it wrong, it's very, very, very expensive. And a lot of things happen because people don't have a plan. So just the adobe hack was like $150 million. I mean, just for a stupid error. I mean, that's a scale that's unbelievable actually, right? So what we did is when we started the company, we didn't have a lot of money. But we've looked at the web stack and we sort of said, what rules can we sort of deduct? What rules can we sort of see that we can extract and act on them accordingly? Security is always a trade-off. I mean, you'll never have the perfect secure system. Never. I mean, in the worst case, people will come along and shoot you. And probably after they've shot your kneecaps, you'll tell them any password. So it's always a trade-off between how secure does it need to be and how secure can I make it? And of course, you've got the monetary aspect. I mean, if you're working in a company, you can't spend all your day working on not every company, but you can't spend all your day thinking about security, right? I mean, there are some companies, obviously, that do that. But most companies don't. Most companies let their programmers decide about security. And that is pretty much the worst case again. So I'm sort of quite convinced that a lot of the errors that come up in normal day-to-day life can be resolved through planning and the same with security. I think if you think about what threats are there towards my software, what do I have, which needs security, and what level of security does that data need, I think you can mitigate quite a lot of these threats. And what is valuable for your company? Is it the data? Is it the code? Maybe you're open source, so you don't have to protect the code, right? You can just say this is the code. Maybe it's false, so use uploads. So basically, you need a sort of roadmap. What's important for your company? And if this leaks, how threatening it is to your business model? And obviously, what to do if it happens. But in this talk, I want to sort of talk about some of the threats we identified at the beginning and how we challenged them and how we sort of tried to mitigate them, at least. So first of all, physical access is fail. If somebody has physical access to your machine, it is compromised. Basically they can do whatever they want with it. They can plug in USB devices. They can plug in keyboard sniffers. They can do whatever they want. It's very, very, if not near to impossible to secure a machine if intruders or attackers have physical access to the machine. There are some approaches and stuff, but in theory, you could change the main board and change the chip or whatever, right? So basically make sure that your servers aren't secure locations. Make sure you know where the servers are, maybe. Make sure who you trust. That's very important. Do you trust that hosting provider, that hosting provider? Maybe just pop by and say hello and see if you can get into the server room, which I actually, for an old job, I tried and actually got physical access to my server without identification without anything. And then we changed. So make sure where you have is secure. And then try to monitor your servers. So they're very easy things just to sort of see if somebody's tempered with your device. I mean, some hosting providers offer like video access that you can sort of see a video stream of your server. If you're handling bank data, that might be very sensitive data that might be something you want to think about. Maybe get the servers in your own location and think about. But then obviously the cleaning lady might have access, which was another hack, the cleaning lady actually hacked the machine. And so physical access in general is fail. So check out who has access to your machine. So the first real threat is somebody steals your hard drive. And this has happened quite a lot of times that somebody just went into a computer center and pulled out a hard drive. And it's astonishing to see how many people don't encrypt their hard drives. I'm astonished how many people don't encrypt their laptops. I mean, imagine all my company secrets are on my laptop. And it's in my bag. And it will be stolen. I know the laptop is going to be gone at some stage in my life. All my telephone, exactly, which I know these things are going to vanish. And they have vanished. And in my old office, they just opened the door and took out all laptops. So what would have happened if all our data would have been on, all our data was on there, but it was encrypted? But let's say somebody else would have come in there and looked at all our business data, looked at all our customer records, looked at all that. So I'm still really, really astonished if somebody doesn't encrypt a laptop. And I can only tell you Mac, it's super easy, Linux, it's super easy. Even OpenBSD, it's two clicks. So encrypt your hard drives. Even on your servers, basically performance hits are near to zero. So I did some benchmarking on this and we had a really, really small performance overhead. But looks on Linux is super easy. And sometimes you get some problems when you have to reboot, when you have to rebuild a RAID array or something. But it's a total trade-off that everybody should take, right? It's so easy to get going to a data center, pull out a hard drive out of RAID. The assist admin won't even know you've lost all your data. You've been compromised and you don't even know it. Yeah? So this is a typical trade-off, right? So on some servers, we just encrypt the database partition. So we just say, okay, if the machine reboots, if someone tampers with it, fair enough. Because we know so that the location is secure. But I would, I mean, on laptop, short out advice, encrypt everything. But on servers, it's a sort of trade-off thing. Because the problem is if you encrypt everything, even your boot partition and you want to reboot, it's very hard to add the password. You have to sort of remote V and C in. Which we do now. On the book, everything's encrypted. And if a machine reboots, I go through the whole horrible V and C thing, enter this stupid password. So we would try not to reboot. But yeah. So it's a trade-off again. You always have to see what data do I want to protect. So clearly, if you don't encrypt boot, anybody can come in and rewrite boot. But if somebody pulls out your hard drive, he won't be able to see your data. Whereas some people actually encrypt their database partition, but don't encrypt their swap partition or don't encrypt temp. And you've still got all the data. So the second threat, someone gets SSH onto your server. And this is like the mega super threat thing. Which happens a lot. I mean, if I look at all the SSH bots, I get that ping onto my servers trying all these different passwords. And I'm like, why are you still around? This is something that everybody should be able to really easily mitigate. And so don't have accounts on your servers is a simple thing. Don't have user accounts on your servers. Of course, if you're hosting, if you're your SSH provider, you should. But on a web server, database server, there's no reason why you should have user accounts on that server. Why should XYZ be able to go onto that server? Use configuration management to do that. There's fabric. There's so many configuration management scripts out there. No developer or admin actually really needs access to a server anymore, in my opinion. Use certificates is the easiest. Don't have passwords, basically. So a few years ago, we did a few thousand machines. I think we did 150,000 machines or something. And we had no passwords. Everything was done with certificates. A simple thing is use fail to ban, which I would also recommend. Then you can, so if somebody gets SSH as a low user permission sort of thing, you can gain some security for randomizing memory and encrypting swap and stuff like that. But this is sort of a last resort. I would always randomize memory anyway because your software, your web server might have some error. So you don't want a noob slide or something happening. Only allow SSH from certain IP addresses is probably the most easy thing to do. Just open the VPN tunnel to your network. And what was one of the most funny experiences in my life was having a honeypot that would open up once in a while to these SSH pings. And a friend copied this idea, but he did it on his production machine. And yeah, it went all quite wrong. So do a file or redirect or something to some honeypot machine. But really good. Try it. Just leave something open. A machine you don't need. Just put a password, one, two, three or something. And then just track what people do. It's hilarious. I can't agree. I'm trying to find out how to get to the end. And there's probably, they don't really know what they're doing. 80% of them are script kiddies. Just, sorry. I got a production machine. It was architected this way. Like a one hour after the installation and before it was up and down. Exactly. Before it was up and down to change everything. Exactly. And they'd score that like two weeks after. It's amazing. Because nobody could be here. They're very good at it. These scripts are extremely well-adapt. They try all different SSH versions. They try to figure out which SSH version you have if there's an error in it. So I don't have port 22 open publicly except if it's a honeypot. So if you try the machines, it's a honeypot. That's exactly what I did there. I had a different SSH port and everything that would come into 22, I had an IP table rule just redirect. Now we go in through VPN. We have a closed VPN. Especially with IP version 6, a lot of these problems go away. Because you can just say this is my private network thing and you don't have. Yes. Yes. Yes. Obviously. I mean, probably don't run it anywhere. That's important. The Amazon free instances are quite funny. You get a considerable amount of hacking out of those free instances. Amazon does shut you off at some stage there. So then we come to the next threat. Somebody uploads dangerous files. And I see a lot of open source Django projects go to PyPI and it's sickening how many people don't check stuff. They just do the standard Django upload field and they accept this as being the perfect security. Which is it not. I mean, it's Django upload. It's literally get a file and save it somewhere. It doesn't check what file it is. It doesn't check what type of file it is. It doesn't check for size. So obviously have a different process serving your media and static files to your actual files. Have an Apache or an NGX or something. Basically have it separated. Another thing which is really easy to do is verify uploaded files through Python magic or some other thing. Just do a fingerprint on the first few bytes and check is it really what I'm expecting. Sorry. So think of maximum file size. This is like a classic. Just upload a five gigabyte file and see if the server holds up. Very nice to fill up temp, for example, because what happens is it saves everything into temp, first of all. And you just open enough threads and then temp is full. Another thing, a lot of services offer you to upload files, right? So why not fill up their storage with uploading millions of files? Script it and that's it. Check file permissions and access. You can see. Yes. I'll show you the script that actually mitigates this in a minute. Check file permission and access. You'll see this a lot. And a lot of the code reviews I've done or I do, I see this done wrong. I mean, I see people CH modding 777 on their web route and because S-square light needs it. Yeah. So basically, you'll see this loads because every time I wrote this slide, I thought, I saw this being done wrong. And every time you'll see this, you'll know I've got a story to tell. Unicode was a problem, isn't so much anymore. There used to be horrible Unicode things happening that will crash Django and do horrible things. Now it's not so much. Check file names is always a good idea. Sort of see if the file already exists, steal this. This becomes very interesting when you have big production environments that are multi-threaded that start writing to different file systems and stuff. So here's just a really simple thing how to check that test.pdf is really a PDF. It's as simple as that. And like this, you can verify is the uploaded file really a PDF. Please keep in mind Django really just uploads the whole file and then does all the checks. So this is a threat you'll have to configure Apache or NGINX whatever server you're using to stop this. Then this is pretty much it, right? You see if a user, the count is over 1,000 you say F of because you know a file can only be five megabytes. You can calculate when you're going to have a problem and you'd send an email when it fills up so you can stop this. But it's as easy as that to stop end files being uploaded, which a lot of services actually allow you to do. I know a lot of Django projects where you can just upload as many files as you want and it will just crash the server. Maybe then somebody downloads files they shouldn't. This is probably one of the problems that comes from having your upload server being different from your file server, your static file server, your media file server. That not everything a user uploads might be public and just protecting it through a view saying is the user allowed to see this is not enough because obviously you're serving a file served by a different server that's not aware of what it's serving. So this URL can be served out to the whole world and everybody can see that file. Does that make sense? That was a little bit confusing but I hope it sort of came through. If you didn't understand it ask me. So there's this private media package which is awesome. It needs a lot of patching but it's still awesome. And what it basically does it says okay I'm going to do all the file serving for you but I'm going to use a patch to extend files to actually then serve the file which basically just says it says to actually serve this file for me but it does all the checking for you so you basically write the check routines and like this you can guarantee that the static files or the media files more that user uploaded actually only seen by the users that should be able to see it. You lose some performance because obviously the media file will first of all go to Django it will be checked and then Django will hand off to Apache and do it. But this is like a really thin layer and I did 10,000 requests and I didn't see any change in performance so I assume it's quite low overhead. Have a look at it it's a really cool tool. Ajax is your enemy as far as it goes with Django. A lot of things that Django brings out of the house that are really nice don't work with Ajax anymore. I've got quite a biased opinion about the Ajax but okay so if you use Ajax if you have to use it be really, really careful. A lot of things are just not done for you anymore. All this standard Django niceness that we'll come to at the end is sort of omitted like escaping doesn't work properly. You can really see it was an afterthought. It was really thought we've got this amazing model view thing and it's request and response are we need Ajax and then they sort of plonk it on top. Performance is not the best. That's probably the thing that annoys me the most. It seems to be quite slow. It's very difficult to get it right and secure. I've seen horrible code there and very unsecure code. So if you have to do Ajax maybe look in some other direction or be really careful. So a threat that I don't think is so big but we still got on our threat list is what happens if that's a security hole in Apache or NJinks. So what we do is we only include models we really, really need. So we're very careful what do we include when we start the service. We're really, really careful when we configure it. Then we only use configuration management to configure it. So nobody is allowed to just SSH into the production machine and change something because it's easy which is the classic. It has to go through the whole vetting process which we have when we do the configuration. And like this we try to avoid stupid configurations and keep your packages up to date. I mean that's probably the most, that's why probably you should not use Gen2 but use, I'm going to make a lot of enemies here. Use something that you pay for is basically what I've had the experience with. I've had very good experience with Red Hat and stuff. I know Ubuntu has very good packages and Devian too. They're very fast in fixing packages. I wouldn't know why you would have production servers that don't update their packages regularly. Especially SSH, Apache, hours count here, right? There's a zero day exploit. You're talking hours to minutes you want to fix this because you're going to be one of the targets. And if you haven't fixed it till then, you're a target. So if you rely on a small Linux distro that's compiled by two people that are sleeping at that time, you're compromised. So if you're really running production-grade stuff, dish out the money, get professional support, pay somebody for it, it's worth it. So we had this case, we were running Gen2, that's why. And they hacked us because of a zero day exploit. And since then we've been running Red Hat and CentOS and it's been amazing. Someone can read source files, which isn't that unlikely, which happens quite a lot, that people gain access and can read your source files. Probably your source code is probably not as big as a problem if you're open source and stuff. A lot of companies rely on their source code as being their business value. That's quite horrible if that leaks. But basically I've seen in settings files, I've seen passwords. I've seen passwords being one, two, three, four. Secret keys being one, two, three, four. It's not hard to get it into the OS environment and not make it visible right away. What I saw last week is Git and SSH keys. I saw the private Git or SSH key in the repo because he was too lazy to have it in his own home folder. So he could do Git pull. He had his private key in there. Yeah, don't do that. So, someone can write into my source files. That's of course really, really bad. But it can be sort of solved by using Tripwire, which is a really simple, easy tool. Works really nicely. We've never had a problem with it. Some stuff can be done with SC Linux and so forth. Git div is astonishingly easy. Just do a Git div on your directory and see what has changed. And probably you know the files that have changed anyway. But if someone can write to it, they normally won't check the Git repository. And again, check your permissions. Again, I've seen break-ins happen because there were bad permissions set. Someone can execute my code. This is probably another really bad case, which is a really easy case is once a user uploads a file and you've written it, just change the permission. So you can't modify it and you can't execute it. Just set it so you only read only every file that is uploaded by the user. You get rid of a lot of problems like that. And it's one line of Python code. So then you have to think about where am I saving my data. And so somebody is writing your code and that's really, really bad. But what else can you do? In theory, he can gain access to the database, right? So what we do is we disallowed a leak on the database. And we go even so far, we say that you're not allowed to update the database. So if somebody can go in, he can still read all the stuff in theory, but he can't modify our data. So he can't inject false data. Have buckups? Really, I'll come to that a little later, which I cannot stress enough. And obviously have source files read only. Someone can modify or read your backup. This is like, so actually what happened in this story is they had this huge server and they backed up on these tape drives. And that night somebody kicked in the door, took the server, and of course he took the tape drives. I mean, they're there. So they had all this encryption and everything on their machine, but the tape drivers weren't. So that company actually went bankrupt because of that. They actually had, they lost all their business because they had to notify all their customers that they were compromised and all the customers left. So yeah, as dark as that may sound, think about your backup. Use an extra process and a different user. Use write-only storage, which is really simple, just like a dropbox. You push it in there. If somebody compromises your machine, he can't compromise your backup. Obviously use encrypted storage on your backup and transmit it encrypted. When I was a student, we would just put a hub in the network and would sniff all the network of the whole network. And in my student house, 10 geeks, amazing stuff happened. So seeing what everybody was doing, it's so easy, right? I mean, just get a little hub and plug it in there. And instantly you see everything. Check the certs. So the typical thing is somebody brings up a machine in your network, starts a DNS server. You're using some DNS server that takes longer to respond and suddenly he's a backup machine, right? So because he answers faster than the DNS? Yeah. Yes, exactly. That basically goes with use encrypted storage and transmission. So yes, yes, that's very true. Very true. I forgot that. Yeah, that's a very good idea. So basically encrypt your backup and sign it. Check the cert. So have a certificate on your backup server and check that it's really the backup server. Don't just push it to any IP address that you hope is in the machine because you can't rely on DNS. DNS is really unsecure. Just start a DNS server in any network and you can take over pretty much anything. Never throw away backups. You never know when you were hacked. You never know when you might need it. The cost of keeping a backup is near to nothing nowadays. So why would you throw away something like this? The next threat, somebody can read your traffic and this is incredibly easy. Use HTTPS. It's known technology. Get a signed cert. I know it's snake all. I know it's useless actually. But people will expect it. And people don't trust your service if you don't have it. And I mean, it doesn't really help if you can't, I mean, enough certificate authorities have been hacked and new certificates have been issued. But I think it's worth it. But just use HTTPS. Sniffing the network here is quite astonishing how many people get their email unencrypted. Just doing pop. I'm just saying that's all unencrypted. Just encrypt. You don't know where your traffic is going over. You just, you don't. I mean, enough people here have got their laptops open, their browsing websites and anybody can see it, right? It's an open network. Or your email that you get over pop if it's not encrypted, everybody can read. So use these encryption technologies. They're standard. They work. The NSA can crack it, but the NSA will come in and shoot your kneecaps away, right? So we're not talking NSA-grade security here. We're talking the average guy with a laptop sitting in a conference room doing a network sniffing. Who's logged on to you? I can see it all here. In Django, there are two very easy configuration setting files or setting parameters to ensure this. Obviously, you don't want your cookies to leak over an unencrypted network, especially not your session cookie. So somebody can read your database. So there's a classical thing when somebody hacks the machine, he does a select star on the user database and saves that, right? And then he adds a new user that has admin rights. So he has admin on the whole system. Those are sort of the normal steps you'll see in any compromising situation. First of all, get all the data you can, all the password hashes. I deliberately didn't go into hash your passwords. If you don't know that, you shouldn't be doing programming. Sorry. If you don't hash your passwords, if you keep your passwords in plain text, leave, get a new job. I'm sorry, you've lost it. I have to be honest with you, if somebody saves clean text passwords, just, I don't know, build, no, just go. I have no advice for you. So what we do now is we do these really interesting Post-Escal stuff with Table Update and stuff. It's not very scalable and not very nice. So we do these other things that we override the delete method and stuff, so if somebody can execute stuff, we say you're not allowed to delete. And we override the base item manager, the models manager, to do like filter, we do some checks in there. But obviously, in a compromising situation, that doesn't really help because somebody can just uncomment those functions. I'm sort of relying if somebody can edit the code, you can't restart the server, so we've sort of got some security there. Yes, yes. So we did in the model now, which is probably the worst idea thinking about it. But anyway, we just don't allow in the database. So this is just sort of backup for security reasons. So I've been working on something which is not finished yet, which I hope I'll finish at some stage. And I'd love your input on this. This is one of the reasons. Yes. Post-Escal, that's what we do. That's what we do. So you have this, basically you have this process, well, permission. So this function is just, we add it to make the life harder and for bad programmers to get a bad response. Because actually somebody was coding and he's like, it doesn't work, it doesn't work. So we added this. Yeah. But exactly. So the problem you have there, if somebody's managed to get into your Python executable, he can override all that. Because what he can do, he can just put the super function here instead of that, because he can just take that method and overwrite it. So if somebody, so the threat scenario is somebody comes in, he manages to open the Python shell in your browser or in your web server somehow. And obviously all your code is in there. And then he can obviously overwrite everything. So the security in the app doesn't really help. Because the only real threat scenario is, A, you've got a bad programmer. Yes, this helps. But you sort of assume your code should be good. Or somebody has come into your web server and he's got access to your executable files and then he can just overwrite this, delete it, whatnot. So you really can have some people or some code to delete. Yes, yeah, of course. Exactly. Exactly. So this would be the right thing to write that into, into the delete. All you, all, what we do is we also overwrite the filter and the all and the get. Exactly. And then you'll sort of secure that everything that builds on top won't, won't destroy anything. But anyway, I didn't find anything I thought was secure. The process well permission stuff is really hard and really easy to get wrong. I didn't know anybody. I couldn't find anybody that really knew it. That was really weird. I hired people and we couldn't because nobody really knew. And we can't fix it in the code. So what can you do? You need to lay in between. And I'd love you to put on this because I've been coding this basically. And I don't know if it really, I hope it makes sense. But maybe you've got some ideas. So I'd love your feedback after in a few minutes. So basically what I've done is I've written a server that identifies himself as process well, that he verifies the login credentials with a real server. And then he takes some queries and he parses these queries and just so the queries Django creates for the OIM are really, really simple. I mean, most of them, you know. So what I do is I, I learn queries from my test cases because I've got tests that run and I know which test cases they run and I also have like some user behavior scripts that I extract from live data. So I see how does the user react? What does the user submit? And I use this to train my security model. And then for now, I have to tag variables manually. But hopefully in the future, the script will learn automatically what's available and what's not. And so I write a regex which says this is how this looks like. And then as soon as I, and then I train it, I've got this data set, I train it, then I've got a big regex tree at the end of the day basically. And as soon as something doesn't match this regex tree, I say, okay, I don't know this query, what am I going to do? And so when we deploy this, we'll get a few notifications, we'll have it running on a shell and the query will just stuck stuck there and I'll just do, yes, okay, this looks good. Well, this doesn't look good. And then it will learn this, hopefully, add this query to the training set and so forth. And so hopefully at some stage we'll have a complete regex with all the queries we will expect from the server. So then I want to, or what I do now is actually I assign a trust variable between zero and one to each query. This is really crude. But ideally you would say, okay, this is my logging database, queries that have a trust of 0.2, they can go on my logging because I don't really care about that. But on my user database, I only want queries that have a trust variable of one, which basically 100%, this is really trustworthy, specifically allowed in the regex. And it can also have handcrafted rules. So basically this is like the standard rule that's disallowed, select start from users. And I'm like, no, this is not happening. And I forgot the semicolon here, by the way. But select start from users where username equal x. That sounds like something a normal app would run. So I say, yeah, this is okay. What I want to add in the future is that I can then specify this is only allowed to return one record. Because I could say, look, if it returns two records, something's wrong here. Or you're not allowed, users is never, actually users is never allowed to return more than one record if you just do authentication on your app, right? And everything that returns more than one record, I'm like, no. But currently, I just pipe the result through. Don't tell me this already exists, please. If you know of something, send me an email tomorrow morning or when I'm asleep or tonight when I'm drunk. Yeah, so I'll get your feedback on that, what you think. I'd love to pick your brains on this. Then another threat, someone sees something you shouldn't, and Django is amazing in this respect. I really like Django. You just do decorators at logonrequired, and we actually wrote our own decorator, check permission that the user has to be owner, and we do edit authors, and that's it. And this does all the checking for you. And this is so easy, so nice. From the experience, throw errors. If prop don't get something, and then if it's empty, just propagate it through and throw a 500 somebody at the end. Throw the errors early as possible. Weird things happen with null values. The next threat, I have to hurry up. Someone can read your database. So let's say somebody hacks your machine, and as I said, the first thing he does is he does select star on users, right? So what you want to do is you want your database to be encrypted, right? You want the user to add some sort of key, the database encrypted. So there's this really nice encryption library, which I don't know how to pronounce, which is by Google. And what we did is we overwrote the dev to Python and get prep value, so basically all the communication Django has with the database, and as long as it's a text field and a char field. You don't encrypt integers, is encrypted by the key of the project. And like this, if somebody manages to do a select star on our databases, he'll at least just see garbage for the text fields or text data. In the future, we want to roll this out on images too. The problem with images is CPU power, that we don't have the CPU power to decrypt an image on the fly and serve it. We had it and we needed these huge servers we couldn't afford and it was just really, really hard to do properly and fast. But for the database, it works fairly, because the text is fairly small, fairly condensed, it's fairly fast. We've got some performance overhead, obviously, and you lose some queries, you shouldn't be doing anyway like queries and stuff like that. But normally these result in full table scans in most cases, so you shouldn't be doing them anyway, you should have an index. Of course you lose indexing. I started writing an encrypted index, but I haven't finished yet. So CPU power is needed for this, but I think it's worth it because the actual overhead, we have, CPU is not what's limiting us on the machines. And Django Extensions has something like this, and a lot of influence came from Django Extensions that we copied a lot. Yes, so this is on a field level. So what we write is models.encrypted text field, models.encrypted char field, and it takes exactly the same parameters, it inherits from those fields. But the only thing we do is when we serialize to the database, we encrypt it and when we basically get the result from the database, we decrypt it. So obviously you've still got your clean text in memory. I sort of looked at memory encryption and couldn't find anything good, and that was performant. OpenBSD has done a lot in that respect, but I sort of assume if somebody's hacked our memory then, yeah, we're f'd anyway. Then I think I'm hearing more and more now is the users don't trust the service anymore, especially with this whole NSA thing coming up, with this whole, yeah, the service is out to get us. And a lot of services I see are like, oh, yeah, we encrypt everything client-side, and we only upload encrypted stuff to the server. And then sort of, we had a very interesting discussion at the Berlin Django stumptish about this. And then so they trust the server and giving them the right JavaScript, but they don't trust the server in handing their data. So surely if the server is bad, we'll just inject some JavaScript, or we'll just inject a bad key, and then we could still decrypt it, right? So this is a discussion I find sort of followed up by quite the people that haven't followed it through yet, if that makes sense. If the user doesn't trust you, then you shouldn't trust the JavaScript, he's getting from you either, right, because it sort of defines the whole logic. So I agree, encryption client-side and JavaScript is really cool. If we would get this to everywhere, it would be amazing, especially the load on the servers would go down. This is why would the user not trust you, but trust your JavaScript. So a way we're investigating right now is that we will have a plug-in that comes from Mozilla or whatever, and that verifies the JavaScript that comes from us, and all the JavaScripts are open source. So even if we would inject some bad JavaScript, the plug-in would alert, and this adds to security if someone would hack our servers and inject bad JavaScript, also the plug-in would come up and say, look, this is not true. And like this, we sort of hope to have a transparent way of doing it. I don't know if we'll be doing this, yeah. Yes. So basically, encryption in JavaScript is really, really slow. So the first problem I ran into is true random numbers. I didn't find any good implementation in JavaScript of a random number generator. The thing I found was, yeah, yeah, it's time, which isn't random. So the first thing, the API will hopefully, they're talking about a true random number generator. And then the problem with JavaScript is that all the modern chips, they have these really good crypto chips. That's why hard disk encryption doesn't cost us anything. That's why all these encryption stuff is really, really fast because they've got a special chip on their chips. And what JavaScript does, if you have all these encryption libraries, it runs on the main CPU. And that's really slow because the CPU is not made for encryption. So I don't... You know, you're publishing your own source of the set. Exactly. Yes. So the problem still stays, yes, you can encrypt it, even through the new JavaScript API stuff, you can encrypt it really nicely on the client, but you're still pushing it to a service you don't trust. So there's a lot of...I think there's a lot of work to be done there. Maybe some offline JavaScript features, yeah. I was saying one possibility was to produce something that I see in the criticism of the DNS and... You know, basically what you're doing is you're operating... Yes. Yes. In the DNS, for example, the incident that happened, I was watching the whole process with DNS. I read about that too, but I don't know any browser that supports it. Yeah. Yeah, exactly. So we sort of... Yeah, but I read about that and that's a very good idea actually to do... One problem I see with that approach is that updates are going to be quite slow because, I mean, we roll out two, three times a day. So we'd have these very small and we're very fast. Knowing DNS is kind of your... Yeah, exactly. You need that. But... So we're just experimenting with this plugin. This is just sort of the direction we're going. Yes? No, you are... Not yet. Not yet. We're playing around with it. Yeah. Now I am... The plugin is okay, but I still need to... So ideally the plugin... Yeah. So basically what the plugin does it verifies the header. So what it does is it takes a whole head section and does a hash on it. But of course you can put scripts somewhere in the text, but we haven't done that yet. But yeah, so the idea is that if you have the JavaScript open source and everybody can sort of generate the hash in some way, he has a plugin where the hash is displayed, he can sort of trust it. There's no... I haven't found a really nice solution actually. Otherwise Django is amazing. It's all there, cross out scripting, use it, enable it, use it. It just does such a good job at it. Never use the safe decorator basically. You'll get in a whole world of pain. It's just so nice. I mean, coming from program web servers and C and just seeing our SQL injection is dealt with. And you're like, this is amazing. An email header injection, all this stuff is just done. Then again, a personal story. I've seen threat detection working perfectly, but nobody responding to the email. And actually this happened a month ago. Oh yeah, it was his job. It was his job and it was a circle. And so nobody did anything. And the server was hacked and it was standing there for days until somebody sort of looked at the server so he's still hacked. And he's like, yeah, it was him and him and him. So be sure that you notice when you're hacked at least. There are some papers, some research done on this by University of Oxford that say it's more important to notice when your server's been hacked and just destroy it than actually preventing hacking. I don't totally agree with that, but I sort of like the approach that you sort of monitor, really hardcore your server. And if you notice anything weird with that server, you just kill it. You just freeze it for analysis later, but it's not in the production server anymore. So I can't stress this enough. Have a central log. I've seen, let me finish and then have a central log. I know people that have 30, 40 servers, huge servers that don't have a central log. Emails are not enough. I get a few hundred emails every day. If that goes into spam, I will never read it. Have error follow-ups. So what we do now is all our error messages go to basically a bug tracking or support tracking system. And so if some monitoring service detects something, it creates a tracking cookie, our tracking task. And one of us has to follow it up because otherwise it's open. So one of us has to go there and say, okay, I've done this. I've solved it. And we have notification propagation. So if somebody hasn't acted on a task in 24 hours, we email his boss, which is incredibly effective. You had a question. Yeah. Yes. So the idea of the security is that we have to do a security check. Yes, so the idea is that you pull up new servers, obviously while you're doing it. But I read the paper two, three years ago, and I was like, I saw a lot of flaws in it too. And you've got like, you've got like, what happens if the notification is wrong, right? You just shoot off a user every time he tries to upload a picture of his family. And somehow it's a different color encoding. And you're like, no, I don't like this color encoding. And the session times out, which is not user experience you want, right? And I find it very hard. So what they did, they froze the virtual machine and they then analyzed it later. But I quite like that we get an email or ticket. And then we just go in and check right away what's happening. And we can sort of see if it's a real hack, which hasn't happened so far, or if it's something we didn't expect and can act on it and stop more notifications coming in. Yeah, so basically ask me stuff and check out Dbook. Yes. So we've got everything virtualized anyway, because we've got multi. I personally think Docker makes a lot worse because I know how programmers do Sys stuff. And I wouldn't trust, OK, there are some programmers that can do it without question. But I've seen some of the fabric scripts I've seen in my time when programmers try to do deployment. And it's probably this whole DevOps thing, right? It's always Sys admins against programmers. And I quite like just deploying from a Git tree and doing the whole staging old school approach. Docker is very, very good if you have lots of developers with different machines that deploy rapidly. Definitely. I wouldn't, I mean, how does it contain security threats in the respect? It might contain it in the Docker, but it also does it in a virtual machine, right? So yeah. But we do that in virtualization. Yeah, it's process isolation, which we do with virtualization. But we do it actually with physical machines nowadays. But yeah, I agree. Process isolation, I'm very astonished. I haven't come to the desktop yet, because a few years ago I had my Firefox in a virtual machine because I was so afraid of security holes in Firefox. But I don't know why, I mean, process isolation has been around for 30, 40 years. And it hasn't really happened yet. But I agree. Yeah, I haven't played around with Docker too much to do it. But yeah, we do it with virtualization. That's basically what I can say, which is just a bigger layer, right? Yeah? So we do it on the local machine. Basically we have a cron job that has a nifty Python script. And then we have Hadoop jobs that take all the log files in and do monitoring on that. But we do that two times a day. We have a big Hadoop job that kicks in with all the log files, because we have some historical analysis going on there. And for example, what we also do is we have user behaviors. So imagine a timeline, and you have clicks or actions on this timeline. And so what we do is user behavior changes over time. So a user will become more proficient as the longer he uses the software. But what you can do is you can sort of see how advanced the user is. And then if he does something really weird at some stage, you can sort of see, okay, this is sort of what he should be doing at the beginning. And so it says girlfriend using it mostly. That's the average case where we've got quite nervous and there was just this girlfriend kid or something using his account. But so we do that in Hadoop. We do classical map reduce. We're looking at stratosphere to do that was their stream processing, which isn't ready yet. I haven't found a really good real-time stream processor yet in big data. And we do really, really slim real-time logging on the machine. Basically, just log file analysis when we stream it off to the master. Yeah, that's an interesting question that I hope nobody would ask that. So the future is that we store the keys on the client and he gives us basically something to decrypt it. What we do now is we have a really locked-down database and that's all I'm going to say. So we still have all the keys in one central repository just because we haven't had the time yet to have a proper key distribution. No, so select or users doesn't work now anyway. But no, it's a total different system. So our key storage is something I don't want to talk too much in public, but I know it's not 100% secure because ideally what you would want to do is have the password generate some sort of key that does a decryption. But I haven't found a really nice way that if somebody loses his password, all we have the sharing feature that many people can access and edit a document. And it becomes incredibly complex if you want to do this key encryption like that. And maybe I'm not good enough at math or something, but I haven't figured this one out yet. So now for now, because time is limited, we just have a really secure key storage which is handwritten, which hopefully is secure than any data. It's a modified Kerberos LDAP thing, basically. So we have 16 shards, so roughly 20 machines, probably a little bit more, and we use SCP. And then we do analysis on it. Clearly, I haven't had the time yet. I know there are lots of solutions out there. I know there's a lot out there I just haven't had the time yet to do. So for now what we do is we do SCP, and then we just have Mooneen that reads those files, which works quite well because we just have an SCP tunnel that is open, and we stream to the file. So you get all the advantages of SSH, it being encrypted, it being fairly fast. We have the key authentication and all that stuff. And we just pipe. We just tree, basically. And SSH has been incredibly good. And it works. It was a hack, and it works. And we detect when the SSH tunnel comes down, so we know when a machine is sort of having a problem. He was first, sorry. No, I guess we're talking about finding lots of life. So we just need to have a compromise. We can do a post-mortem analysis. Exactly. Exactly. And it's the same thing you said, that there's compromise in just a minute of production, but then you should go post-mortem analysis on the post-mortem analysis. Exactly. And like you have a problem memory, over the binaries and the post-mortem, everything you can get, it was dense, out, and sort of, literally where it is, where it is, and there it is. So basically just freeze it, right? We have virtualization, just freeze the machine and put it somewhere where it's safe and then analyze it, not being live on the network. Yeah, yeah. The problem with the entity is, oh, he can't detect or edit with my data. Yes. Probably he can't edit it. No, he can't edit it. Why don't you do so? Probably he could modify the file pointer and, yeah, probably he could modify it. Now you all know, right? And the internet knows. We have to change this tonight. Yeah, of course, it's not perfect. I agree. Obviously, I just hadn't had the time yet to optimize it. I'm a very small team, and I'm luckily exponential growth, which is horrible right now. But, yeah. It will go to the honeypot. It's fine. Yeah. We've already known about production. Yes. I mean, obviously, you need one user to get in. We knew that, so that's good. And originally, we knew that we were running the configuration. Okay, so no user that, so what we have, we have a normal login user. Root is not allowed in by SSH. So we have the standard, I don't actually know the password, because on user creation, I just set a massively random password. But that user has sudo, and then can get root, and that's how we run the configuration management. But, yeah, you always need one user to get in. I wouldn't know a way. I mean, what I've been thinking about is using LDAP to basically have one central user that we sort of manage, but I don't see the big advantage yet. Yeah. Yeah. Yes. Yes. I mean, we have accounts. We have WW data. We have process squirrel. But obviously, those users don't have a shell. They don't have SSH access. They don't have, so we still have that one user that we can get into. But only a few people have the certificate to go into. And I hope that people don't do stupid things. Yeah. After you finished the conference, you know, it's like the other, our thing is the password. And you can say, hey, I've got my TPC S35 here. Based on all of our servers, and be sure that this one user is available on our fan site. It's a cylinder. How do they call that? They've got the secure storage thing, haven't they? A silo or something? Salt has this really cool secure storage container where you can put your ETC password in it and we'll distribute it in a secure way. But I... So if you're in a void, there's someone going to add something like that because it files a whole bunch of servers and takes my orders. So actually you can configure a salt that will actually phone out. So you could get rid of your user if you really wanted to by getting salt to get the salt master and get this configuration file from the salt master periodically. But I wouldn't, I haven't dared to do that yet. So again, physical access is fail, right? But just a few days ago, exactly that happened, I locked myself out of the machine. Not for this project, for another project. And I just went to the virtual machine, to the console. I rebooted the machine, put single at the end of Grubb and I was root. It was that easy, right? So that's probably something I should have added to. The dashboard of your virtual machines is probably the most important thing you've got. If that password isn't that long, because as soon as you can reboot the machine and you've got the dashboard, it's compromised, right? So yes, yes, that will come, definitely. I mean, we've got that with email and all that already. But I've got the app on my phone and I haven't gone that far to have the SSH stuff go that far. So in the future, we'll probably have that. So the PAM two-factor authentication rules have been around for years and it's well tested. Years back, I mean, this was 2007, we used LDAP with two-factor authentication, Kerberos, and it worked perfectly. I mean, you just have to buy these quite expensive RSI tokens and that were hacked by the NSA, by the way. We bought thousands of them and everybody had to carry it around and it was horrible to maintain. And then the news came, NSA had hacked them and we were like, ah. But yeah, so that worked really nice. So getting two-factor authentication in Linux is editing five lines, I think, of code. Just add the PAM rule in the PAM stack and it works. The biggest problem is the synchronization of the seed in all servers. There is a product here, a open-key, I don't know, that's basically a separate that's comprised of the seeds of the two-factor tokens, the inside of the seed. You have a range, if you want, of the range, the other server, the one-fringer, that's it. Exactly, yeah. We just did it with ranges. We didn't have a synchronized seed. If you want to fix that, you can use the antipyro-spoke code, you have much more space. Exactly. Yes, I agree. Probably the future, I think, is with phones. Everybody's going to have an app on their phone and it's going to phone in and phone out because my phone's online all the time. But I haven't seen anything useful there with Linux. Maybe there is. Questions? Or comments? Nobody commented on my genius idea of the self-learning SQL algorithm. This is hours of my life. That was the main point. Cool, so thank you very much for listening. I'll see you at the beer booth thing.
 So good to see some faces that I saw in the previous talk.  Can't have been that bad then.  So coming from the same project, we're going to extend on security today.  A little bit about myself.  You don't know me yet.  I'm born in London.  I work for quite a few companies.  Now I live in Berlin.  I'm in the startup scene, probably everybody in Berlin is.  Studied in England.  I'm originally a C programmer that sort of moved up the ranks to do a lot of project  management now.  I tried to do some coding.  I tried to do most of my coding actually.  And so anyway, currently I'm working on a new project that's called D-Book, which is  a document creation tool.  And so the talk I held today in the morning or the early afternoon was about how we do  data management.  And another thing we did is security.  And so security is if you get it wrong, it's very, very, very expensive.  And a lot of things happen because people don't have a plan.  So just the adobe hack was like $150 million.  I mean, just for a stupid error.  I mean, that's a scale that's unbelievable actually, right?  So what we did is when we started the company, we didn't have a lot of money.  But we've looked at the web stack and we sort of said, what rules can we sort of deduct?  What rules can we sort of see that we can extract and act on them accordingly?  Security is always a trade-off.  I mean, you'll never have the perfect secure system.  Never.  I mean, in the worst case, people will come along and shoot you.  And probably after they've shot your kneecaps, you'll tell them any password.  So it's always a trade-off between how secure does it need to be and how secure can I  make it?  And of course, you've got the monetary aspect.  I mean, if you're working in a company, you can't spend all your day working on not every  company, but you can't spend all your day thinking about security, right?  I mean, there are some companies, obviously, that do that.  But most companies don't.  Most companies let their programmers decide about security.  And that is pretty much the worst case again.  So I'm sort of quite convinced that a lot of the errors that come up in normal day-to-day  life can be resolved through planning and the same with security.  I think if you think about what threats are there towards my software, what do I have,  which needs security, and what level of security does that data need, I think you can mitigate  quite a lot of these threats.  And what is valuable for your company?  Is it the data?  Is it the code?  Maybe you're open source, so you don't have to protect the code, right?  You can just say this is the code.  Maybe it's false, so use uploads.  So basically, you need a sort of roadmap.  What's important for your company?  And if this leaks, how threatening it is to your business model?  And obviously, what to do if it happens.  But in this talk, I want to sort of talk about some of the threats we identified at the beginning  and how we challenged them and how we sort of tried to mitigate them, at least.  So first of all, physical access is fail.  If somebody has physical access to your machine, it is compromised.  Basically they can do whatever they want with it.  They can plug in USB devices.  They can plug in keyboard sniffers.  They can do whatever they want.  It's very, very, if not near to impossible to secure a machine if intruders or attackers  have physical access to the machine.  There are some approaches and stuff, but in theory, you could change the main board and  change the chip or whatever, right?  So basically make sure that your servers aren't secure locations.  Make sure you know where the servers are, maybe.  Make sure who you trust.  That's very important.  Do you trust that hosting provider, that hosting provider?  Maybe just pop by and say hello and see if you can get into the server room, which I  actually, for an old job, I tried and actually got physical access to my server without identification  without anything.  And then we changed.  So make sure where you have is secure.  And then try to monitor your servers.  So they're very easy things just to sort of see if somebody's tempered with your device.  I mean, some hosting providers offer like video access that you can sort of see a video stream  of your server.  If you're handling bank data, that might be very sensitive data that might be something  you want to think about.  Maybe get the servers in your own location and think about.  But then obviously the cleaning lady might have access, which was another hack, the cleaning  lady actually hacked the machine.  And so physical access in general is fail.  So check out who has access to your machine.  So the first real threat is somebody steals your hard drive.  And this has happened quite a lot of times that somebody just went into a computer center  and pulled out a hard drive.  And it's astonishing to see how many people don't encrypt their hard drives.  I'm astonished how many people don't encrypt their laptops.  I mean, imagine all my company secrets are on my laptop.  And it's in my bag.  And it will be stolen.  I know the laptop is going to be gone at some stage in my life.  All my telephone, exactly, which I know these things are going to vanish.  And they have vanished.  And in my old office, they just opened the door and took out all laptops.  So what would have happened if all our data would have been on, all our data was on there,  but it was encrypted?  But let's say somebody else would have come in there and looked at all our business data,  looked at all our customer records, looked at all that.  So I'm still really, really astonished if somebody doesn't encrypt a laptop.  And I can only tell you Mac, it's super easy, Linux, it's super easy.  Even OpenBSD, it's two clicks.  So encrypt your hard drives.  Even on your servers, basically performance hits are near to zero.  So I did some benchmarking on this and we had a really, really small performance overhead.  But looks on Linux is super easy.  And sometimes you get some problems when you have to reboot, when you have to rebuild a  RAID array or something.  But it's a total trade-off that everybody should take, right?  It's so easy to get going to a data center, pull out a hard drive out of RAID.  The assist admin won't even know you've lost all your data.  You've been compromised and you don't even know it.  Yeah?  So this is a typical trade-off, right?  So on some servers, we just encrypt the database partition.  So we just say, okay, if the machine reboots, if someone tampers with it, fair enough.  Because we know so that the location is secure.  But I would, I mean, on laptop, short out advice, encrypt everything.  But on servers, it's a sort of trade-off thing.  Because the problem is if you encrypt everything, even your boot partition and you want to reboot,  it's very hard to add the password.  You have to sort of remote V and C in.  Which we do now.  On the book, everything's encrypted.  And if a machine reboots, I go through the whole horrible V and C thing, enter this stupid  password.  So we would try not to reboot.  But yeah.  So it's a trade-off again.  You always have to see what data do I want to protect.  So clearly, if you don't encrypt boot, anybody can come in and rewrite boot.  But if somebody pulls out your hard drive, he won't be able to see your data.  Whereas some people actually encrypt their database partition, but don't encrypt their  swap partition or don't encrypt temp.  And you've still got all the data.  So the second threat, someone gets SSH onto your server.  And this is like the mega super threat thing.  Which happens a lot.  I mean, if I look at all the SSH bots, I get that ping onto my servers trying all these  different passwords.  And I'm like, why are you still around?  This is something that everybody should be able to really easily mitigate.  And so don't have accounts on your servers is a simple thing.  Don't have user accounts on your servers.  Of course, if you're hosting, if you're your SSH provider, you should.  But on a web server, database server, there's no reason why you should have user accounts  on that server.  Why should XYZ be able to go onto that server?  Use configuration management to do that.  There's fabric.  There's so many configuration management scripts out there.  No developer or admin actually really needs access to a server anymore, in my opinion.  Use certificates is the easiest.  Don't have passwords, basically.  So a few years ago, we did a few thousand machines.  I think we did 150,000 machines or something.  And we had no passwords.  Everything was done with certificates.  A simple thing is use fail to ban, which I would also recommend.  Then you can, so if somebody gets SSH as a low user permission sort of thing, you can  gain some security for randomizing memory and encrypting swap and stuff like that.  But this is sort of a last resort.  I would always randomize memory anyway because your software, your web server might have  some error.  So you don't want a noob slide or something happening.  Only allow SSH from certain IP addresses is probably the most easy thing to do.  Just open the VPN tunnel to your network.  And what was one of the most funny experiences in my life was having a honeypot that would  open up once in a while to these SSH pings.  And a friend copied this idea, but he did it on his production machine.  And yeah, it went all quite wrong.  So do a file or redirect or something to some honeypot machine.  But really good.  Try it.  Just leave something open.  A machine you don't need.  Just put a password, one, two, three or something.  And then just track what people do.  It's hilarious.  I can't agree.  I'm trying to find out how to get to the end.  And there's probably, they don't really know what they're doing.  80% of them are script kiddies.  Just, sorry.  I got a production machine.  It was architected this way.  Like a one hour after the installation and before it was up and down.  Exactly.  Before it was up and down to change everything.  Exactly.  And they'd score that like two weeks after.  It's amazing.  Because nobody could be here.  They're very good at it.  These scripts are extremely well-adapt.  They try all different SSH versions.  They try to figure out which SSH version you have if there's an error in it.  So I don't have port 22 open publicly except if it's a honeypot.  So if you try the machines, it's a honeypot.  That's exactly what I did there.  I had a different SSH port and everything that would come into 22, I had an IP table  rule just redirect.  Now we go in through VPN.  We have a closed VPN.  Especially with IP version 6, a lot of these problems go away.  Because you can just say this is my private network thing and you don't have.  Yes.  Yes.  Yes.  Obviously.  I mean, probably don't run it anywhere.  That's important.  The Amazon free instances are quite funny.  You get a considerable amount of hacking out of those free instances.  Amazon does shut you off at some stage there.  So then we come to the next threat.  Somebody uploads dangerous files.  And I see a lot of open source Django projects go to PyPI and it's sickening how many people  don't check stuff.  They just do the standard Django upload field and they accept this as being the perfect  security.  Which is it not.  I mean, it's Django upload.  It's literally get a file and save it somewhere.  It doesn't check what file it is.  It doesn't check what type of file it is.  It doesn't check for size.  So obviously have a different process serving your media and static files to your actual  files.  Have an Apache or an NGX or something.  Basically have it separated.  Another thing which is really easy to do is verify uploaded files through Python magic  or some other thing.  Just do a fingerprint on the first few bytes and check is it really what I'm expecting.  Sorry.  So think of maximum file size.  This is like a classic.  Just upload a five gigabyte file and see if the server holds up.  Very nice to fill up temp, for example, because what happens is it saves everything into temp,  first of all.  And you just open enough threads and then temp is full.  Another thing, a lot of services offer you to upload files, right?  So why not fill up their storage with uploading millions of files?  Script it and that's it.  Check file permissions and access.  You can see.  Yes.  I'll show you the script that actually mitigates this in a minute.  Check file permission and access.  You'll see this a lot.  And a lot of the code reviews I've done or I do, I see this done wrong.  I mean, I see people CH modding 777 on their web route and because S-square light needs  it.  Yeah.  So basically, you'll see this loads because every time I wrote this slide, I thought,  I saw this being done wrong.  And every time you'll see this, you'll know I've got a story to tell.  Unicode was a problem, isn't so much anymore.  There used to be horrible Unicode things happening that will crash Django and do horrible things.  Now it's not so much.  Check file names is always a good idea.  Sort of see if the file already exists, steal this.  This becomes very interesting when you have big production environments that are multi-threaded  that start writing to different file systems and stuff.  So here's just a really simple thing how to check that test.pdf is really a PDF.  It's as simple as that.  And like this, you can verify is the uploaded file really a PDF.  Please keep in mind Django really just uploads the whole file and then does all the checks.  So this is a threat you'll have to configure Apache or NGINX whatever server you're using  to stop this.  Then this is pretty much it, right?  You see if a user, the count is over 1,000 you say F of because you know a file can only  be five megabytes.  You can calculate when you're going to have a problem and you'd send an email when it  fills up so you can stop this.  But it's as easy as that to stop end files being uploaded, which a lot of services actually  allow you to do.  I know a lot of Django projects where you can just upload as many files as you want and  it will just crash the server.  Maybe then somebody downloads files they shouldn't.  This is probably one of the problems that comes from having your upload server being  different from your file server, your static file server, your media file server.  That not everything a user uploads might be public and just protecting it through a  view saying is the user allowed to see this is not enough because obviously you're serving  a file served by a different server that's not aware of what it's serving.  So this URL can be served out to the whole world and everybody can see that file.  Does that make sense?  That was a little bit confusing but I hope it sort of came through.  If you didn't understand it ask me.  So there's this private media package which is awesome.  It needs a lot of patching but it's still awesome.  And what it basically does it says okay I'm going to do all the file serving for you but  I'm going to use a patch to extend files to actually then serve the file which basically  just says it says to actually serve this file for me but it does all the checking for you  so you basically write the check routines and like this you can guarantee that the static  files or the media files more that user uploaded actually only seen by the users that should  be able to see it.  You lose some performance because obviously the media file will first of all go to Django  it will be checked and then Django will hand off to Apache and do it.  But this is like a really thin layer and I did 10,000 requests and I didn't see any  change in performance so I assume it's quite low overhead.  Have a look at it it's a really cool tool.  Ajax is your enemy as far as it goes with Django.  A lot of things that Django brings out of the house that are really nice don't work  with Ajax anymore.  I've got quite a biased opinion about the Ajax but okay so if you use Ajax if you have  to use it be really, really careful.  A lot of things are just not done for you anymore.  All this standard Django niceness that we'll come to at the end is sort of omitted like  escaping doesn't work properly.  You can really see it was an afterthought.  It was really thought we've got this amazing model view thing and it's request and response  are we need Ajax and then they sort of plonk it on top.  Performance is not the best.  That's probably the thing that annoys me the most.  It seems to be quite slow.  It's very difficult to get it right and secure.  I've seen horrible code there and very unsecure code.  So if you have to do Ajax maybe look in some other direction or be really careful.  So a threat that I don't think is so big but we still got on our threat list is what happens  if that's a security hole in Apache or NJinks.  So what we do is we only include models we really, really need.  So we're very careful what do we include when we start the service.  We're really, really careful when we configure it.  Then we only use configuration management to configure it.  So nobody is allowed to just SSH into the production machine and change something because it's  easy which is the classic.  It has to go through the whole vetting process which we have when we do the configuration.  And like this we try to avoid stupid configurations and keep your packages up to date.  I mean that's probably the most, that's why probably you should not use Gen2 but use,  I'm going to make a lot of enemies here.  Use something that you pay for is basically what I've had the experience with.  I've had very good experience with Red Hat and stuff.  I know Ubuntu has very good packages and Devian too.  They're very fast in fixing packages.  I wouldn't know why you would have production servers that don't update their packages regularly.  Especially SSH, Apache, hours count here, right?  There's a zero day exploit.  You're talking hours to minutes you want to fix this because you're going to be one  of the targets.  And if you haven't fixed it till then, you're a target.  So if you rely on a small Linux distro that's compiled by two people that are sleeping at  that time, you're compromised.  So if you're really running production-grade stuff, dish out the money, get professional  support, pay somebody for it, it's worth it.  So we had this case, we were running Gen2, that's why.  And they hacked us because of a zero day exploit.  And since then we've been running Red Hat and CentOS and it's been amazing.  Someone can read source files, which isn't that unlikely, which happens quite a lot,  that people gain access and can read your source files.  Probably your source code is probably not as big as a problem if you're open source  and stuff.  A lot of companies rely on their source code as being their business value.  That's quite horrible if that leaks.  But basically I've seen in settings files, I've seen passwords.  I've seen passwords being one, two, three, four.  Secret keys being one, two, three, four.  It's not hard to get it into the OS environment and not make it visible right away.  What I saw last week is Git and SSH keys.  I saw the private Git or SSH key in the repo because he was too lazy to have it in his  own home folder.  So he could do Git pull.  He had his private key in there.  Yeah, don't do that.  So, someone can write into my source files.  That's of course really, really bad.  But it can be sort of solved by using Tripwire, which is a really simple, easy tool.  Works really nicely.  We've never had a problem with it.  Some stuff can be done with SC Linux and so forth.  Git div is astonishingly easy.  Just do a Git div on your directory and see what has changed.  And probably you know the files that have changed anyway.  But if someone can write to it, they normally won't check the Git repository.  And again, check your permissions.  Again, I've seen break-ins happen because there were bad permissions set.  Someone can execute my code.  This is probably another really bad case, which is a really easy case is once a user  uploads a file and you've written it, just change the permission.  So you can't modify it and you can't execute it.  Just set it so you only read only every file that is uploaded by the user.  You get rid of a lot of problems like that.  And it's one line of Python code.  So then you have to think about where am I saving my data.  And so somebody is writing your code and that's really, really bad.  But what else can you do?  In theory, he can gain access to the database, right?  So what we do is we disallowed a leak on the database.  And we go even so far, we say that you're not allowed to update the database.  So if somebody can go in, he can still read all the stuff in theory, but he can't modify  our data.  So he can't inject false data.  Have buckups?  Really, I'll come to that a little later, which I cannot stress enough.  And obviously have source files read only.  Someone can modify or read your backup.  This is like, so actually what happened in this story is they had this huge server and  they backed up on these tape drives.  And that night somebody kicked in the door, took the server, and of course he took the  tape drives.  I mean, they're there.  So they had all this encryption and everything on their machine, but the tape drivers weren't.  So that company actually went bankrupt because of that.  They actually had, they lost all their business because they had to notify all their customers  that they were compromised and all the customers left.  So yeah, as dark as that may sound, think about your backup.  Use an extra process and a different user.  Use write-only storage, which is really simple, just like a dropbox.  You push it in there.  If somebody compromises your machine, he can't compromise your backup.  Obviously use encrypted storage on your backup and transmit it encrypted.  When I was a student, we would just put a hub in the network and would sniff all the  network of the whole network.  And in my student house, 10 geeks, amazing stuff happened.  So seeing what everybody was doing, it's so easy, right?  I mean, just get a little hub and plug it in there.  And instantly you see everything.  Check the certs.  So the typical thing is somebody brings up a machine in your network, starts a DNS server.  You're using some DNS server that takes longer to respond and suddenly he's a backup machine,  right?  So because he answers faster than the DNS?  Yeah.  Yes, exactly.  That basically goes with use encrypted storage and transmission.  So yes, yes, that's very true.  Very true.  I forgot that.  Yeah, that's a very good idea.  So basically encrypt your backup and sign it.  Check the cert.  So have a certificate on your backup server and check that it's really the backup server.  Don't just push it to any IP address that you hope is in the machine because you can't  rely on DNS.  DNS is really unsecure.  Just start a DNS server in any network and you can take over pretty much anything.  Never throw away backups.  You never know when you were hacked.  You never know when you might need it.  The cost of keeping a backup is near to nothing nowadays.  So why would you throw away something like this?  The next threat, somebody can read your traffic and this is incredibly easy.  Use HTTPS.  It's known technology.  Get a signed cert.  I know it's snake all.  I know it's useless actually.  But people will expect it.  And people don't trust your service if you don't have it.  And I mean, it doesn't really help if you can't, I mean, enough certificate authorities have  been hacked and new certificates have been issued.  But I think it's worth it.  But just use HTTPS.  Sniffing the network here is quite astonishing how many people get their email unencrypted.  Just doing pop.  I'm just saying that's all unencrypted.  Just encrypt.  You don't know where your traffic is going over.  You just, you don't.  I mean, enough people here have got their laptops open, their browsing websites and anybody  can see it, right?  It's an open network.  Or your email that you get over pop if it's not encrypted, everybody can read.  So use these encryption technologies.  They're standard.  They work.  The NSA can crack it, but the NSA will come in and shoot your kneecaps away, right?  So we're not talking NSA-grade security here.  We're talking the average guy with a laptop sitting in a conference room doing a network  sniffing.  Who's logged on to you?  I can see it all here.  In Django, there are two very easy configuration setting files or setting parameters to ensure  this.  Obviously, you don't want your cookies to leak over an unencrypted network, especially  not your session cookie.  So somebody can read your database.  So there's a classical thing when somebody hacks the machine, he does a select star on  the user database and saves that, right?  And then he adds a new user that has admin rights.  So he has admin on the whole system.  Those are sort of the normal steps you'll see in any compromising situation.  First of all, get all the data you can, all the password hashes.  I deliberately didn't go into hash your passwords.  If you don't know that, you shouldn't be doing programming.  Sorry.  If you don't hash your passwords, if you keep your passwords in plain text, leave, get a  new job.  I'm sorry, you've lost it.  I have to be honest with you, if somebody saves clean text passwords, just, I don't know,  build, no, just go.  I have no advice for you.  So what we do now is we do these really interesting Post-Escal stuff with Table Update and stuff.  It's not very scalable and not very nice.  So we do these other things that we override the delete method and stuff, so if somebody  can execute stuff, we say you're not allowed to delete.  And we override the base item manager, the models manager, to do like filter, we do some  checks in there.  But obviously, in a compromising situation, that doesn't really help because somebody  can just uncomment those functions.  I'm sort of relying if somebody can edit the code, you can't restart the server, so  we've sort of got some security there.  Yes, yes.  So we did in the model now, which is probably the worst idea thinking about it.  But anyway, we just don't allow in the database.  So this is just sort of backup for security reasons.  So I've been working on something which is not finished yet, which I hope I'll finish  at some stage.  And I'd love your input on this.  This is one of the reasons.  Yes.  Post-Escal, that's what we do.  That's what we do.  So you have this, basically you have this process, well, permission.  So this function is just, we add it to make the life harder and for bad programmers to  get a bad response.  Because actually somebody was coding and he's like, it doesn't work, it doesn't work.  So we added this.  Yeah.  But exactly.  So the problem you have there, if somebody's managed to get into your Python executable,  he can override all that.  Because what he can do, he can just put the super function here instead of that, because  he can just take that method and overwrite it.  So if somebody, so the threat scenario is somebody comes in, he manages to open the Python  shell in your browser or in your web server somehow.  And obviously all your code is in there.  And then he can obviously overwrite everything.  So the security in the app doesn't really help.  Because the only real threat scenario is, A, you've got a bad programmer.  Yes, this helps.  But you sort of assume your code should be good.  Or somebody has come into your web server and he's got access to your executable files  and then he can just overwrite this, delete it, whatnot.  So you really can have some people or some code to delete.  Yes, yeah, of course.  Exactly.  Exactly.  So this would be the right thing to write that into, into the delete.  All you, all, what we do is we also overwrite the filter and the all and the get.  Exactly.  And then you'll sort of secure that everything that builds on top won't, won't destroy anything.  But anyway, I didn't find anything I thought was secure.  The process well permission stuff is really hard and really easy to get wrong.  I didn't know anybody.  I couldn't find anybody that really knew it.  That was really weird.  I hired people and we couldn't because nobody really knew.  And we can't fix it in the code.  So what can you do?  You need to lay in between.  And I'd love you to put on this because I've been coding this basically.  And I don't know if it really, I hope it makes sense.  But maybe you've got some ideas.  So I'd love your feedback after in a few minutes.  So basically what I've done is I've written a server that identifies himself as process  well, that he verifies the login credentials with a real server.  And then he takes some queries and he parses these queries and just so the queries Django  creates for the OIM are really, really simple.  I mean, most of them, you know.  So what I do is I, I learn queries from my test cases because I've got tests that run  and I know which test cases they run and I also have like some user behavior scripts  that I extract from live data.  So I see how does the user react?  What does the user submit?  And I use this to train my security model.  And then for now, I have to tag variables manually.  But hopefully in the future, the script will learn automatically what's available and what's  not.  And so I write a regex which says this is how this looks like.  And then as soon as I, and then I train it, I've got this data set, I train it, then I've  got a big regex tree at the end of the day basically.  And as soon as something doesn't match this regex tree, I say, okay, I don't know this  query, what am I going to do?  And so when we deploy this, we'll get a few notifications, we'll have it running on a  shell and the query will just stuck stuck there and I'll just do, yes, okay, this looks  good.  Well, this doesn't look good.  And then it will learn this, hopefully, add this query to the training set and so forth.  And so hopefully at some stage we'll have a complete regex with all the queries we will  expect from the server.  So then I want to, or what I do now is actually I assign a trust variable between zero and  one to each query.  This is really crude.  But ideally you would say, okay, this is my logging database, queries that have a trust  of 0.2, they can go on my logging because I don't really care about that.  But on my user database, I only want queries that have a trust variable of one, which basically  100%, this is really trustworthy, specifically allowed in the regex.  And it can also have handcrafted rules.  So basically this is like the standard rule that's disallowed, select start from users.  And I'm like, no, this is not happening.  And I forgot the semicolon here, by the way.  But select start from users where username equal x.  That sounds like something a normal app would run.  So I say, yeah, this is okay.  What I want to add in the future is that I can then specify this is only allowed to return  one record.  Because I could say, look, if it returns two records, something's wrong here.  Or you're not allowed, users is never, actually users is never allowed to return more than  one record if you just do authentication on your app, right?  And everything that returns more than one record, I'm like, no.  But currently, I just pipe the result through.  Don't tell me this already exists, please.  If you know of something, send me an email tomorrow morning or when I'm asleep or tonight  when I'm drunk.  Yeah, so I'll get your feedback on that, what you think.  I'd love to pick your brains on this.  Then another threat, someone sees something you shouldn't, and Django is amazing in this  respect.  I really like Django.  You just do decorators at logonrequired, and we actually wrote our own decorator, check  permission that the user has to be owner, and we do edit authors, and that's it.  And this does all the checking for you.  And this is so easy, so nice.  From the experience, throw errors.  If prop don't get something, and then if it's empty, just propagate it through and throw  a 500 somebody at the end.  Throw the errors early as possible.  Weird things happen with null values.  The next threat, I have to hurry up.  Someone can read your database.  So let's say somebody hacks your machine, and as I said, the first thing he does is he  does select star on users, right?  So what you want to do is you want your database to be encrypted, right?  You want the user to add some sort of key, the database encrypted.  So there's this really nice encryption library, which I don't know how to pronounce, which  is by Google.  And what we did is we overwrote the dev to Python and get prep value, so basically all  the communication Django has with the database, and as long as it's a text field and a char  field.  You don't encrypt integers, is encrypted by the key of the project.  And like this, if somebody manages to do a select star on our databases, he'll at least  just see garbage for the text fields or text data.  In the future, we want to roll this out on images too.  The problem with images is CPU power, that we don't have the CPU power to decrypt an  image on the fly and serve it.  We had it and we needed these huge servers we couldn't afford and it was just really,  really hard to do properly and fast.  But for the database, it works fairly, because the text is fairly small, fairly condensed,  it's fairly fast.  We've got some performance overhead, obviously, and you lose some queries, you shouldn't  be doing anyway like queries and stuff like that.  But normally these result in full table scans in most cases, so you shouldn't be doing them  anyway, you should have an index.  Of course you lose indexing.  I started writing an encrypted index, but I haven't finished yet.  So CPU power is needed for this, but I think it's worth it because the actual overhead,  we have, CPU is not what's limiting us on the machines.  And Django Extensions has something like this, and a lot of influence came from Django Extensions  that we copied a lot.  Yes, so this is on a field level.  So what we write is models.encrypted text field, models.encrypted char field, and it  takes exactly the same parameters, it inherits from those fields.  But the only thing we do is when we serialize to the database, we encrypt it and when we  basically get the result from the database, we decrypt it.  So obviously you've still got your clean text in memory.  I sort of looked at memory encryption and couldn't find anything good, and that was  performant.  OpenBSD has done a lot in that respect, but I sort of assume if somebody's hacked our  memory then, yeah, we're f'd anyway.  Then I think I'm hearing more and more now is the users don't trust the service anymore,  especially with this whole NSA thing coming up, with this whole, yeah, the service is  out to get us.  And a lot of services I see are like, oh, yeah, we encrypt everything client-side, and  we only upload encrypted stuff to the server.  And then sort of, we had a very interesting discussion at the Berlin Django stumptish  about this.  And then so they trust the server and giving them the right JavaScript, but they don't  trust the server in handing their data.  So surely if the server is bad, we'll just inject some JavaScript, or we'll just inject  a bad key, and then we could still decrypt it, right?  So this is a discussion I find sort of followed up by quite the people that haven't followed  it through yet, if that makes sense.  If the user doesn't trust you, then you shouldn't trust the JavaScript, he's getting from you  either, right, because it sort of defines the whole logic.  So I agree, encryption client-side and JavaScript is really cool.  If we would get this to everywhere, it would be amazing, especially the load on the servers  would go down.  This is why would the user not trust you, but trust your JavaScript.  So a way we're investigating right now is that we will have a plug-in that comes from  Mozilla or whatever, and that verifies the JavaScript that comes from us, and all the  JavaScripts are open source.  So even if we would inject some bad JavaScript, the plug-in would alert, and this adds to  security if someone would hack our servers and inject bad JavaScript, also the plug-in  would come up and say, look, this is not true.  And like this, we sort of hope to have a transparent way of doing it.  I don't know if we'll be doing this, yeah.  Yes.  So basically, encryption in JavaScript is really, really slow.  So the first problem I ran into is true random numbers.  I didn't find any good implementation in JavaScript of a random number generator.  The thing I found was, yeah, yeah, it's time, which isn't random.  So the first thing, the API will hopefully, they're talking about a true random number  generator.  And then the problem with JavaScript is that all the modern chips, they have these really  good crypto chips.  That's why hard disk encryption doesn't cost us anything.  That's why all these encryption stuff is really, really fast because they've got a special  chip on their chips.  And what JavaScript does, if you have all these encryption libraries, it runs on the  main CPU.  And that's really slow because the CPU is not made for encryption.  So I don't...  You know, you're publishing your own source of the set.  Exactly.  Yes.  So the problem still stays, yes, you can encrypt it, even through the new JavaScript API stuff,  you can encrypt it really nicely on the client, but you're still pushing it to a service you  don't trust.  So there's a lot of...I think there's a lot of work to be done there.  Maybe some offline JavaScript features, yeah.  I was saying one possibility was to produce something that I see in the criticism of the  DNS and...  You know, basically what you're doing is you're operating...  Yes.  Yes.  In the DNS, for example, the incident that happened, I was watching the whole process  with DNS.  I read about that too, but I don't know any browser that supports it.  Yeah.  Yeah, exactly.  So we sort of...  Yeah, but I read about that and that's a very good idea actually to do...  One problem I see with that approach is that updates are going to be quite slow because,  I mean, we roll out two, three times a day.  So we'd have these very small and we're very fast.  Knowing DNS is kind of your...  Yeah, exactly.  You need that.  But...  So we're just experimenting with this plugin.  This is just sort of the direction we're going.  Yes?  No, you are...  Not yet.  Not yet.  We're playing around with it.  Yeah.  Now I am...  The plugin is okay, but I still need to...  So ideally the plugin...  Yeah.  So basically what the plugin does it verifies the header.  So what it does is it takes a whole head section and does a hash on it.  But of course you can put scripts somewhere in the text, but we haven't done that yet.  But yeah, so the idea is that if you have the JavaScript open source and everybody can  sort of generate the hash in some way, he has a plugin where the hash is displayed, he  can sort of trust it.  There's no...  I haven't found a really nice solution actually.  Otherwise Django is amazing.  It's all there, cross out scripting, use it, enable it, use it.  It just does such a good job at it.  Never use the safe decorator basically.  You'll get in a whole world of pain.  It's just so nice.  I mean, coming from program web servers and C and just seeing our SQL injection is dealt  with.  And you're like, this is amazing.  An email header injection, all this stuff is just done.  Then again, a personal story.  I've seen threat detection working perfectly, but nobody responding to the email.  And actually this happened a month ago.  Oh yeah, it was his job.  It was his job and it was a circle.  And so nobody did anything.  And the server was hacked and it was standing there for days until somebody sort of looked  at the server so he's still hacked.  And he's like, yeah, it was him and him and him.  So be sure that you notice when you're hacked at least.  There are some papers, some research done on this by University of Oxford that say it's  more important to notice when your server's been hacked and just destroy it than actually  preventing hacking.  I don't totally agree with that, but I sort of like the approach that you sort of monitor,  really hardcore your server.  And if you notice anything weird with that server, you just kill it.  You just freeze it for analysis later, but it's not in the production server anymore.  So I can't stress this enough.  Have a central log.  I've seen, let me finish and then have a central log.  I know people that have 30, 40 servers, huge servers that don't have a central log.  Emails are not enough.  I get a few hundred emails every day.  If that goes into spam, I will never read it.  Have error follow-ups.  So what we do now is all our error messages go to basically a bug tracking or support  tracking system.  And so if some monitoring service detects something, it creates a tracking cookie, our tracking  task.  And one of us has to follow it up because otherwise it's open.  So one of us has to go there and say, okay, I've done this.  I've solved it.  And we have notification propagation.  So if somebody hasn't acted on a task in 24 hours, we email his boss, which is incredibly  effective.  You had a question.  Yeah.  Yes.  So the idea of the security is that we have to do a security check.  Yes, so the idea is that you pull up new servers, obviously while you're doing it.  But I read the paper two, three years ago, and I was like, I saw a lot of flaws in it  too.  And you've got like, you've got like, what happens if the notification is wrong, right?  You just shoot off a user every time he tries to upload a picture of his family.  And somehow it's a different color encoding.  And you're like, no, I don't like this color encoding.  And the session times out, which is not user experience you want, right?  And I find it very hard.  So what they did, they froze the virtual machine and they then analyzed it later.  But I quite like that we get an email or ticket.  And then we just go in and check right away what's happening.  And we can sort of see if it's a real hack, which hasn't happened so far, or if it's something  we didn't expect and can act on it and stop more notifications coming in.  Yeah, so basically ask me stuff and check out Dbook.  Yes.  So we've got everything virtualized anyway, because we've got multi.  I personally think Docker makes a lot worse because I know how programmers do Sys stuff.  And I wouldn't trust, OK, there are some programmers that can do it without question.  But I've seen some of the fabric scripts I've seen in my time when programmers try to do  deployment.  And it's probably this whole DevOps thing, right?  It's always Sys admins against programmers.  And I quite like just deploying from a Git tree and doing the whole staging old school  approach.  Docker is very, very good if you have lots of developers with different machines that  deploy rapidly.  Definitely.  I wouldn't, I mean, how does it contain security threats in the respect?  It might contain it in the Docker, but it also does it in a virtual machine, right?  So yeah.  But we do that in virtualization.  Yeah, it's process isolation, which we do with virtualization.  But we do it actually with physical machines nowadays.  But yeah, I agree.  Process isolation, I'm very astonished.  I haven't come to the desktop yet, because a few years ago I had my Firefox in a virtual  machine because I was so afraid of security holes in Firefox.  But I don't know why, I mean, process isolation has been around for 30, 40 years.  And it hasn't really happened yet.  But I agree.  Yeah, I haven't played around with Docker too much to do it.  But yeah, we do it with virtualization.  That's basically what I can say, which is just a bigger layer, right?  Yeah?  So we do it on the local machine.  Basically we have a cron job that has a nifty Python script.  And then we have Hadoop jobs that take all the log files in and do monitoring on that.  But we do that two times a day.  We have a big Hadoop job that kicks in with all the log files, because we have some historical  analysis going on there.  And for example, what we also do is we have user behaviors.  So imagine a timeline, and you have clicks or actions on this timeline.  And so what we do is user behavior changes over time.  So a user will become more proficient as the longer he uses the software.  But what you can do is you can sort of see how advanced the user is.  And then if he does something really weird at some stage, you can sort of see, okay,  this is sort of what he should be doing at the beginning.  And so it says girlfriend using it mostly.  That's the average case where we've got quite nervous and there was just this girlfriend  kid or something using his account.  But so we do that in Hadoop.  We do classical map reduce.  We're looking at stratosphere to do that was their stream processing, which isn't ready  yet.  I haven't found a really good real-time stream processor yet in big data.  And we do really, really slim real-time logging on the machine.  Basically, just log file analysis when we stream it off to the master.  Yeah, that's an interesting question that I hope nobody would ask that.  So the future is that we store the keys on the client and he gives us basically something  to decrypt it.  What we do now is we have a really locked-down database and that's all I'm going to say.  So we still have all the keys in one central repository just because we haven't had the  time yet to have a proper key distribution.  No, so select or users doesn't work now anyway.  But no, it's a total different system.  So our key storage is something I don't want to talk too much in public, but I know it's  not 100% secure because ideally what you would want to do is have the password generate some  sort of key that does a decryption.  But I haven't found a really nice way that if somebody loses his password, all we have  the sharing feature that many people can access and edit a document.  And it becomes incredibly complex if you want to do this key encryption like that.  And maybe I'm not good enough at math or something, but I haven't figured this one out yet.  So now for now, because time is limited, we just have a really secure key storage which  is handwritten, which hopefully is secure than any data.  It's a modified Kerberos LDAP thing, basically.  So we have 16 shards, so roughly 20 machines, probably a little bit more, and we use SCP.  And then we do analysis on it.  Clearly, I haven't had the time yet.  I know there are lots of solutions out there.  I know there's a lot out there I just haven't had the time yet to do.  So for now what we do is we do SCP, and then we just have Mooneen that reads those files,  which works quite well because we just have an SCP tunnel that is open, and we stream  to the file.  So you get all the advantages of SSH, it being encrypted, it being fairly fast.  We have the key authentication and all that stuff.  And we just pipe.  We just tree, basically.  And SSH has been incredibly good.  And it works.  It was a hack, and it works.  And we detect when the SSH tunnel comes down, so we know when a machine is sort of having  a problem.  He was first, sorry.  No, I guess we're talking about finding lots of life.  So we just need to have a compromise.  We can do a post-mortem analysis.  Exactly.  Exactly.  And it's the same thing you said, that there's compromise in just a minute of production,  but then you should go post-mortem analysis on the post-mortem analysis.  Exactly.  And like you have a problem memory, over the binaries and the post-mortem, everything  you can get, it was dense, out, and sort of, literally where it is, where it is, and  there it is.  So basically just freeze it, right?  We have virtualization, just freeze the machine and put it somewhere where it's safe and  then analyze it, not being live on the network.  Yeah, yeah.  The problem with the entity is, oh, he can't detect or edit with my data.  Yes.  Probably he can't edit it.  No, he can't edit it.  Why don't you do so?  Probably he could modify the file pointer and, yeah, probably he could modify it.  Now you all know, right?  And the internet knows.  We have to change this tonight.  Yeah, of course, it's not perfect.  I agree.  Obviously, I just hadn't had the time yet to optimize it.  I'm a very small team, and I'm luckily exponential growth, which is horrible right now.  But, yeah.  It will go to the honeypot.  It's fine.  Yeah.  We've already known about production.  Yes.  I mean, obviously, you need one user to get in.  We knew that, so that's good.  And originally, we knew that we were running the configuration.  Okay, so no user that, so what we have, we have a normal login user.  Root is not allowed in by SSH.  So we have the standard, I don't actually know the password, because on user creation,  I just set a massively random password.  But that user has sudo, and then can get root, and that's how we run the configuration management.  But, yeah, you always need one user to get in.  I wouldn't know a way.  I mean, what I've been thinking about is using LDAP to basically have one central user that  we sort of manage, but I don't see the big advantage yet.  Yeah.  Yeah.  Yes.  Yes.  I mean, we have accounts.  We have WW data.  We have process squirrel.  But obviously, those users don't have a shell.  They don't have SSH access.  They don't have, so we still have that one user that we can get into.  But only a few people have the certificate to go into.  And I hope that people don't do stupid things.  Yeah.  After you finished the conference, you know, it's like the other, our thing is the password.  And you can say, hey, I've got my TPC S35 here.  Based on all of our servers, and be sure that this one user is available on our fan site.  It's a cylinder.  How do they call that?  They've got the secure storage thing, haven't they?  A silo or something?  Salt has this really cool secure storage container where you can put your ETC password in it  and we'll distribute it in a secure way.  But I...  So if you're in a void, there's someone going to add something like that because it files  a whole bunch of servers and takes my orders.  So actually you can configure a salt that will actually phone out.  So you could get rid of your user if you really wanted to by getting salt to get the salt master  and get this configuration file from the salt master periodically.  But I wouldn't, I haven't dared to do that yet.  So again, physical access is fail, right?  But just a few days ago, exactly that happened, I locked myself out of the machine.  Not for this project, for another project.  And I just went to the virtual machine, to the console.  I rebooted the machine, put single at the end of Grubb and I was root.  It was that easy, right?  So that's probably something I should have added to.  The dashboard of your virtual machines is probably the most important thing you've got.  If that password isn't that long, because as soon as you can reboot the machine and you've got  the dashboard, it's compromised, right?  So yes, yes, that will come, definitely.  I mean, we've got that with email and all that already.  But I've got the app on my phone and I haven't gone that far to have the SSH stuff go that far.  So in the future, we'll probably have that.  So the PAM two-factor authentication rules have been around for years and it's well tested.  Years back, I mean, this was 2007, we used LDAP with two-factor authentication, Kerberos,  and it worked perfectly.  I mean, you just have to buy these quite expensive RSI tokens and that were hacked by the NSA, by the way.  We bought thousands of them and everybody had to carry it around and it was horrible to maintain.  And then the news came, NSA had hacked them and we were like, ah.  But yeah, so that worked really nice.  So getting two-factor authentication in Linux is editing five lines, I think, of code.  Just add the PAM rule in the PAM stack and it works.  The biggest problem is the synchronization of the seed in all servers.  There is a product here, a open-key, I don't know, that's basically a separate that's comprised of the seeds of the two-factor tokens,  the inside of the seed.  You have a range, if you want, of the range, the other server, the one-fringer, that's it.  Exactly, yeah.  We just did it with ranges.  We didn't have a synchronized seed.  If you want to fix that, you can use the antipyro-spoke code, you have much more space.  Exactly.  Yes, I agree.  Probably the future, I think, is with phones.  Everybody's going to have an app on their phone and it's going to phone in and phone out  because my phone's online all the time.  But I haven't seen anything useful there with Linux.  Maybe there is.  Questions?  Or comments?  Nobody commented on my genius idea of the self-learning SQL algorithm.  This is hours of my life.  That was the main point.  Cool, so thank you very much for listening.  I'll see you at the beer booth thing.
Web-App-Encryption__FrOSCon14__0_47.png Web-App-Encryption__FrOSCon14__48_107.png Web-App-Encryption__FrOSCon14__108_155.png Web-App-Encryption__FrOSCon14__156_203.png Web-App-Encryption__FrOSCon14__204_275.png Web-App-Encryption__FrOSCon14__276_323.png Web-App-Encryption__FrOSCon14__324_395.png Web-App-Encryption__FrOSCon14__396_719.png Web-App-Encryption__FrOSCon14__720_1139.png Web-App-Encryption__FrOSCon14__1140_1535.png Web-App-Encryption__FrOSCon14__1536_2195.png Web-App-Encryption__FrOSCon14__2196_2255.png Web-App-Encryption__FrOSCon14__2256_2303.png Web-App-Encryption__FrOSCon14__2304_2423.png Web-App-Encryption__FrOSCon14__2424_2843.png Web-App-Encryption__FrOSCon14__2844_2951.png Web-App-Encryption__FrOSCon14__2952_3071.png Web-App-Encryption__FrOSCon14__3072_3659.png Web-App-Encryption__FrOSCon14__3660_3791.png Web-App-Encryption__FrOSCon14__3792_3839.png Web-App-Encryption__FrOSCon14__3840_3899.png Web-App-Encryption__FrOSCon14__3900_4127.png Web-App-Encryption__FrOSCon14__4128_4175.png Web-App-Encryption__FrOSCon14__4176_4487.png Web-App-Encryption__FrOSCon14__4488_5063.png Web-App-Encryption__FrOSCon14__5064_5135.png Web-App-Encryption__FrOSCon14__5136_5495.png Web-App-Encryption__FrOSCon14__5496_6191.png Web-App-Encryption__FrOSCon14__6192_8015.png Web-App-Encryption__FrOSCon14__8016_8711.png Web-App-Encryption__FrOSCon14__8712_12383.png Web-App-Encryption__FrOSCon14__12384_13931.png Web-App-Encryption__FrOSCon14__13932_16511.png Web-App-Encryption__FrOSCon14__16512_19247.png Web-App-Encryption__FrOSCon14__19248_20099.png Web-App-Encryption__FrOSCon14__20100_23399.png Web-App-Encryption__FrOSCon14__23400_23915.png Web-App-Encryption__FrOSCon14__23916_24647.png Web-App-Encryption__FrOSCon14__24648_25631.png Web-App-Encryption__FrOSCon14__25632_27431.png Web-App-Encryption__FrOSCon14__27432_28235.png Web-App-Encryption__FrOSCon14__28236_28331.png Web-App-Encryption__FrOSCon14__28332_28391.png Web-App-Encryption__FrOSCon14__28392_29051.png Web-App-Encryption__FrOSCon14__29052_30047.png Web-App-Encryption__FrOSCon14__30048_30839.png Web-App-Encryption__FrOSCon14__30840_31715.png Web-App-Encryption__FrOSCon14__31716_32723.png Web-App-Encryption__FrOSCon14__32724_33335.png Web-App-Encryption__FrOSCon14__33336_34403.png Web-App-Encryption__FrOSCon14__34404_35231.png Web-App-Encryption__FrOSCon14__35232_36311.png Web-App-Encryption__FrOSCon14__36312_37043.png Web-App-Encryption__FrOSCon14__37044_37499.png Web-App-Encryption__FrOSCon14__37500_40355.png Web-App-Encryption__FrOSCon14__40356_41303.png Web-App-Encryption__FrOSCon14__41304_42779.png Web-App-Encryption__FrOSCon14__42780_44003.png Web-App-Encryption__FrOSCon14__44004_45827.png Web-App-Encryption__FrOSCon14__45828_46031.png Web-App-Encryption__FrOSCon14__46032_46403.png Web-App-Encryption__FrOSCon14__46404_49295.png Web-App-Encryption__FrOSCon14__49296_50615.png Web-App-Encryption__FrOSCon14__50616_54803.png Web-App-Encryption__FrOSCon14__54804_55679.png Web-App-Encryption__FrOSCon14__55680_55919.png Web-App-Encryption__FrOSCon14__55920_56123.png Web-App-Encryption__FrOSCon14__56124_56615.png Web-App-Encryption__FrOSCon14__56616_57227.png Web-App-Encryption__FrOSCon14__57228_60551.png Web-App-Encryption__FrOSCon14__60552_61427.png Web-App-Encryption__FrOSCon14__61428_68183.png Web-App-Encryption__FrOSCon14__68184_68915.png Web-App-Encryption__FrOSCon14__68916_68963.png Web-App-Encryption__FrOSCon14__68964_69611.png Web-App-Encryption__FrOSCon14__69612_72971.png Web-App-Encryption__FrOSCon14__72972_73631.png Web-App-Encryption__FrOSCon14__73632_73739.png Web-App-Encryption__FrOSCon14__73740_73871.png Web-App-Encryption__FrOSCon14__73872_74039.png Web-App-Encryption__FrOSCon14__74040_86531.png Web-App-Encryption__FrOSCon14__86532_95339.png Web-App-Encryption__FrOSCon14__95340_95435.png Web-App-Encryption__FrOSCon14__95436_95483.png Web-App-Encryption__FrOSCon14__95484_95531.png Web-App-Encryption__FrOSCon14__95532_95579.png Web-App-Encryption__FrOSCon14__95580_95639.png Web-App-Encryption__FrOSCon14__95640_95679.png