Twisted Names
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Wall, Richard
Richard Wall - Twisted Names: DNS Building Blocks for Python Programmers In this talk I will report on my efforts to update the DNS components of Twisted and discuss some of the things I've learned along the way. I'll demonstrate the EDNS0, DNSSEC and DANE client support which I have been working on and show how these new Twisted Names components can be glued together to build novel DNS servers and clients. Twisted is an event-driven networking engine written in Python and licensed under the open source MIT license. It is a platform for developing internet applications.
 My name is John Pinner. I am by team organized Picon UK and Euro Python 2009, 2010 we did. One of our team members is Richard, who is going to tell us all about DNS and Twisted. Thanks John. Thank you, thank you all for coming. Yes, so my name is Richard Wall. I'll start by telling you a little bit about me. I don't want to spend too long. I'm a Python programmer from the UK and as John says, I'm involved in the Picon UK conference as an occasional speaker and my most important job there is to organize the conference dinner table plan. I run that, I maintain the table plan software. I'm also an enthusiast, a real enthusiast for Twisted, which is a framework which hopefully you've all heard of. It's one of the oldest Python frameworks. And I have become over the last year a contributor and a core contributor, I suppose I could sort of call myself a core contributor and I guess the de facto maintainer of Twisted names which is the components of Twisted which I'm going to be talking to you today about. And I'm currently working in Bristol in the UK for a company called Cluster HQ where we're working on some software to manage the deployment and the state of Docker containers. It's a really interesting open source project that we're working on called Flocker. So you should look that up if you're interested in any of those technologies. I'm working on that with some of the Twisted founders which is super exciting for me. Okay. Well, I haven't got long this talk and I, when I did this in the UK last year, I over underestimated how long it was going to take and so I've cut out a lot of stuff. This is going to be a much shorter version if anyone saw the talk in the UK. I'm going to talk less about myself, less about the history of Twisted and more about the technology in Twisted names and also the project that I've been working on recently to implement EDNS and DNS support in Twisted. I'm going to start with an overview of DNS. Well, very short overview. I'll explain why in a minute. I'll give you a tour of the components in Twisted names. I'll hopefully give you some interesting examples, some quite interesting examples. I'll give you a status report on this project and then I hope to have some time at the end to answer any questions. I hope there will be some questions. So I had planned to give an overview of the domain name system but I don't think I'm going to have time and I don't think I need to anyway because you've probably already been to a talk on Wednesday by Lin Root who explained really clearly about the domain name system and its structure, its operation and the terminology and I think some of the software that you may be familiar with for serving and sending DNS requests. So she did a much better job than I probably can at explaining it and so what I'm going to say is you should just go and watch her talk on YouTube. It's a great talk. I didn't make it to the talk. I wish I had been able to but I watched it last night and I'm glad she did it because it means I can get to the interesting bits, the Twisted bits that I want to talk about. So I'm going to skip this, skip this, skip this, skip this, skip this, skip this and somewhere. I might talk briefly about the software that you may be familiar with just as a contrast to the Twisted name system which I'll explain in a minute. So probably you're all familiar with Bind which is the original DNS server. I think that's true to say it's the original DNS server. It's an authoritative DNS server and a recursive DNS server and a forwarding DNS server and all sorts of other gubbins that are mixed in with it and that's part of its problem. It tries to do too much. It's feature packed but it's over complicated and it's full of vulnerabilities. This idea of one binary to satisfy all the different DNS server requirements is a mistake which has been learned and implemented better in another piece of software called PowerDNS. So if you're using Bind then I recommend you go and look at PowerDNS which is a much more modern, much better designed, a much more secure DNS server. It's actually more powerful than Bind in a way because it has a much cleaner way of interfacing with a database back end for example and it also splits the duty of authoritative server from the duty of recursive DNS server which is important to avoid cache poisoning attacks. Other servers you may have come across are Unbound and NSD. I mentioned those because they are written by an organisation that I've been involved with this project that I'm going to tell you about, NLNet Labs. And again they are much more modern, much more secure DNS servers dedicated to, Unbound is dedicated to answering recursive requests, NSD is dedicated to answering authoritative requests. So let's now get to the subject of this talk, Twisted Names. So Twisted Names is kind of as old as Twisted itself. It's celebrating its 13th birthday this year. It probably started life as you may or may not be able to see from this check-in. This is the first change set where Twisted Names was first landed. It was probably introduced as a sort of demo of what was then new UDP transport facility in Twisted. I did a little bit of digging and found the commits from the beginning of Twisted Names life and some of the newer commits that I've been working on and Julian's worked on a bit, I see. So you can see it was originally written by a guy called Moshi Zadka back in 2001. And that was in the good old days when Twisted had a kind of wild west development process. Everyone was just committing randomly to trunk. And I guess they hadn't yet implemented what is now called the ultimate quality development system, which is a talk in its own right. It's a way of, it's a way, it's the way we develop in Twisted. It's the way of developing in branches and ensuring that every change that gets merged to trunk has been code reviewed, that it's fully tested, that the code is fully covered and that there's an audit trail showing between the ticket and the code that lands in trunk. But you should go and read about the ultimate quality development system if you haven't already. So Twisted Names was, it was kind of actively maintained to start with. And over the years, it kind of, I think it's true to say it's been neglected a bit. And then I started getting involved about two years ago and I had a background in DNS, so I thought that's part of Twisted where I could help out. And I've been busily updating the documentation, adding test coverage, adding some new examples to demonstrating how to use Twisted Names and you'll find all of those on the new Read the Docs documentation website. I'll link to that in a minute. So yeah, like the rest of Twisted, the Twisted Names package is really well tested. It's got comprehensive unit tests which are run using a tool called trial, which is a great test runner. The unit tests in Twisted Names, if you care to read the code, and probably not the most sophisticated unit tests in the world, but that's a reflection of the way these testing techniques improve over time. Twisted is, as I said earlier, it's over 13 years old. So if you do go and start hacking on it, you'll find that there are bits of it which are kind of hard to read, hard to look at without bursting into tears. But then again, you have to recognize this history of the project and it's actually quite interesting to see how particular developers who have been with Twisted from the very start have changed their ideas and their approaches to things like testing. I think that would be another interesting talk in its own right. It's interesting from the point of view of a new contributor who has to deal with this old style code and the new style code and understand what's the current best way of doing this, developing the Twisted. So we've got plenty of unit tests and we've got reasonable coverage of the code. Some of the modules, I don't know whether you can see it on the slide, are not very well covered, but those are areas which I'm working on now. Modes such as the authoritative DNS server and the secondary DNS server are not particularly well covered. And in fact, that's sort of really its ugly head lately in a bug that's been discovered in the secondary name server in the latest release of Twisted. It's partly down to a lack of test coverage. This bug wasn't noticed earlier and it's partly down to the old style design of that part of the package. Hopefully I and others are going to improve that over time. Twisted names wasn't particularly well documented, but that's improving. As I said earlier, I've been working quite hard on improving the documentation for Twisted names and Twisted as a whole is better documented these days. You can go like most of the projects these days and read the docs on Read the Docs. And it's nicely presented and nicely indexed and easily searchable. So I recommend if you're interested that you go and read the documentation for Twisted names and for the rest of Twisted because it's much easier these days to navigate the documentation. So how am I doing for time? I'm running out of time. I'm going over time. I'm running out of time. So let's crack on. We'll have a look now at the different modules in Twisted names. I'm going to start at the lowest level and work up. Like everywhere in Twisted, there are layers of abstraction, layers upon layers and on layers. And we'll start at the bottom and give some examples of how you can use these low level APIs and then later we'll see some of the higher level abstractions. So let's start with the Twisted Names DNS module. Now this contains protocol level APIs, representations of the DNS records, representations of DNS messages, routines for serializing and deserializing these messages from the wire. And it's also in this module that you'll find the protocol implementations both for UDP and TCP because DNS operates over both of those transports. So we've got a little example here which I'll try and talk you through. Can you see that? Let me try and zoom in a bit. This is Reveal.js and I'm not sure whether it's going to... Oh, yeah. Yeah. Can everyone see that? Great. Okay. So what we're looking at here, there's a couple of things I need to explain. We have got, first of all, let's start at the bottom and look at the last line which is task.react. Now if you've used Twisted before, you may not have come across this API, but this is a new way for you to start the reactor for a short-lived Twisted program. And what it does, it supplies a method, you supply it with a function that you want to run, a function which must return a deferred. And TaskReact will take that function and supply it with the reactor, run your function and then wait for the deferred that it returns to fire. And upon firing, TaskReact will then tear down the reactor, take care of stopping the services in the right order, and it will log any errors that haven't been handled on that deferred. So if we then move up to the main method, we can see that having supplied the reactor, we're instantiating a DNS datagram protocol. So in this example, we're only going to be, this DNS client example, we're only going to be dealing with UDP. And we instantiate the protocol and then we pass that to React to listen UDP on port zero, which means any high ephemeral port. And because this is UDP, we don't have any connections. So we have to, so it may look odd to be in a client calling a method called listen, but we are going to send a UDP datagram and we have to be listening for the response, whereas in TCP, the operating system would set up the connection for you and you wouldn't have to choose the ephemeral port that the response comes in on. So we listen on UDP port XYZ and then we send a query using the protocol, using, we call protocquery to send our DNS query to, in this case, the Google DNS servers. And then when that query has been answered, we're going to print the result and that's by way of adding a callback to the deferred returned by protocall.query. Again, I sometimes think we should start every talk with an introduction to deferreds, but I guess everyone's heard it and more people these days are familiar with the idea because it's now part of JavaScript, I think. So yeah, when the answer comes back, we simply, we take the result and we take it, the result is a message, a DNS message, which I'll explain in a little bit more detail later, but a message has three attributes, has an answer's attribute, an authority attribute and an additional attribute. And those represent the three categories of records that might be returned by a DNS server. And in this case, all we're doing is printing out the answers returned by the DNS server and in particular, we're returning, we're going to print out the payload, which is the, either the, in this case, the quad A record, it might be the A record or the MX record. We're not interested in printing out the header information which wraps around that payload. And so I'll show you the, show you the output. Oh, I'm running out of time rapidly. Okay, so we've got an answer from the server, a quad A, one single quad A record. Now let's quickly move on to the next example. So that was a client. The next example is a server. And in this case, it's quite similar, but we instantiate the datagram protocol this time with a controller which takes care of handling the, the, the, the, the query which comes into our server. And when a query is received by the protocol on port 10, 10,053, our protocol then calls out to the controller and calls its message received method. And it's the message received method on the controller, which is responsible for, for constructing an answer to that message. So this is how we write low level servers in low level DNS servers in Twisted. And in this case, we're just going to respond with a count A record with a, with a fixed IP address. So hopefully that makes sense. I haven't got time. I'd like to go into it in more detail, but I haven't got time. There's the, there's the server running and there's us issuing a request to it using dig. Okay. So now, those are low level APIs. If we move up now to twistednames.client, this is a much higher level API, a much more friendly way of interacting with Twisted names. And in this example, we're going to, you, we're going to introduce a couple of new concepts. We're going to use Twisted names to look up concurrently the, the reverse DNS records for a whole class C network. And so you can see in our main method that we are constructing a list of all the IP addresses in a, in a, in a slash 24 network using a really useful module called net adder, which does all the, the construction of those reverse DNS names for us. I think there was a, I think there was a lightning talk on it yesterday. So I recommend that module. And for each of those reverse DNS names, we're going to call client.lookup pointer. And client has a, has a series of these lookup methods, one for each type of DNS record that you can receive from a DNS server. It doesn't have all of them that we're working on implementing some of the missing ones, but it has a lookup method for almost every common DNS type. And so we construct a series of, a list of deferreds, all of them in flight. And all of them are then added to what's called a deferred list. Now a deferred list is a, a really useful API for collecting the responses to a list of deferreds. And then it fires its callback when all of the deferreds have themselves fired or, or failed. So when we handle the results in this example, we are looping through the results, checking whether the result was a success or a failure. And if it was a, if it was a success, we print the, we again print the payload. And we also print a summary, summarizing how many of the requests were answered successfully and how many of them were not answered, either because the record didn't exist or perhaps the, the, the query timed out. So the, the results to that are as follows. So you can see it all happens rather quickly. And because everything is happening, happening concurrently. So that's a real advantage of using Twisted for this sort of work. I might skip now to a better example of that. One which follows on from Lynn's talk on Wednesday. So let me, let me quickly summarize some of the other modules. We have, in Twisted, we have the modules for, for creating DNS servers. I had an example of that. And it's really easy to use because there's a Twisted DNS plugin for the Twisted command which comes with Twisted. And so you should explore that and explore all the options that you have using that command. Twisted itself runs its, TwistedMatrix.com, that domain is actually served from, from a Twisted DNS server. So it's, it's, it's pretty, it's pretty stable. And it's, it's not, it's not a, it's not a fully featured DNS server, but it's good enough for some cases. You can see that when you start the server, it logs to standard out and that we can query that server once it started up. We also have an authoritative server and it's interesting, but I haven't got time to go into it, that you can load a DNS zone based by defining it as a Python module. And so here we have a Python module with describing the zone, but the, the, the, and the, the objects that you see there are all globals which are imported at the time that this module is evaluated by Twisted names. It's quite a clever mechanism, but you should look into that too. It's an interesting piece of code. And again, there's the example of how it runs. And again, these examples are all on the documentation, the Twisted documentation site. So I recommend you go and read those. There's a bunch of other modules which I have to skip through. Common contains some helpers and some, some APIs common to all of the Twisted clients and servers. Resolve, I won't go into cache, is about caching the responses to, to queries. Root is about doing recursive DNS resolution, which Lynn talked about in her talk. C is about transferring zones and serving them authoritatively from another authoritative server, which some of you may be familiar with. And the point I wanted to make by describing all of these is that these, all of these building blocks can be put together in interesting ways. And I've done a couple of examples of this on the website. For example, you could create very easily using the low-level APIs, a module or a script for test, for compliance testing of DNS servers or clients, because you have complete control over the, the, the flags and the, and the payloads that you put into messages. So it's easy to construct non-compliant messages to see how DNS servers respond to those. Or it's easy to, to, to, it's easy to see how clients respond to non-compliant responses from servers. So that's a, that's a good use of these building blocks. We use the, we use Twisted at Work for, Twisted Names at Work for functional testing. So we have a bunch of code which does DNS lookups. And we want to, in our tests, supply canned responses to those DNS lookups. And it's very easy using Twisted to set up a, set up a lightweight DNS server and then tear it down at the end of the test. It would be really easy to set up a database-backed DNS server or a DNS server which looked up its data from a REST API, for example. Or it'd be easy you to combine these with other parts, other components in Twisted, like the Web module or the LDAP module, to look up DNS records from an LDAP database or to, to control and manage the DNS records in your server using a REST API. So now let me see if I can quickly show you, finish off with a quick example, a more complicated example of using Twisted Names. So in Lin's talk, it was really interesting, there's a tool called DNS Map which is a tool for, for brute forcing a zone, for guessing which names may be in a zone, a DNS zone. And it does that using a dictionary of words or common sub-domains. And as, as you said in your talk, it does it in series. It's quite, quite a slow, it's quite slow to complete because there's about a thousand words in its dictionary. So I thought it'd be interesting to write a tool, write the same tool in Twisted because it can do all of these lookups concurrently. So this is, this is how DNS, the DNS, the original DNS map is documented. You pass it a domain name and you pass it, you can pass it a list of words which it will then look up each of those words as a sub-domain of the supplied parent domain. But as you can see, it is quite slow. So I started this going against Spotify.com and 48 seconds later it had only reached G. So it was going to take forever. Now I want to compare that to another example, the example I wrote which, I don't know whether I've actually given you the link, but I've put all this code on my GitHub page. I've got the link at the end of my talk. And in this example, we are, we're actually sending all of our requests concurrently, but in, not, not, we're not sending a thousand requests at a time. We're using some, another interesting part of Twisted called the co-operator, the co-operator API in the task module. And what we can do using that is to limit the concurrency so that we can say there's only ever a hundred in-flight DNS requests at a time. So we're not going to overwhelm the DNS server that we're querying. I haven't implemented in this the random time-outs that DNS map actually puts in between its requests. So it's not quite the same. But if I show you the results, in this case, it's looked up all, it's about a thousand sub-domains in two and a half seconds. So that's a good demonstration of the power of Twisted and the power of the APIs and the way that it can efficiently process, efficiently send out requests and process the responses. All this code is on my GitHub page. Now I think I've run out of time, so I wanted to talk about my project. I'd love to talk to you about it. I'm going to be sprinting at the end, tomorrow at least, on Twisted. So if any of you are interested in helping out or learning about the development process, if any of you are DNS experts and want to help me with my project, then I'd love to hear from you. It's all about EDNS, DNSSEC, I've got funding, so you might get paid for it. I've made modest progress and I think that's the summary of this talk. It's the summary of what it's how I wanted this talk to be, but I haven't had time to cover it all. So those are the links, those are the links to the documentation, that's the GitHub link to the examples in my talk if you want to investigate those. If you, I'll put these on GitHub as well, and I'll link to these from Twitter or somewhere. I'll make them available on the conference website. Have I got any time for questions? No. If there's any questions, catch up with me afterwards and I'd be delighted to talk to you about it. Thank you. Thank you.
 My name is John Pinner.  I am by team organized Picon UK and Euro Python 2009, 2010 we did.  One of our team members is Richard, who is going to tell us all about DNS and Twisted.  Thanks John.  Thank you, thank you all for coming.  Yes, so my name is Richard Wall.  I'll start by telling you a little bit about me.  I don't want to spend too long.  I'm a Python programmer from the UK and as John says, I'm involved in the Picon UK conference  as an occasional speaker and my most important job there is to organize the conference dinner  table plan.  I run that, I maintain the table plan software.  I'm also an enthusiast, a real enthusiast for Twisted, which is a framework which hopefully  you've all heard of.  It's one of the oldest Python frameworks.  And I have become over the last year a contributor and a core contributor, I suppose I could  sort of call myself a core contributor and I guess the de facto maintainer of Twisted  names which is the components of Twisted which I'm going to be talking to you today  about.  And I'm currently working in Bristol in the UK for a company called Cluster HQ where we're  working on some software to manage the deployment and the state of Docker containers.  It's a really interesting open source project that we're working on called Flocker.  So you should look that up if you're interested in any of those technologies.  I'm working on that with some of the Twisted founders which is super exciting for me.  Okay.  Well, I haven't got long this talk and I, when I did this in the UK last year, I over  underestimated how long it was going to take and so I've cut out a lot of stuff.  This is going to be a much shorter version if anyone saw the talk in the UK.  I'm going to talk less about myself, less about the history of Twisted and more about  the technology in Twisted names and also the project that I've been working on recently  to implement EDNS and DNS support in Twisted.  I'm going to start with an overview of DNS.  Well, very short overview.  I'll explain why in a minute.  I'll give you a tour of the components in Twisted names.  I'll hopefully give you some interesting examples, some quite interesting examples.  I'll give you a status report on this project and then I hope to have some time at the end  to answer any questions.  I hope there will be some questions.  So I had planned to give an overview of the domain name system but I don't think I'm going  to have time and I don't think I need to anyway because you've probably already been to a  talk on Wednesday by Lin Root who explained really clearly about the domain name system  and its structure, its operation and the terminology and I think some of the software that you  may be familiar with for serving and sending DNS requests.  So she did a much better job than I probably can at explaining it and so what I'm going  to say is you should just go and watch her talk on YouTube.  It's a great talk.  I didn't make it to the talk.  I wish I had been able to but I watched it last night and I'm glad she did it because  it means I can get to the interesting bits, the Twisted bits that I want to talk about.  So I'm going to skip this, skip this, skip this, skip this, skip this, skip this and  somewhere.  I might talk briefly about the software that you may be familiar with just as a contrast  to the Twisted name system which I'll explain in a minute.  So probably you're all familiar with Bind which is the original DNS server.  I think that's true to say it's the original DNS server.  It's an authoritative DNS server and a recursive DNS server and a forwarding DNS server and  all sorts of other gubbins that are mixed in with it and that's part of its problem.  It tries to do too much.  It's feature packed but it's over complicated and it's full of vulnerabilities.  This idea of one binary to satisfy all the different DNS server requirements is a mistake  which has been learned and implemented better in another piece of software called PowerDNS.  So if you're using Bind then I recommend you go and look at PowerDNS which is a much more  modern, much better designed, a much more secure DNS server.  It's actually more powerful than Bind in a way because it has a much cleaner way of interfacing  with a database back end for example and it also splits the duty of authoritative server  from the duty of recursive DNS server which is important to avoid cache poisoning attacks.  Other servers you may have come across are Unbound and NSD.  I mentioned those because they are written by an organisation that I've been involved  with this project that I'm going to tell you about, NLNet Labs.  And again they are much more modern, much more secure DNS servers dedicated to, Unbound  is dedicated to answering recursive requests, NSD is dedicated to answering authoritative  requests.  So let's now get to the subject of this talk, Twisted Names.  So Twisted Names is kind of as old as Twisted itself.  It's celebrating its 13th birthday this year.  It probably started life as you may or may not be able to see from this check-in.  This is the first change set where Twisted Names was first landed.  It was probably introduced as a sort of demo of what was then new UDP transport facility  in Twisted.  I did a little bit of digging and found the commits from the beginning of Twisted Names  life and some of the newer commits that I've been working on and Julian's worked on a bit,  I see.  So you can see it was originally written by a guy called Moshi Zadka back in 2001.  And that was in the good old days when Twisted had a kind of wild west development process.  Everyone was just committing randomly to trunk.  And I guess they hadn't yet implemented what is now called the ultimate quality development  system, which is a talk in its own right.  It's a way of, it's a way, it's the way we develop in Twisted.  It's the way of developing in branches and ensuring that every change that gets merged  to trunk has been code reviewed, that it's fully tested, that the code is fully covered  and that there's an audit trail showing between the ticket and the code that lands in trunk.  But you should go and read about the ultimate quality development system if you haven't  already.  So Twisted Names was, it was kind of actively maintained to start with.  And over the years, it kind of, I think it's true to say it's been neglected a bit.  And then I started getting involved about two years ago and I had a background in DNS,  so I thought that's part of Twisted where I could help out.  And I've been busily updating the documentation, adding test coverage, adding some new examples  to demonstrating how to use Twisted Names and you'll find all of those on the new Read  the Docs documentation website.  I'll link to that in a minute.  So yeah, like the rest of Twisted, the Twisted Names package is really well tested.  It's got comprehensive unit tests which are run using a tool called trial, which is a great  test runner.  The unit tests in Twisted Names, if you care to read the code, and probably not the most  sophisticated unit tests in the world, but that's a reflection of the way these testing  techniques improve over time.  Twisted is, as I said earlier, it's over 13 years old.  So if you do go and start hacking on it, you'll find that there are bits of it which are kind  of hard to read, hard to look at without bursting into tears.  But then again, you have to recognize this history of the project and it's actually quite  interesting to see how particular developers who have been with Twisted from the very start  have changed their ideas and their approaches to things like testing.  I think that would be another interesting talk in its own right.  It's interesting from the point of view of a new contributor who has to deal with this  old style code and the new style code and understand what's the current best way of  doing this, developing the Twisted.  So we've got plenty of unit tests and we've got reasonable coverage of the code.  Some of the modules, I don't know whether you can see it on the slide, are not very  well covered, but those are areas which I'm working on now.  Modes such as the authoritative DNS server and the secondary DNS server are not particularly  well covered.  And in fact, that's sort of really its ugly head lately in a bug that's been discovered  in the secondary name server in the latest release of Twisted.  It's partly down to a lack of test coverage.  This bug wasn't noticed earlier and it's partly down to the old style design of that part  of the package.  Hopefully I and others are going to improve that over time.  Twisted names wasn't particularly well documented, but that's improving.  As I said earlier, I've been working quite hard on improving the documentation for Twisted  names and Twisted as a whole is better documented these days.  You can go like most of the projects these days and read the docs on Read the Docs.  And it's nicely presented and nicely indexed and easily searchable.  So I recommend if you're interested that you go and read the documentation for Twisted  names and for the rest of Twisted because it's much easier these days to navigate the  documentation.  So how am I doing for time?  I'm running out of time.  I'm going over time.  I'm running out of time.  So let's crack on.  We'll have a look now at the different modules in Twisted names.  I'm going to start at the lowest level and work up.  Like everywhere in Twisted, there are layers of abstraction, layers upon layers and on  layers.  And we'll start at the bottom and give some examples of how you can use these low level  APIs and then later we'll see some of the higher level abstractions.  So let's start with the Twisted Names DNS module.  Now this contains protocol level APIs, representations of the DNS records, representations of DNS  messages, routines for serializing and deserializing these messages from the wire.  And it's also in this module that you'll find the protocol implementations both for UDP  and TCP because DNS operates over both of those transports.  So we've got a little example here which I'll try and talk you through.  Can you see that?  Let me try and zoom in a bit.  This is Reveal.js and I'm not sure whether it's going to...  Oh, yeah.  Yeah.  Can everyone see that?  Great.  Okay.  So what we're looking at here, there's a couple of things I need to explain.  We have got, first of all, let's start at the bottom and look at the last line which  is task.react.  Now if you've used Twisted before, you may not have come across this API, but this is  a new way for you to start the reactor for a short-lived Twisted program.  And what it does, it supplies a method, you supply it with a function that you want to  run, a function which must return a deferred.  And TaskReact will take that function and supply it with the reactor, run your function  and then wait for the deferred that it returns to fire.  And upon firing, TaskReact will then tear down the reactor, take care of stopping the  services in the right order, and it will log any errors that haven't been handled on that  deferred.  So if we then move up to the main method, we can see that having supplied the reactor,  we're instantiating a DNS datagram protocol.  So in this example, we're only going to be, this DNS client example, we're only going  to be dealing with UDP.  And we instantiate the protocol and then we pass that to React to listen UDP on port zero,  which means any high ephemeral port.  And because this is UDP, we don't have any connections.  So we have to, so it may look odd to be in a client calling a method called listen, but  we are going to send a UDP datagram and we have to be listening for the response, whereas  in TCP, the operating system would set up the connection for you and you wouldn't have  to choose the ephemeral port that the response comes in on.  So we listen on UDP port XYZ and then we send a query using the protocol, using, we call  protocquery to send our DNS query to, in this case, the Google DNS servers.  And then when that query has been answered, we're going to print the result and that's  by way of adding a callback to the deferred returned by protocall.query.  Again, I sometimes think we should start every talk with an introduction to deferreds, but  I guess everyone's heard it and more people these days are familiar with the idea because  it's now part of JavaScript, I think.  So yeah, when the answer comes back, we simply, we take the result and we take it, the result  is a message, a DNS message, which I'll explain in a little bit more detail later, but a message  has three attributes, has an answer's attribute, an authority attribute and an additional attribute.  And those represent the three categories of records that might be returned by a DNS server.  And in this case, all we're doing is printing out the answers returned by the DNS server  and in particular, we're returning, we're going to print out the payload, which is the,  either the, in this case, the quad A record, it might be the A record or the MX record.  We're not interested in printing out the header information which wraps around that  payload.  And so I'll show you the, show you the output.  Oh, I'm running out of time rapidly.  Okay, so we've got an answer from the server, a quad A, one single quad A record.  Now let's quickly move on to the next example.  So that was a client.  The next example is a server.  And in this case, it's quite similar, but we instantiate the datagram protocol this  time with a controller which takes care of handling the, the, the, the, the query which  comes into our server.  And when a query is received by the protocol on port 10, 10,053, our protocol then calls  out to the controller and calls its message received method.  And it's the message received method on the controller, which is responsible for, for  constructing an answer to that message.  So this is how we write low level servers in low level DNS servers in Twisted.  And in this case, we're just going to respond with a count A record with a, with a fixed  IP address.  So hopefully that makes sense.  I haven't got time.  I'd like to go into it in more detail, but I haven't got time.  There's the, there's the server running and there's us issuing a request to it using  dig.  Okay.  So now, those are low level APIs.  If we move up now to twistednames.client, this is a much higher level API, a much more  friendly way of interacting with Twisted names.  And in this example, we're going to, you, we're going to introduce a couple of new concepts.  We're going to use Twisted names to look up concurrently the, the reverse DNS records  for a whole class C network.  And so you can see in our main method that we are constructing a list of all the IP addresses  in a, in a, in a slash 24 network using a really useful module called net adder, which  does all the, the construction of those reverse DNS names for us.  I think there was a, I think there was a lightning talk on it yesterday.  So I recommend that module.  And for each of those reverse DNS names, we're going to call client.lookup pointer.  And client has a, has a series of these lookup methods, one for each type of DNS record  that you can receive from a DNS server.  It doesn't have all of them that we're working on implementing some of the missing ones,  but it has a lookup method for almost every common DNS type.  And so we construct a series of, a list of deferreds, all of them in flight.  And all of them are then added to what's called a deferred list.  Now a deferred list is a, a really useful API for collecting the responses to a list  of deferreds.  And then it fires its callback when all of the deferreds have themselves fired or, or  failed.  So when we handle the results in this example, we are looping through the results, checking  whether the result was a success or a failure.  And if it was a, if it was a success, we print the, we again print the payload.  And we also print a summary, summarizing how many of the requests were answered successfully  and how many of them were not answered, either because the record didn't exist or perhaps  the, the, the query timed out.  So the, the results to that are as follows.  So you can see it all happens rather quickly.  And because everything is happening, happening concurrently.  So that's a real advantage of using Twisted for this sort of work.  I might skip now to a better example of that.  One which follows on from Lynn's talk on Wednesday.  So let me, let me quickly summarize some of the other modules.  We have, in Twisted, we have the modules for, for creating DNS servers.  I had an example of that.  And it's really easy to use because there's a Twisted DNS plugin for the Twisted command  which comes with Twisted.  And so you should explore that and explore all the options that you have using that command.  Twisted itself runs its, TwistedMatrix.com, that domain is actually served from, from  a Twisted DNS server.  So it's, it's, it's pretty, it's pretty stable.  And it's, it's not, it's not a, it's not a fully featured DNS server, but it's good enough  for some cases.  You can see that when you start the server, it logs to standard out and that we can query  that server once it started up.  We also have an authoritative server and it's interesting, but I haven't got time to go  into it, that you can load a DNS zone based by defining it as a Python module.  And so here we have a Python module with describing the zone, but the, the, the, and the, the  objects that you see there are all globals which are imported at the time that this module  is evaluated by Twisted names.  It's quite a clever mechanism, but you should look into that too.  It's an interesting piece of code.  And again, there's the example of how it runs.  And again, these examples are all on the documentation, the Twisted documentation site.  So I recommend you go and read those.  There's a bunch of other modules which I have to skip through.  Common contains some helpers and some, some APIs common to all of the Twisted clients  and servers.  Resolve, I won't go into cache, is about caching the responses to, to queries.  Root is about doing recursive DNS resolution, which Lynn talked about in her talk.  C is about transferring zones and serving them authoritatively from another authoritative  server, which some of you may be familiar with.  And the point I wanted to make by describing all of these is that these, all of these building  blocks can be put together in interesting ways.  And I've done a couple of examples of this on the website.  For example, you could create very easily using the low-level APIs, a module or a script  for test, for compliance testing of DNS servers or clients, because you have complete control  over the, the, the flags and the, and the payloads that you put into messages.  So it's easy to construct non-compliant messages to see how DNS servers respond to those.  Or it's easy to, to, to, it's easy to see how clients respond to non-compliant responses  from servers.  So that's a, that's a good use of these building blocks.  We use the, we use Twisted at Work for, Twisted Names at Work for functional testing.  So we have a bunch of code which does DNS lookups.  And we want to, in our tests, supply canned responses to those DNS lookups.  And it's very easy using Twisted to set up a, set up a lightweight DNS server and then  tear it down at the end of the test.  It would be really easy to set up a database-backed DNS server or a DNS server which looked up  its data from a REST API, for example.  Or it'd be easy you to combine these with other parts, other components in Twisted, like  the Web module or the LDAP module, to look up DNS records from an LDAP database or to,  to control and manage the DNS records in your server using a REST API.  So now let me see if I can quickly show you, finish off with a quick example, a more complicated  example of using Twisted Names.  So in Lin's talk, it was really interesting, there's a tool called DNS Map which is a tool  for, for brute forcing a zone, for guessing which names may be in a zone, a DNS zone.  And it does that using a dictionary of words or common sub-domains.  And as, as you said in your talk, it does it in series.  It's quite, quite a slow, it's quite slow to complete because there's about a thousand  words in its dictionary.  So I thought it'd be interesting to write a tool, write the same tool in Twisted because  it can do all of these lookups concurrently.  So this is, this is how DNS, the DNS, the original DNS map is documented.  You pass it a domain name and you pass it, you can pass it a list of words which it will  then look up each of those words as a sub-domain of the supplied parent domain.  But as you can see, it is quite slow.  So I started this going against Spotify.com and 48 seconds later it had only reached G.  So it was going to take forever.  Now I want to compare that to another example, the example I wrote which, I don't know whether  I've actually given you the link, but I've put all this code on my GitHub page.  I've got the link at the end of my talk.  And in this example, we are, we're actually sending all of our requests concurrently, but  in, not, not, we're not sending a thousand requests at a time.  We're using some, another interesting part of Twisted called the co-operator, the co-operator  API in the task module.  And what we can do using that is to limit the concurrency so that we can say there's  only ever a hundred in-flight DNS requests at a time.  So we're not going to overwhelm the DNS server that we're querying.  I haven't implemented in this the random time-outs that DNS map actually puts in between its  requests.  So it's not quite the same.  But if I show you the results, in this case, it's looked up all, it's about a thousand  sub-domains in two and a half seconds.  So that's a good demonstration of the power of Twisted and the power of the APIs and the  way that it can efficiently process, efficiently send out requests and process the responses.  All this code is on my GitHub page.  Now I think I've run out of time, so I wanted to talk about my project.  I'd love to talk to you about it.  I'm going to be sprinting at the end, tomorrow at least, on Twisted.  So if any of you are interested in helping out or learning about the development process,  if any of you are DNS experts and want to help me with my project, then I'd love to  hear from you.  It's all about EDNS, DNSSEC, I've got funding, so you might get paid for it.  I've made modest progress and I think that's the summary of this talk.  It's the summary of what it's how I wanted this talk to be, but I haven't had time to  cover it all.  So those are the links, those are the links to the documentation, that's the GitHub link  to the examples in my talk if you want to investigate those.  If you, I'll put these on GitHub as well, and I'll link to these from Twitter or somewhere.  I'll make them available on the conference website.  Have I got any time for questions?  No.  If there's any questions, catch up with me afterwards and I'd be delighted to talk to  you about it.  Thank you.  Thank you.
Richard_Wall_-_Twisted_Names__DNS_Building_Blocks_for_Python_Programmers_0_1859.png Richard_Wall_-_Twisted_Names__DNS_Building_Blocks_for_Python_Programmers_1860_6239.png Richard_Wall_-_Twisted_Names__DNS_Building_Blocks_for_Python_Programmers_6240_6443.png Richard_Wall_-_Twisted_Names__DNS_Building_Blocks_for_Python_Programmers_6444_6623.png Richard_Wall_-_Twisted_Names__DNS_Building_Blocks_for_Python_Programmers_6624_7283.png Richard_Wall_-_Twisted_Names__DNS_Building_Blocks_for_Python_Programmers_7284_7379.png Richard_Wall_-_Twisted_Names__DNS_Building_Blocks_for_Python_Programmers_7380_7883.png Richard_Wall_-_Twisted_Names__DNS_Building_Blocks_for_Python_Programmers_7884_8135.png Richard_Wall_-_Twisted_Names__DNS_Building_Blocks_for_Python_Programmers_8136_9059.png Richard_Wall_-_Twisted_Names__DNS_Building_Blocks_for_Python_Programmers_9060_12251.png Richard_Wall_-_Twisted_Names__DNS_Building_Blocks_for_Python_Programmers_12252_14255.png Richard_Wall_-_Twisted_Names__DNS_Building_Blocks_for_Python_Programmers_14256_14687.png Richard_Wall_-_Twisted_Names__DNS_Building_Blocks_for_Python_Programmers_14688_17183.png Richard_Wall_-_Twisted_Names__DNS_Building_Blocks_for_Python_Programmers_17184_18119.png Richard_Wall_-_Twisted_Names__DNS_Building_Blocks_for_Python_Programmers_18120_18383.png Richard_Wall_-_Twisted_Names__DNS_Building_Blocks_for_Python_Programmers_18384_33635.png Richard_Wall_-_Twisted_Names__DNS_Building_Blocks_for_Python_Programmers_33636_40019.png Richard_Wall_-_Twisted_Names__DNS_Building_Blocks_for_Python_Programmers_40020_41243.png Richard_Wall_-_Twisted_Names__DNS_Building_Blocks_for_Python_Programmers_41244_41751.png