Django for IOT: From Hackathon to Production
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Schneider, Anna
It’s Friday night of hackathon weekend. The latest snazzy Internet-connected thingy is sitting on the table next to your beverage of choice, the device’s API docs are open in a browser tab, and your fingers are itching to write some Django. What’s the fastest way to get started? And next month when you come back to it, what will you want to upgrade? This talk will walk through a common IoT use case, sending HTTP requests to turn on and off a device in response to some external data. I do this all the time at WattTime and I'll share some of the tricks I've picked up over the last couple years. We’ll focus on two big differences from your typical blog or polls app: the data model abstractions that fit the problem, and the need to run frequent periodic tasks to hit the device’s API. I'll share a data model that's worked well for me across a bunch of IoT apps. And I'll show you two ways to run those periodic background tasks in Django: a hackathon-friendly version, and a production-friendly version using Celery. You'll walk away with a complete demo template that you can use in your own projects!
 Come on, Noah. Yeah, yeah. Yeah, yeah. Yeah, yeah. Yeah, yeah. Yeah, yeah. Yeah, yeah. Yeah, yeah. Yeah, yeah. Yeah, yeah. Yeah, yeah. Yeah, yeah. Yeah, yeah. Yeah, yeah. Yeah, yeah. Yeah, so I'm Anna. I'm proud to be speaking opposite another Anna. So if you don't want to hang out at the overflow room, you have two Anna's to go see. So I was introduced to Django at the hackathon where my startup was born. So we're called WattTime and we use Django to fight climate change. So we do that by enabling internet connected devices like smart thermostats or the electric vehicle charging stations that you see pictured here to turn on when energy is clean coming from the grid and turn off when it's coming from dirty fossil fuels. Another side project that I'm working on right now is the unconscious bias project. And for that, we're collecting evidence-based resources to help you identify, reduce, and cope with the effects of unconscious bias in science and tech. But later, those are what I'm actually talking about here today. So I'm talking about Django for the Internet of Things from Hackathon to Production. What? That was a lot of the words. Let's break this down a bit. So Django, we all know and love everyone's favorite web framework. The Internet of Things. So this is when something that you don't normally think of as a computer, like a thermostat or a car, it can receive and transmit data and even respond to controls all in real time. And when some people talk about the Internet of Things, they talk about writing the embedded code that runs on the device. But a lot of the devices that you can buy these days have APIs that the vendor provides. So today, I'm going to focus on the use case where you're writing code that interacts with that API. Hackathon. So hackathons are weekend events where you form a team, write a bunch of code, drink a bunch of coffee and or beer. These usually winners and prizes and everyone has a great time. And as IoT has grown in general, it's become particularly popular for hackathon projects. But because hackathons are so short and compressed, they're just the weekends, they're kind of notorious for producing really bad code. So what if you want to put your code into production with real users? So reliability matters in a different way for Internet of Things apps than for a lot of other things you might work on. So say you have a Wi-Fi enabled lock on your house. What happens if the server goes down? Can you still get into your house? Can anyone get into your house? So you'll want to be a little bit more careful when you're coding for production. Eat your fruits and veggies while you're drinking your beer and make sure that your code is extensible and testable and uses dependencies that are reliable. So that when you ship it, you can trust it and go have fun. So today we're going to talk about some design patterns that have worked for us at Watt time while writing and deploying Django projects that monitor and control Internet of Things devices using the vendor provided APIs. Some things will streamline your work for your next hackathon and other things will help you as you move into production. And then as we go, I'll be using this emoji for hackathons, this one for production, and this one for anti-patterns to avoid. So that'll be a couple of those. Great. So imagine it's Friday night. You're at a hackathon and you have this awesome idea for an IoT project. Where do you start? So usually start with some models. So for comparison, let's think about the standard book app that you may have seen in the tutorial. We have an author model with a name and hometown attributes, and we have a book model with title, a foreign piece author, and the year that the book was published. And then now I guess IoT model would look pretty similar to start with. So instead of book, we have a device with a name and a location. And instead of author, sorry, other way around there. So instead of device, we have an author, we have a device with a name and a location. And so the book, we have an observation with a value, a foreign key back to the device and the timestamp. So even though I said the word's wrong, there's a lot that's similar. So what's different? The main change to the device model is to add a vendor provided an ID, something like a MAC address maybe. You'll almost always need this to pass on to the vendor's API to uniquely identify what device you want to be talking to. So this might be analogous to the author's social security number rather than national ID number. The bigger sort of mental change when you're writing these apps is thinking about the observations. So IoT is fundamentally about time series data in the way that most other Jango starter projects aren't. And the number of observations that you'll be dealing with grows a lot faster than the content in the early stage of a typical content driven app. So at a lot of times we tend to deal with medium sized data since we pull new energy data from the power grid every five minutes. So it's not like huge data. Your project might enter big data territory much faster than you expect because many sensors have readings that pull data extremely quickly. And even that my very first hackathon, adding a database index to timestamp, it was the difference between our demo timing out and it actually running quickly and smoothly. So I think the underlying structure of your database tables might be more important. And the last difference I'm going to talk about is you want to be storing different types of observations in a lot of cases. So with books, it's a good bet that the title is going to be well represented by character data. But with IoT, you'll probably want to be storing both numerical data, like how much energy is being used by the device and maybe Boolean data, like whether the device is on or off, possibly also care data like display messages or data with choices like the color of a light bulb. So you might want to use different models for those different data types. Great. So we have a basic data model onto views, right? Wrong. So instead of views, which are for interactions with people, our basic unit of interactivity is going to be tasks, which are for interactions that can be triggered without a human pressing a button. And to see why this is important, let's think about the normal vanilla NBC Django project. In this case, your app is only responsible for sending data to a user if they request it from you. But most third party APIs, including almost every API views for an IoT project, they put the responsibility on your app to make the request to them whenever you want to push your poll data. And of course, a big part of the point of the Internet of Things is machine-to-machine communication, doing what the user magically wants without making a mask for it. So it's crucial that all the important stuff that happens in your app can be initiated outside of a user-driven request response cycle. And then pretty much the entire rest of the talk today is going to be focusing on how to make that happen, because it's pretty tough with the normal views. So what tasks do we need to be doing? The two basic kinds of functionality for an IoT application are going to be monitoring, so getting the state of the device and control, changing the state of the device. So for each kind of observation you have, you might want a separate task function to get the data and to set it if the vendor lets you modify it. Some APIs don't let you modify as many attributes as anybody wants to, so check that before getting your heart to them with specific applications. You'll also need a driver task, so let's call it the do something awesome task. It's going to stitch all of this together. Okay, that's more or less readable. So for this particular demo application, we're going to pass in a device. We're going to, based on some external data, decide whether to turn it on or off. We will pass that is on message onto the device using the device's API. We will save that new status to the database. So that's what's happening here. It's kind of funny, after working on Internet of Things apps for a while, the part where you interact with the device itself has become less exciting to me, because that's the part that's kind of the same between most apps. Like you're talking to a light bulb, that part's going to be the same. But the part that I get excited about now is the decision criteria for why you're turning that light on or off, why you're actually taking action, because that makes your app unique. There's the awesome part. So once you've decided what sort of awesome thing you actually want to do, there you go. So writing the code is going to be as simple or as hard as your idea. But there is one gotcha that we're going to talk about quickly. So we're building up to running these tasks asynchronously. So in a production situation, you can't necessarily count on the state of the database being the same when the task runs as when you called it. So here, since we're passing in the device object as the parameter, what happens with the database may be different by the time we get here. So a better convention that I like is passing in the primary key to the device, and then you can make the query to get exactly the right object that you want from the database right when you need it. Also, if you use a convention of returning a primary key to new objects instead of something like error codes, then you can change different tasks together really easily. Great. So we have our models. We have our tasks. Let's put it together into apps. So in a hackathon situation, just toss it all into one app. No one really cares. But if you want to organize a little bit better, I like to split up my apps this way. So Devices has the device model. Observations might have attributes that is anything else. Interactions is where the tasks go, and I'll have the wrapper around the vendor client API in its own app. So this might seem like a lot of structure for now, but each app will grow over time as you start adding views for adding and removing devices. So the user can manage what they have. If you want to start adding fancy V3 dashboards, you can put those views in the observations app. You're probably going to want to log the interactions that you're making with a device, so that can go into the interactions app. And if you want to start having multiple kinds of devices that you're supporting, then you can swap out the vendor APIs pretty easily. Cool. So you have models. You have tasks. There are in apps. Let's deploy it. How are we going to deploy these tasks? Running the web server isn't actually going to help us here. But we really need something to act like if you're familiar with the concept of Cron or CronTabs, where we want something like that that can run in a distributed cloud environment. So we want to be able to run our tasks anytime we want at frequent and deterministic times and outside of the request response cycle. And this is the part of code where hackathon and production differ most. So I'm going to talk about two different ways to run these tasks, both of which you can deploy to or Haroku or a lot of other cloud environments. So let's do the hackathon way first. I'm using Haroku. It has an add-on called scheduler. We'll use that to run management commands. So you're already familiar with the Django management commands like make migrations or run server probably. And you can write your own management commands too. There's boilerplate in the Django docs and you essentially just copy that and that's a great way to get started. In this case, we'll just have a simple management command that wraps the tasks that we already have. And then to get it running on Haroku scheduler, there's this free add-on with a cute little web interface to set up to run your tasks. So it's pretty easy to set up, which is good. The downsides to using this, which make it only suitable for hackathon situations, is that it has very limited frequencies that it lets you on the tasks. So you can do daily, hourly, or every 10 minutes. But if you want to do every five minutes, you are at a lock. And it's also, they only operate as a best effort service. So I have noticed it's skipped some tasks if you let it run long enough and then are sad when you don't have that one data point. But it's awesome for hackathons. If you need to go into production and do it for real, you'll want both more flexibility and more reliability than these free add-ons can give you. So as the fits production, you got to get your veggies. So we're going to use a package called Celery. So what is Celery? Celery is a distributed message queuing system or asynchronous stuff. So most tutorials for Celery focus on how it's really good for long-running event-driven background tasks, like maybe sending the user a sign-up email after they sign up for your site. That's something that doesn't come up as much in tutorials, but that we're going to be using it for today is scheduling periodic tasks. So in a generic app, you may want to run some daily analytics like Clinton was just using for aggregating all those data for the police calls. But here we're going to be using it for doing our current cloud solution. The main reason I don't recommend Celery for hackathons is that it complicates your architecture a lot. So do a brief overview of everything that you need to have set up to run a Celery app. So we have our web servers. Those can trigger the event-driven tasks. We also run a scheduler server that can trigger the periodic tasks. Both of those put messages on a message broker transport queue. That's what Celery calls it. But it's just a first-in, first-out queue where wherever tasks get put on, they can then get consumed by worker servers that actually do the work of running that task function. So maybe they'll go out and hit external APIs. Maybe they'll put things in the database. They'll do whatever you need them to do. And if there's return data from that function, it will go into something called the re-salt store. There's a bunch of different options for backing services for both the queue and the re-salt store. Redis is a popular one because it's really good for both of them. Rabbit AMQP services are also good for the queue, but not for the re-salt store. So to make it more concrete for IoT application, we're going to have every few minutes, our scheduler is going to put a do something awesome message into the queue. Our worker will pull it off. It will decide whatever you want to do. Maybe turn on our light bulb. It will send the API request to our vendor to say, turn on the light bulb. The vendor will return, yes, it's on now. Good job. The worker will then store that new status in the database and return the primary key to the result store. Once you've got all that architecture configured, actually changing our Python tasks into the salary tasks is really easy. Just add a decorator. And then to configure when the tasks are going to run, there is this dictionary syntax. And the syntax for defining the period to see itself is pretty similar to the UNX CONTAD. So you can have really flexible intervals anywhere between the one minute and once a year, including things like day of the week or like only at noon on Tuesdays or anything like that. And after all my talk about passing primary keys around, you probably tempted to do a query here and pass the results as argument sentences task, right? Unfortunately, this doesn't work very well because this dictionary is only evaluated once, like anything else in your settings file, for instance. So you can only use static arguments here if that's an option for you. If that's not an option, one good sort of workaround is to have a wrapper task that spawns all of the daughter tasks that you need. And so your database call only happens here. So to that end, we have the final production ready project. So we have the same apps as before at the bottom. The schedule file has been added to the interaction. And then there's a bunch of other boilerplate to configure all of the cellular architecture that's involved. So there's some pretty important subtleties to that, but I don't have time to get into you today, but the docs are good. And if you didn't memorize all that, there's also put together a cookie cutter template. GitHub.com. ACHN slash cookie cutter Django IoT. So this has all the boilerplate configured for you. It has all of the data models configured for you. All you have to do is write your actual interactions with the vendor API. And whatever you do, something awesome is this should make things a lot faster for you to deploy. Either with salary or with the management commands. That's all configured to one on Heroku. So hopefully this will be helpful for your next hackathon. So what have we learned? IoT is fundamentally about time series data in a way that most other Django starter projects aren't. So that will mean bigger data, and that will mean you have to think a little bit differently about which of your models are going to be big and which of them are going to be small. IoT is also fundamentally about asynchronous processes instead of the standard MBC request response cycle. So think about putting the actually important stuff not in views, but in somewhere that can be accessed through tasks or something else. A torrento's tasks, easy but really rigid way that's going to hack down situations is Heroku scheduler. And the flexible, more complex way is using salary. And I keep talking about Heroku just because that's really common for hackathons, but salary of course can run anywhere. If you want to get started using some of this better and faster, then cookie cutter Django IoT is ready for your perusal. And just to get a little more philosophical for a sec, I'm getting back to the keynote this morning. IoT is typically thought of in very privileged situations. It makes life easier for people who already have it really easy. But a lot of time we think that the Internet of Things can be more powerful than that. So we like to use IoT for environmental applications, but I encourage you to think about ways in your life where having devices respond to what's happening around them could be much more broadly impactful. Thank you. Thank you. So I kind of too. The person was with the time series data and the primary key. Were you saying make the time the primary key and why? Well, like I did, you kind of went over that pretty quick. I didn't understand. Sorry. So I haven't actually tried doing it that way. I just use the standard Django primary key. And then the other one was why use the salary? Like what benefits did you find in the salary if you got a task over Cron? So if you're running on one of these platform as a service situations, you can't access Cron. Oh, well, I meant that. Oh, I guess he might have answered that question. I understand the salary has the advantage of the acuing for asynchronous tasks. So you have the consumer producer model rather than invocation. But why Heroku versus Cron? Is that also because you may not have access to Cron on the devices? I'm not sure if I understand. Well, just setting up a job in a Cron tab and invoking a Django command rather than having Heroku invoke the command. Yeah, so if you're running an environment where you have access to Cron and you know that you're going to only be running on that machine, then Cron can do the job for you. But in some sort of more distributed situation where you can't rely on having access to those. Question about running a salary in production and how do you, what you're doing against a monitor tasks and making sure that they're running successfully and all of that. Yeah, I didn't have a lot of success with salary flower flower, however you say it. I wanted to like it. You couldn't get a lot out of it. I actually don't use Redis for my production situations. I use cloud AMQP and another AMQP services. And I find the monitor on that is pretty good, but again, not quite as much detail as I want. So lately I've been looking into using Librado and some of those sort of logging things. One nice thing that I just found out about Librado is that you can send alerts based on when a line show up in your log file. So either based on the content or if it just like, if your queue stops reporting for 10 minutes, then you can have Librado trigger and alert based on that. So my internet setting that up as we speak today is so I'm pretty excited to see how that works for us. Is there a reason that I don't use the task or periodic task decorator? Sorry what? The task decorator. Those are really helpful. Yeah, so the task decorator here? Yeah, there's a periodic task decorator where you can just feed it to the content object and is there a reason not to use it? I haven't because I don't like to have the tasks only be available periodically. So if you do it this way you can call it from multiple places in different ways. What's the biggest challenge you've had to solve with doing an IOT project? I guess one application that we're working on right now is controlling smart thermostats. So we have data coming in about whether the power grid in Chicago, how much carbon pollution there is in that electricity. So the API that I'm working with for that partner is a SOAP API. It's been incredibly annoying to deal with. So that little line where it's like status is actually like a huge really annoying method there. So picking your partners carefully is important. Never mind another question. I noticed early on in your slides you had a set status and then a status set create. And I'm wondering what's the purpose of duplicating that. In other words, I'm thinking about the echo beat thermostat where everything you do it just queries everything again from scratch. And I wonder if that model avoids desynchronization of the state of the device versus the database's view of the state of the device. Yeah, getting those out of sync can be a problem. For some devices, they only let you access maybe like 10 days of historical data. And so having your own record of it can be really useful too, even if that does create the additional challenge of trying to keep them up to date. It makes sense maybe to just depend on re-asking the state if that doesn't take too long rather than depending on the state in the database being... Yeah, so if you are able to ask the device for its own historical data, that is likely to be more accurate. So if that's an option, do it. Last question. Do you have anything that's open to prevent some growth in any sense of the past from trying to new date some device state at the same time? That is something that Celery has some helper functions for that is located. Yeah.
 Come on, Noah.  Yeah, yeah.  Yeah, yeah.  Yeah, yeah.  Yeah, yeah.  Yeah, yeah.  Yeah, yeah.  Yeah, yeah.  Yeah, yeah.  Yeah, yeah.  Yeah, yeah.  Yeah, yeah.  Yeah, yeah.  Yeah, yeah.  Yeah, yeah.  Yeah, so I'm Anna.  I'm proud to be speaking opposite another Anna.  So if you don't want to hang out at the overflow room, you have two Anna's to go see.  So I was introduced to Django at the hackathon where my startup was born.  So we're called WattTime and we use Django to fight climate change.  So we do that by enabling internet connected devices like smart thermostats or the electric  vehicle charging stations that you see pictured here to turn on when energy is clean coming  from the grid and turn off when it's coming from dirty fossil fuels.  Another side project that I'm working on right now is the unconscious bias project.  And for that, we're collecting evidence-based resources to help you identify, reduce, and  cope with the effects of unconscious bias in science and tech.  But later, those are what I'm actually talking about here today.  So I'm talking about Django for the Internet of Things from Hackathon to Production.  What?  That was a lot of the words.  Let's break this down a bit.  So Django, we all know and love everyone's favorite web framework.  The Internet of Things.  So this is when something that you don't normally think of as a computer, like a thermostat  or a car, it can receive and transmit data and even respond to controls all in real time.  And when some people talk about the Internet of Things, they talk about writing the embedded  code that runs on the device.  But a lot of the devices that you can buy these days have APIs that the vendor provides.  So today, I'm going to focus on the use case where you're writing code that interacts with that API.  Hackathon.  So hackathons are weekend events where you form a team, write a bunch of code, drink a bunch of coffee and or beer.  These usually winners and prizes and everyone has a great time.  And as IoT has grown in general, it's become particularly popular for hackathon projects.  But because hackathons are so short and compressed, they're just the weekends, they're kind of notorious for producing really bad code.  So what if you want to put your code into production with real users?  So reliability matters in a different way for Internet of Things apps than for a lot of other things you might work on.  So say you have a Wi-Fi enabled lock on your house.  What happens if the server goes down?  Can you still get into your house?  Can anyone get into your house?  So you'll want to be a little bit more careful when you're coding for production.  Eat your fruits and veggies while you're drinking your beer and make sure that your code is extensible and testable and uses dependencies that are reliable.  So that when you ship it, you can trust it and go have fun.  So today we're going to talk about some design patterns that have worked for us at Watt time while writing and deploying Django projects that monitor and control Internet of Things devices using the vendor provided APIs.  Some things will streamline your work for your next hackathon and other things will help you as you move into production.  And then as we go, I'll be using this emoji for hackathons, this one for production, and this one for anti-patterns to avoid.  So that'll be a couple of those.  Great.  So imagine it's Friday night.  You're at a hackathon and you have this awesome idea for an IoT project.  Where do you start?  So usually start with some models.  So for comparison, let's think about the standard book app that you may have seen in the tutorial.  We have an author model with a name and hometown attributes, and we have a book model with title, a foreign piece author, and the year that the book was published.  And then now I guess IoT model would look pretty similar to start with.  So instead of book, we have a device with a name and a location.  And instead of author, sorry, other way around there.  So instead of device, we have an author, we have a device with a name and a location.  And so the book, we have an observation with a value, a foreign key back to the device and the timestamp.  So even though I said the word's wrong, there's a lot that's similar.  So what's different?  The main change to the device model is to add a vendor provided an ID, something like a MAC address maybe.  You'll almost always need this to pass on to the vendor's API to uniquely identify what device you want to be talking to.  So this might be analogous to the author's social security number rather than national ID number.  The bigger sort of mental change when you're writing these apps is thinking about the observations.  So IoT is fundamentally about time series data in the way that most other Jango starter projects aren't.  And the number of observations that you'll be dealing with grows a lot faster than the content in the early stage of a typical content driven app.  So at a lot of times we tend to deal with medium sized data since we pull new energy data from the power grid every five minutes.  So it's not like huge data.  Your project might enter big data territory much faster than you expect because many sensors have readings that pull data extremely quickly.  And even that my very first hackathon, adding a database index to timestamp,  it was the difference between our demo timing out and it actually running quickly and smoothly.  So I think the underlying structure of your database tables might be more important.  And the last difference I'm going to talk about is you want to be storing different types of observations in a lot of cases.  So with books, it's a good bet that the title is going to be well represented by character data.  But with IoT, you'll probably want to be storing both numerical data, like how much energy is being used by the device and maybe Boolean data,  like whether the device is on or off, possibly also care data like display messages or data with choices like the color of a light bulb.  So you might want to use different models for those different data types.  Great. So we have a basic data model onto views, right?  Wrong.  So instead of views, which are for interactions with people, our basic unit of interactivity is going to be tasks,  which are for interactions that can be triggered without a human pressing a button.  And to see why this is important, let's think about the normal vanilla NBC Django project.  In this case, your app is only responsible for sending data to a user if they request it from you.  But most third party APIs, including almost every API views for an IoT project,  they put the responsibility on your app to make the request to them whenever you want to push your poll data.  And of course, a big part of the point of the Internet of Things is machine-to-machine communication,  doing what the user magically wants without making a mask for it.  So it's crucial that all the important stuff that happens in your app can be initiated outside of a user-driven request response cycle.  And then pretty much the entire rest of the talk today is going to be focusing on how to make that happen, because it's pretty tough with the normal views.  So what tasks do we need to be doing?  The two basic kinds of functionality for an IoT application are going to be monitoring, so getting the state of the device  and control, changing the state of the device.  So for each kind of observation you have, you might want a separate task function to get the data and to set it if the vendor lets you modify it.  Some APIs don't let you modify as many attributes as anybody wants to, so check that before getting your heart to them with specific applications.  You'll also need a driver task, so let's call it the do something awesome task.  It's going to stitch all of this together.  Okay, that's more or less readable.  So for this particular demo application, we're going to pass in a device.  We're going to, based on some external data, decide whether to turn it on or off.  We will pass that is on message onto the device using the device's API.  We will save that new status to the database.  So that's what's happening here.  It's kind of funny, after working on Internet of Things apps for a while, the part where you interact with the device itself has become less exciting to me,  because that's the part that's kind of the same between most apps.  Like you're talking to a light bulb, that part's going to be the same.  But the part that I get excited about now is the decision criteria for why you're turning that light on or off, why you're actually taking action,  because that makes your app unique.  There's the awesome part.  So once you've decided what sort of awesome thing you actually want to do,  there you go.  So writing the code is going to be as simple or as hard as your idea.  But there is one gotcha that we're going to talk about quickly.  So we're building up to running these tasks asynchronously.  So in a production situation, you can't necessarily count on the state of the database being the same when the task runs as when you called it.  So here, since we're passing in the device object as the parameter, what happens with the database may be different by the time we get here.  So a better convention that I like is passing in the primary key to the device,  and then you can make the query to get exactly the right object that you want from the database right when you need it.  Also, if you use a convention of returning a primary key to new objects instead of something like error codes,  then you can change different tasks together really easily.  Great. So we have our models. We have our tasks. Let's put it together into apps.  So in a hackathon situation, just toss it all into one app. No one really cares.  But if you want to organize a little bit better, I like to split up my apps this way.  So Devices has the device model. Observations might have attributes that is anything else.  Interactions is where the tasks go, and I'll have the wrapper around the vendor client API in its own app.  So this might seem like a lot of structure for now, but each app will grow over time as you start adding views for adding and removing devices.  So the user can manage what they have. If you want to start adding fancy V3 dashboards,  you can put those views in the observations app.  You're probably going to want to log the interactions that you're making with a device, so that can go into the interactions app.  And if you want to start having multiple kinds of devices that you're supporting, then you can swap out the vendor APIs pretty easily.  Cool. So you have models. You have tasks. There are in apps. Let's deploy it.  How are we going to deploy these tasks? Running the web server isn't actually going to help us here.  But we really need something to act like if you're familiar with the concept of Cron or CronTabs,  where we want something like that that can run in a distributed cloud environment.  So we want to be able to run our tasks anytime we want at frequent and deterministic times and outside of the request response cycle.  And this is the part of code where hackathon and production differ most.  So I'm going to talk about two different ways to run these tasks, both of which you can deploy to or Haroku or a lot of other cloud environments.  So let's do the hackathon way first.  I'm using Haroku. It has an add-on called scheduler. We'll use that to run management commands.  So you're already familiar with the Django management commands like make migrations or run server probably.  And you can write your own management commands too.  There's boilerplate in the Django docs and you essentially just copy that and that's a great way to get started.  In this case, we'll just have a simple management command that wraps the tasks that we already have.  And then to get it running on Haroku scheduler, there's this free add-on with a cute little web interface to set up to run your tasks.  So it's pretty easy to set up, which is good.  The downsides to using this, which make it only suitable for hackathon situations, is that it has very limited frequencies that it lets you on the tasks.  So you can do daily, hourly, or every 10 minutes.  But if you want to do every five minutes, you are at a lock.  And it's also, they only operate as a best effort service.  So I have noticed it's skipped some tasks if you let it run long enough and then are sad when you don't have that one data point.  But it's awesome for hackathons.  If you need to go into production and do it for real, you'll want both more flexibility and more reliability than these free add-ons can give you.  So as the fits production, you got to get your veggies.  So we're going to use a package called Celery.  So what is Celery?  Celery is a distributed message queuing system or asynchronous stuff.  So most tutorials for Celery focus on how it's really good for long-running event-driven background tasks, like maybe sending the user a sign-up email after they sign up for your site.  That's something that doesn't come up as much in tutorials, but that we're going to be using it for today is scheduling periodic tasks.  So in a generic app, you may want to run some daily analytics like Clinton was just using for aggregating all those data for the police calls.  But here we're going to be using it for doing our current cloud solution.  The main reason I don't recommend Celery for hackathons is that it complicates your architecture a lot.  So do a brief overview of everything that you need to have set up to run a Celery app.  So we have our web servers. Those can trigger the event-driven tasks.  We also run a scheduler server that can trigger the periodic tasks.  Both of those put messages on a message broker transport queue.  That's what Celery calls it.  But it's just a first-in, first-out queue where wherever tasks get put on, they can then get consumed by worker servers that actually do the work of running that task function.  So maybe they'll go out and hit external APIs. Maybe they'll put things in the database. They'll do whatever you need them to do.  And if there's return data from that function, it will go into something called the re-salt store.  There's a bunch of different options for backing services for both the queue and the re-salt store.  Redis is a popular one because it's really good for both of them.  Rabbit AMQP services are also good for the queue, but not for the re-salt store.  So to make it more concrete for IoT application, we're going to have every few minutes,  our scheduler is going to put a do something awesome message into the queue.  Our worker will pull it off. It will decide whatever you want to do.  Maybe turn on our light bulb.  It will send the API request to our vendor to say, turn on the light bulb.  The vendor will return, yes, it's on now. Good job.  The worker will then store that new status in the database and return the primary key to the result store.  Once you've got all that architecture configured,  actually changing our Python tasks into the salary tasks is really easy.  Just add a decorator.  And then to configure when the tasks are going to run, there is this dictionary syntax.  And the syntax for defining the period to see itself is pretty similar to the UNX CONTAD.  So you can have really flexible intervals anywhere between the one minute and once a year,  including things like day of the week or like only at noon on Tuesdays or anything like that.  And after all my talk about passing primary keys around,  you probably tempted to do a query here and pass the results as argument sentences task, right?  Unfortunately, this doesn't work very well because this dictionary is only evaluated once,  like anything else in your settings file, for instance.  So you can only use static arguments here if that's an option for you.  If that's not an option, one good sort of workaround is to have a wrapper task that spawns all of the daughter tasks that you need.  And so your database call only happens here.  So to that end, we have the final production ready project.  So we have the same apps as before at the bottom.  The schedule file has been added to the interaction.  And then there's a bunch of other boilerplate to configure all of the cellular architecture that's involved.  So there's some pretty important subtleties to that, but I don't have time to get into you today, but the docs are good.  And if you didn't memorize all that, there's also put together a cookie cutter template.  GitHub.com.  ACHN slash cookie cutter Django IoT.  So this has all the boilerplate configured for you.  It has all of the data models configured for you.  All you have to do is write your actual interactions with the vendor API.  And whatever you do, something awesome is this should make things a lot faster for you to deploy.  Either with salary or with the management commands.  That's all configured to one on Heroku.  So hopefully this will be helpful for your next hackathon.  So what have we learned?  IoT is fundamentally about time series data in a way that most other Django starter projects aren't.  So that will mean bigger data, and that will mean you have to think a little bit differently about which of your models are going to be big and which of them are going to be small.  IoT is also fundamentally about asynchronous processes instead of the standard MBC request response cycle.  So think about putting the actually important stuff not in views, but in somewhere that can be accessed through tasks or something else.  A torrento's tasks, easy but really rigid way that's going to hack down situations is Heroku scheduler.  And the flexible, more complex way is using salary.  And I keep talking about Heroku just because that's really common for hackathons, but salary of course can run anywhere.  If you want to get started using some of this better and faster, then cookie cutter Django IoT is ready for your perusal.  And just to get a little more philosophical for a sec, I'm getting back to the keynote this morning.  IoT is typically thought of in very privileged situations.  It makes life easier for people who already have it really easy.  But a lot of time we think that the Internet of Things can be more powerful than that.  So we like to use IoT for environmental applications, but I encourage you to think about ways in your life where having devices respond to what's happening around them could be much more broadly impactful.  Thank you.  Thank you.  So I kind of too. The person was with the time series data and the primary key.  Were you saying make the time the primary key and why?  Well, like I did, you kind of went over that pretty quick. I didn't understand.  Sorry. So I haven't actually tried doing it that way.  I just use the standard Django primary key.  And then the other one was why use the salary?  Like what benefits did you find in the salary if you got a task over Cron?  So if you're running on one of these platform as a service situations, you can't access Cron.  Oh, well, I meant that.  Oh, I guess he might have answered that question.  I understand the salary has the advantage of the acuing for asynchronous tasks.  So you have the consumer producer model rather than invocation.  But why Heroku versus Cron? Is that also because you may not have access to Cron on the devices?  I'm not sure if I understand.  Well, just setting up a job in a Cron tab and invoking a Django command rather than having Heroku invoke the command.  Yeah, so if you're running an environment where you have access to Cron and you know that you're going to only be running on that machine,  then Cron can do the job for you.  But in some sort of more distributed situation where you can't rely on having access to those.  Question about running a salary in production and how do you, what you're doing against a monitor tasks and making sure that they're running successfully and all of that.  Yeah, I didn't have a lot of success with salary flower flower, however you say it.  I wanted to like it. You couldn't get a lot out of it.  I actually don't use Redis for my production situations. I use cloud AMQP and another AMQP services.  And I find the monitor on that is pretty good, but again, not quite as much detail as I want.  So lately I've been looking into using Librado and some of those sort of logging things.  One nice thing that I just found out about Librado is that you can send alerts based on when a line show up in your log file.  So either based on the content or if it just like, if your queue stops reporting for 10 minutes, then you can have Librado trigger and alert based on that.  So my internet setting that up as we speak today is so I'm pretty excited to see how that works for us.  Is there a reason that I don't use the task or periodic task decorator?  Sorry what?  The task decorator.  Those are really helpful.  Yeah, so the task decorator here?  Yeah, there's a periodic task decorator where you can just feed it to the content object and is there a reason not to use it?  I haven't because I don't like to have the tasks only be available periodically.  So if you do it this way you can call it from multiple places in different ways.  What's the biggest challenge you've had to solve with doing an IOT project?  I guess one application that we're working on right now is controlling smart thermostats.  So we have data coming in about whether the power grid in Chicago, how much carbon pollution there is in that electricity.  So the API that I'm working with for that partner is a SOAP API.  It's been incredibly annoying to deal with. So that little line where it's like status is actually like a huge really annoying method there.  So picking your partners carefully is important.  Never mind another question. I noticed early on in your slides you had a set status and then a status set create.  And I'm wondering what's the purpose of duplicating that. In other words, I'm thinking about the echo beat thermostat where everything you do it just queries everything again from scratch.  And I wonder if that model avoids desynchronization of the state of the device versus the database's view of the state of the device.  Yeah, getting those out of sync can be a problem. For some devices, they only let you access maybe like 10 days of historical data.  And so having your own record of it can be really useful too, even if that does create the additional challenge of trying to keep them up to date.  It makes sense maybe to just depend on re-asking the state if that doesn't take too long rather than depending on the state in the database being...  Yeah, so if you are able to ask the device for its own historical data, that is likely to be more accurate.  So if that's an option, do it.  Last question.  Do you have anything that's open to prevent some growth in any sense of the past from trying to new date some device state at the same time?  That is something that Celery has some helper functions for that is located.  Yeah.
DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_0_1211.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_1212_10787.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_10788_11591.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_11592_14879.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_14880_15815.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_15816_21239.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_21240_23579.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_23580_25583.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_25584_26291.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_26292_27215.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_27216_30023.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_30024_30215.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_30216_31703.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_31704_34031.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_34032_34127.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_34128_34175.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_34176_34259.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_34260_34319.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_34320_34463.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_34464_34619.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_34620_34715.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_34716_34775.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_34776_35951.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_35952_37751.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_37752_38315.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_38316_38555.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_38556_38711.png DjangoCon_US_2016_-_Django_for_IOT__From_Hackathon_to_Production_by_Anna_Schneider_yv9xKEN4rXg_38712_38737.png