Slow Food Digests Better
CC Attribution - NoDerivatives 4.0 International:
You are free to use, copy, distribute and transmit the work or content in unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Grebs, Christopher
AMO - was originally written as a PHP web application, ported to Python / Django 1.1 in 2010, more or less maintained over time and only recently got much more traction because of Firefox Quantum and Mozilla's move to WebExtensions. The talk will show our approach to maintaining very old code, handling refactoring, adding new features as well as feature/code removal while slowly upgrading our way to a Python 3 and Django 2.0 ecosystem and why we chose that approach over a rewrite.
 The next talk is again describing something that many of us have gone through in the past. If you've used Django for a bit or if you've started on a Django project that has been written some time ago, you may have run into the issue of reading the Django documentation and seeing some solution to your problems and starting to use that solution only to notice that you were accidentally browsing the incredibly version Django documentation at the latest version, but the project is some ancient version from like two years ago or something. And so this incredible solution won't work or you had to upgrade Django to the next version and while Django helps you a lot by providing a very detailed change log, this isn't exactly pleasant either. Christopher is going to tell us about a very old Django project and how they at Mozadeh worked with it. Thank you. Thank you. All right. So, hey, I'm Chris. I'm here to talk about a very, very old Python project that originally was a PHP project, but let's start a little bit with the introduction. I'm a software developer for almost 10 years now. And during that time, I worked with many smaller and bigger companies, startups, fully grown companies and have been working with many, many code bases as probably a few other people here have as well. And notice that over time, while also while adding new features and implementing that new cool async stuff and other things can be very exciting or and exhilarating, it's I was more torn towards old, ancient, big projects and had funny enough, lots of fun to tackle them, touch them, work with them, create proper linting, create proper dependency management, continuous integration tests and kind of trying to modernize those projects. One of the projects that I've been working on now since October 2015 on or at Mozilla is addons.mozilla.org, which is actually one of the most or one of the oldest websites at Mozilla and has been around for approximately 13 years now. Addons.mozilla.org has gone through a huge transition from a quickly hacked together PHP project to a now almost modern Django application and will be heading towards Python 3 and Django 2 hopefully sometime in 2019 and maybe starting already in 2018. A few more numbers addons.mozilla.org serves approximately 3 million unique users per week at the code base is something around 250,000 lines of code counting Python, JavaScript, CSS and SQL migrations. So let's start with a little history lesson that is not complete at all because that would take far too long. So in 2005, Mozilla.org, okay, that screenshot is not good, anyhow, started with flat HTML files and sprinkled with PHP code and if you did database queries here and there, so basically just exactly like everyone told you even back then not to do it. It was a beta though, so nobody cared. It has been redesigned over the years many, many times, so here's a new fresh Mozilla design that kind of focused on the idea of promoting customization and customization in Firefox through Firefox addons. The PHP application grew more and more complex over time and so it has been rewritten to be based on a PHP framework, cake PHP version 1.1, which as far as I researched wasn't that mature back then and only version 1.2 made it much easier. So as it happens in life, you settle on the wrong technology or it was probably a very good decision back then, but it turns out later that it isn't. So you have to stick with it and have to work with it over time. Even using a more professional PHP web framework, the whole application got more and more complex over time. Now in 2009, the whole page is still using PHP, but after Django 1.0 got released in September 2008, a few people inside Mozilla started to promote Python and Django as the new technology to move forward and tried to port many of our projects to Python and Django. It started with smaller websites and eventually got also towards projects like AMO. The transitioning time from the PHP version to a Python and Django version took approximately 16 months, plus a little bit of transitioning time, going live, going back and forth, having lots of down times and stuff like that. The whole 16 months were very stressful for the users, obviously, because the site was unreliable. They just wanted to search for an add-on, wanted to install it, wanted to be happy, but that didn't work. And there were bugs, maybe some translation errors, stuff like that. It was also very stressful for the QA team and management, obviously, because, well, issues arise, things take longer than you expect, because you can't just rewrite one code to the other and expect to just type it down and it works. You have to implement new features. You have to work with the data that you originally had in the database, and you have to migrate to the whole new system. So that is very error-prone, and there's only a few ways that you can use to actually test that. And the best way to do that is manual testing, and that's very error-prone as well. So in January 2010, plus a few more months transitioning time, the whole system got pushed into the live system. The big advantage of the rewrite to Django was that during that time, or to Python and Django during that time, lots of new unit tests got added, which simply didn't exist before. It didn't have 100% coverage, but it was close enough. And yeah, lots of new features did happen, and a few bug fixes and wrongly designed features in the PHP version were also corrected as much as possible. So almost six years before, I joined Mozilla, and yeah, the whole porting process started, and this whole, or the new version, or Python and Django version, was called Zimboni. Now Canadians were involved, Canadians loved their ice hockey, so that's Zimboni for everyone who doesn't know what that is. It's one of these ice machines that makes sure the ice is perfectly clean, and yeah, works like expected. So between 2013 and 2015, a few more redesigns. On the left-hand side, you see the result basically after the website got ported to Python and Django, and yeah, over those two years, two or three redesigns happened, which is basically the one thing Mo is famous for every new year gets in your design, because why not? The whole site wasn't too special, actually, so it was primarily using Django. It was sprinkled with lots of raw SQL as it was before, but it had some cool new things like Celery and RabbitMQ for asynchronous task management, and yeah, one problem right after that port basically, and between 2013 and 2015, was that most of the development resources weren't working on AMO, so it kind of got paused during the development, or the development got paused on AMO, and it was left in that state that it was. It was rewritten from scratch. Maybe a few workarounds were there, and essentially, many of the old entries and the old schema of the database back from the PHP world was still there, so as a site note, the page was built around, or the new page was built around the existing database, so that was cool, or not so much. That was because during that time Firefox Marketplace happened, which is now not there, or was discontinued, and many of the AMO resources were working on the Firefox Marketplace site. And, making sure that AMO was ready for Firefox OS, and their apps, so that Firefox OS could use the marketplace to, or users of Firefox OS could use the marketplace to search for an app, install that app, just like everyone knows, Android or iOS, or any other mobile system works. And the interesting part here was that the Firefox Marketplace actually was derived from the AMO work, and initially, the same code base tried to be both, Firefox Marketplace and AMO. So we had funny hacks like there was a model from an add-on, and that add-on model changed depending on a setting, and it was either an app or an add-on, depending on the site note it was running on. Zimboni, on the other hand, so the system that's now running Firefox Marketplace, got cleaned up a lot, and implemented a completely new frontend based on React.js, adding lots and lots of new functionality, and pushing more and more functionality towards elastic search for performance reasons. We do that because our database clusters are very performant, can serve a lot of queries, but there is a huge amount of just static data on our site, and performance tests have shown that elastic search can serve the data that we need much faster than our database cluster, which is also used for some statistical data, so the data isn't stored in the perfect way that we need it. Now all that changed again in 2015, and Olympia was born. Also again, a Zimboni, but a super cool, shiny new version of a Zimboni named Olympia. Around that time, a new team got formed, including myself, a few long-term AMO developers, Marketplace developers, and we all got faced with a very difficult task to use the old add-on-statemzilla.org code, and reanimate that code, basically, and had a huge list of features that we wanted to add, and also needed to make sure that that code will be living in the next few years. So the code back then was ancient. Was barely using Django 1.6 in every attempt to port to a newer version before 1.6 was kind of like installing 1.6, making sure the tests run, making sure the site run, and that's basically it pushed that to the live system as soon as possible, and deprecated features or things that will be eventually removed in Django largely weren't touched. So it was running Django 1.6, but many of the old and internal APIs were still used. We had essentially no unit tests for our JavaScript code, and the site was very JavaScript heavy. We have super awesome, shiny install buttons that have 20-something different states. We have the whole add-on developer upload system that's running asynchronous JavaScript and accessing some APIs, and all that code was tested by hand manually, but didn't have any unit tests at all. So that was very error-prone. Our CSS also was very massive. It had been through many, many redesigns over the time, and we had plain CSS, we had SCSS, we had some less code, and some stylus code, just to name a few systems that we used, and were actually running in production to style the page. So that was painful to see. We had to make a few hard decisions whether or not we want to rewrite the whole thing completely from scratch again, or if we want to improve incrementally. We essentially decided to improve incrementally the code and system over time, because rewriting from scratch for us was almost impossible. We were a very small team and unable to maintain two different versions at the same time, because you have to write the new version, you have to maintain the existing version, and then again you have to transition between both versions back and forth. Also the existing version had a huge amount of features that many of us knew but never had seen since quite some time in the past. Some support for Firefox, Thunderbirds, Cmonkey, themes, different types of add-ons, and still some legacy code when the code was used for both Firefox marketplace and AMO. So a rewrite from scratch wasn't possible, and we decided to incrementally improve the whole system. Another factor was that we had thousands of unit tests that made maintenance much easier, and at least got us some peaceful nights when we pushed some code live and our tests were running or some of the existing tests didn't break. So over the time we had been tasked to port the whole system to Django 1.8 during 2016 and 2017. Removed old code were possible, restructured the whole repository, added proper dependency management, used much more of Travis, moved to GitHub for our issue tracking, involved the community much more, added more documentation, and also had to work towards the bigger goal of Mozilla to move towards the new add-on system which is web extensions, which is using HTML5, JavaScript, and other technologies to build your extension or add-on in the browser. So that happened, and we also had to redesign because we do that every year, and that was one of the bigger challenges back then, and we kind of ended up with one new file that was called restyle.less, which essentially took all the old styles, changed them back and forth, so that the restyle looked good, but all that never ended up actually touching the existing styles that we had in the repository because that would have been way too complex and error-prone because we didn't properly know where they're used. So we ended up writing a completely new front-end from scratch based on React and as a single-page app, which is accessing the add-on server via a new REST API. Much more functionality got moved to Elasticsearch and APIs so that the front-end could use them, and we started with a small experiment during one of our all-hands, which was a very small experiment that used React as a single-page app with server-side rendering and implemented a very small amount of our page, which is the discovery pane that you see here when you go in your Firefox browser to the add-ons page or about add-ons and see recommendations of add-ons that you can install simply by swiping that button to the right-hand side. And that worked out nicely, was pushed to production after a few months of development and kind of laid it or set the groundwork for future development of AMO and the front-end itself. So we rewrote the whole front-end from scratch as well. This work got released in November 2006, 17, and this is how the page was looking in December. The whole work got released right before Firefox 57 or Firefox Quantum got released and had the luxury of focusing entirely on web extensions and themes and a few other pieces. So essentially we did both with AMO. We incrementally improved the back-end of our site and server over time because we had lots of unit tests but a very, very complex system to maintain and had the decision to completely rewrite our front-end from scratch because we didn't have the luxury of unit tests and other maintenance helpers for the front-end and essentially needed something very clean and ended up writing something from scratch. So that now leaves us with a completely new system and we're slowly deprecating features on the old site. For example, when we launch new replacement sites for apps like Thunderbirds, Cmonkey, for example, we can get rid of lots of new code which will make the maintenance of our back-end much, much easier and potentially allow us to move towards more recent Django versions and dependencies and implement new features much, much quicker. So what did we learn? We are heavy users of feature flags and switches and are using Django Waffle for that. We usually try to develop new features that sit behind a waffle flag so that we can push them to production once new things are finished and we have a QA team that can test them on our development and staging systems and have our UX and UI team look at the features that we implemented and have them give us the go when things look good again and then we just have to flip the switch. So that makes it much easier to push new code, make sure it runs on your development and staging systems, also push that code to production without putting it live but also making sure that you didn't break any other systems that you rely on. And that makes it much easier to revert any broken deployments obviously because you only have to flip back that switch that can be or is usually very easy but depends on your feature and if you need any database migrations, for example. So if you have that, you need to take some special care. So another learning that we recently made or almost a year now, or made since almost a year now is third party dependencies can be a huge pain. Have been always since we moved from Django 1.6 to 1.7 and 1.8 with a huge or very old database as ours and using many, many internal functionality of Django, that was a huge pain. And the move from 1.6 to 1.8 alone cost us around three months of more or less full-time development. Also while dependencies and semi-automatic upgrades by PyApp or GreenKipper are awesome and they usually work on the end server, we have often the problem on our front end that we get swamped with JavaScript dependency upgrades every now and then and it's very hard to keep on top of them. So be aware of that. You don't have to be on the bleeding edge. But also make sure you're subscribed to the correct mailing lists and project bug trackers so that you can be aware of any security issues or bug fixes that are potentially impacting your site. Also upgrading ancient dependencies can be very daunting and very time-consuming and it's often hard to find the right balance between doing the upgrade and knowing how long that will take. Experience obviously helps. But what also helps is a very time-boxed work window and experimental branch that you just start up, give you like two hours or three hours and work on that, see what breaks and make a list of things that you think will happen or must happen so that the upgrade can happen. It also helps adding new dependencies to your continuous integration system while working on the system as well so that you have both running at the same time. That makes sure, for example, if you're upgrading Django, that your code base can run, for example, both versions. And then essentially you only have to flip over to, for example, from Django 1.8 to 1.11 in your production system and essentially can remove the old waffle flags or flags and code and work arounds from your 1.8 times. That's something we didn't do for our upgrade from 1.8 to 1.11 and that bit us a bit and it cost us many, many months. We're actually a year of work during that upgrade because we ran into lots and lots of third-party dependencies that were broken, un-maintained because we used many, or back in the days, many of the dependencies that we used were written by Mozilla, maintained by Mozilla, but over time the people changed positions and the projects are un-maintained now and we had to upgrade them step by step or get rid of them. So that's very time-consuming and daunting. So make sure the dependencies that you're using are up-to-date, are well-maintained, or also make sure that you're on top of them and know how to handle them. And if they're un-maintained, make sure that you get rid of them or replace them or contribute to them on time before you run into any problems. I actually can't stress this enough. A QA team actually saves lives. While unit tests are awesome, automated Selenium UI tests are awesome as well, but having a QA team that manually tests your system and also makes sure that edge cases are covered, the whole new features that you add are working as expected is a huge time-saver. Because often, if you're using, for example, MOCs, you can end up not actually testing the right system, so having a QA team that has a clear list of things that need to be tested helps pushing your features, upgrades, dependency upgrades, and even migrations makes all that work much, much easier. Also explaining what you did and how to test it to someone else makes it much easier to reason over the code and the functionality that you implemented. Makes a reason about that or makes yourself reason about that and allows you to see any potential edge cases that you might have forgotten over time. Implement features incrementally and flip the switch to that feature once they're ready. It's not often as simple as that, but if you can do that, do that, because it helps having that code in production sooner rather than later and catch any potential errors on time. Also don't panic. Usually you're there for the long run and you have to make sure the project works for quite some time. If you see some code that really doesn't look that great, there usually was a reason for the code back when it was written like that and no one actually wanted to do something bad because when you write your code, you use your best judgment and go forth with that. That was my talk. Thanks a lot for listening. You can find me on GitHub, Twitter, and there's also the awesome mizela.org. We can find a lot more technical information about AMO and the stuff that we do. If you have any questions, just find me after the talk. Thank you. Thank you.
 The next talk is again describing something that many of us have gone through in the past.  If you've used Django for a bit or if you've started on a Django project that has been  written some time ago, you may have run into the issue of reading the Django documentation  and seeing some solution to your problems and starting to use that solution only to  notice that you were accidentally browsing the incredibly version Django documentation  at the latest version, but the project is some ancient version from like two years ago  or something.  And so this incredible solution won't work or you had to upgrade Django to the next version  and while Django helps you a lot by providing a very detailed change log, this isn't exactly  pleasant either.  Christopher is going to tell us about a very old Django project and how they at Mozadeh  worked with it.  Thank you.  Thank you.  All right.  So, hey, I'm Chris.  I'm here to talk about a very, very old Python project that originally was a PHP project,  but let's start a little bit with the introduction.  I'm a software developer for almost 10 years now.  And during that time, I worked with many smaller and bigger companies, startups, fully grown  companies and have been working with many, many code bases as probably a few other people  here have as well.  And notice that over time, while also while adding new features and implementing that  new cool async stuff and other things can be very exciting or and exhilarating, it's  I was more torn towards old, ancient, big projects and had funny enough, lots of fun  to tackle them, touch them, work with them, create proper linting, create proper dependency  management, continuous integration tests and kind of trying to modernize those projects.  One of the projects that I've been working on now since October 2015 on or at Mozilla  is addons.mozilla.org, which is actually one of the most or one of the oldest websites  at Mozilla and has been around for approximately 13 years now.  Addons.mozilla.org has gone through a huge transition from a quickly hacked together  PHP project to a now almost modern Django application and will be heading towards Python 3 and Django  2 hopefully sometime in 2019 and maybe starting already in 2018.  A few more numbers addons.mozilla.org serves approximately 3 million unique users per week  at the code base is something around 250,000 lines of code counting Python, JavaScript,  CSS and SQL migrations.  So let's start with a little history lesson that is not complete at all because that would  take far too long.  So in 2005, Mozilla.org, okay, that screenshot is not good, anyhow, started with flat HTML  files and sprinkled with PHP code and if you did database queries here and there, so basically  just exactly like everyone told you even back then not to do it.  It was a beta though, so nobody cared.  It has been redesigned over the years many, many times, so here's a new fresh Mozilla design  that kind of focused on the idea of promoting customization and customization in Firefox  through Firefox addons.  The PHP application grew more and more complex over time and so it has been rewritten to  be based on a PHP framework, cake PHP version 1.1, which as far as I researched wasn't that  mature back then and only version 1.2 made it much easier.  So as it happens in life, you settle on the wrong technology or it was probably a very  good decision back then, but it turns out later that it isn't.  So you have to stick with it and have to work with it over time.  Even using a more professional PHP web framework, the whole application got more and more complex  over time.  Now in 2009, the whole page is still using PHP, but after Django 1.0 got released in  September 2008, a few people inside Mozilla started to promote Python and Django as the  new technology to move forward and tried to port many of our projects to Python and Django.  It started with smaller websites and eventually got also towards projects like AMO.  The transitioning time from the PHP version to a Python and Django version took approximately  16 months, plus a little bit of transitioning time, going live, going back and forth, having  lots of down times and stuff like that.  The whole 16 months were very stressful for the users, obviously, because the site was  unreliable.  They just wanted to search for an add-on, wanted to install it, wanted to be happy, but that  didn't work.  And there were bugs, maybe some translation errors, stuff like that.  It was also very stressful for the QA team and management, obviously, because, well,  issues arise, things take longer than you expect, because you can't just rewrite one  code to the other and expect to just type it down and it works.  You have to implement new features.  You have to work with the data that you originally had in the database, and you have to migrate  to the whole new system.  So that is very error-prone, and there's only a few ways that you can use to actually test  that.  And the best way to do that is manual testing, and that's very error-prone as well.  So in January 2010, plus a few more months transitioning time, the whole system got pushed  into the live system.  The big advantage of the rewrite to Django was that during that time, or to Python and  Django during that time, lots of new unit tests got added, which simply didn't exist  before.  It didn't have 100% coverage, but it was close enough.  And yeah, lots of new features did happen, and a few bug fixes and wrongly designed features  in the PHP version were also corrected as much as possible.  So almost six years before, I joined Mozilla, and yeah, the whole porting process started,  and this whole, or the new version, or Python and Django version, was called Zimboni.  Now Canadians were involved, Canadians loved their ice hockey, so that's Zimboni for everyone  who doesn't know what that is.  It's one of these ice machines that makes sure the ice is perfectly clean, and yeah, works  like expected.  So between 2013 and 2015, a few more redesigns.  On the left-hand side, you see the result basically after the website got ported to Python  and Django, and yeah, over those two years, two or three redesigns happened, which is  basically the one thing Mo is famous for every new year gets in your design, because why not?  The whole site wasn't too special, actually, so it was primarily using Django.  It was sprinkled with lots of raw SQL as it was before, but it had some cool new things  like Celery and RabbitMQ for asynchronous task management, and yeah, one problem right  after that port basically, and between 2013 and 2015, was that most of the development  resources weren't working on AMO, so it kind of got paused during the development, or the  development got paused on AMO, and it was left in that state that it was.  It was rewritten from scratch.  Maybe a few workarounds were there, and essentially, many of the old entries and the old schema  of the database back from the PHP world was still there, so as a site note, the page was  built around, or the new page was built around the existing database, so that was cool, or  not so much.  That was because during that time Firefox Marketplace happened, which is now not there,  or was discontinued, and many of the AMO resources were working on the Firefox Marketplace site.  And, making sure that AMO was ready for Firefox OS, and their apps, so that Firefox OS could  use the marketplace to, or users of Firefox OS could use the marketplace to search for  an app, install that app, just like everyone knows, Android or iOS, or any other mobile  system works.  And the interesting part here was that the Firefox Marketplace actually was derived from  the AMO work, and initially, the same code base tried to be both, Firefox Marketplace  and AMO.  So we had funny hacks like there was a model from an add-on, and that add-on model changed  depending on a setting, and it was either an app or an add-on, depending on the site note  it was running on.  Zimboni, on the other hand, so the system that's now running Firefox Marketplace, got  cleaned up a lot, and implemented a completely new frontend based on React.js, adding lots  and lots of new functionality, and pushing more and more functionality towards elastic  search for performance reasons.  We do that because our database clusters are very performant, can serve a lot of queries,  but there is a huge amount of just static data on our site, and performance tests have  shown that elastic search can serve the data that we need much faster than our database  cluster, which is also used for some statistical data, so the data isn't stored in the perfect  way that we need it.  Now all that changed again in 2015, and Olympia was born.  Also again, a Zimboni, but a super cool, shiny new version of a Zimboni named Olympia.  Around that time, a new team got formed, including myself, a few long-term AMO developers, Marketplace  developers, and we all got faced with a very difficult task to use the old add-on-statemzilla.org  code, and reanimate that code, basically, and had a huge list of features that we wanted  to add, and also needed to make sure that that code will be living in the next few years.  So the code back then was ancient.  Was barely using Django 1.6 in every attempt to port to a newer version before 1.6 was  kind of like installing 1.6, making sure the tests run, making sure the site run, and that's  basically it pushed that to the live system as soon as possible, and deprecated features  or things that will be eventually removed in Django largely weren't touched.  So it was running Django 1.6, but many of the old and internal APIs were still used.  We had essentially no unit tests for our JavaScript code, and the site was very JavaScript heavy.  We have super awesome, shiny install buttons that have 20-something different states.  We have the whole add-on developer upload system that's running asynchronous JavaScript  and accessing some APIs, and all that code was tested by hand manually, but didn't have  any unit tests at all.  So that was very error-prone.  Our CSS also was very massive.  It had been through many, many redesigns over the time, and we had plain CSS, we had SCSS,  we had some less code, and some stylus code, just to name a few systems that we used, and  were actually running in production to style the page.  So that was painful to see.  We had to make a few hard decisions whether or not we want to rewrite the whole thing  completely from scratch again, or if we want to improve incrementally.  We essentially decided to improve incrementally the code and system over time, because rewriting  from scratch for us was almost impossible.  We were a very small team and unable to maintain two different versions at the same time, because  you have to write the new version, you have to maintain the existing version, and then  again you have to transition between both versions back and forth.  Also the existing version had a huge amount of features that many of us knew but never  had seen since quite some time in the past.  Some support for Firefox, Thunderbirds, Cmonkey, themes, different types of add-ons, and still  some legacy code when the code was used for both Firefox marketplace and AMO.  So a rewrite from scratch wasn't possible, and we decided to incrementally improve the  whole system.  Another factor was that we had thousands of unit tests that made maintenance much easier,  and at least got us some peaceful nights when we pushed some code live and our tests were  running or some of the existing tests didn't break.  So over the time we had been tasked to port the whole system to Django 1.8 during 2016  and 2017.  Removed old code were possible, restructured the whole repository, added proper dependency  management, used much more of Travis, moved to GitHub for our issue tracking, involved  the community much more, added more documentation, and also had to work towards the bigger goal  of Mozilla to move towards the new add-on system which is web extensions, which is using  HTML5, JavaScript, and other technologies to build your extension or add-on in the browser.  So that happened, and we also had to redesign because we do that every year, and that was  one of the bigger challenges back then, and we kind of ended up with one new file that  was called restyle.less, which essentially took all the old styles, changed them back  and forth, so that the restyle looked good, but all that never ended up actually touching  the existing styles that we had in the repository because that would have been way too complex  and error-prone because we didn't properly know where they're used.  So we ended up writing a completely new front-end from scratch based on React and as a single-page  app, which is accessing the add-on server via a new REST API.  Much more functionality got moved to Elasticsearch and APIs so that the front-end could use them,  and we started with a small experiment during one of our all-hands, which was a very small  experiment that used React as a single-page app with server-side rendering and implemented  a very small amount of our page, which is the discovery pane that you see here when  you go in your Firefox browser to the add-ons page or about add-ons and see recommendations  of add-ons that you can install simply by swiping that button to the right-hand side.  And that worked out nicely, was pushed to production after a few months of development  and kind of laid it or set the groundwork for future development of AMO and the front-end  itself.  So we rewrote the whole front-end from scratch as well.  This work got released in November 2006, 17, and this is how the page was looking in December.  The whole work got released right before Firefox 57 or Firefox Quantum got released and had  the luxury of focusing entirely on web extensions and themes and a few other pieces.  So essentially we did both with AMO.  We incrementally improved the back-end of our site and server over time because we had  lots of unit tests but a very, very complex system to maintain and had the decision to  completely rewrite our front-end from scratch because we didn't have the luxury of unit tests  and other maintenance helpers for the front-end and essentially needed something very clean  and ended up writing something from scratch.  So that now leaves us with a completely new system and we're slowly deprecating features  on the old site.  For example, when we launch new replacement sites for apps like Thunderbirds, Cmonkey,  for example, we can get rid of lots of new code which will make the maintenance of our  back-end much, much easier and potentially allow us to move towards more recent Django  versions and dependencies and implement new features much, much quicker.  So what did we learn?  We are heavy users of feature flags and switches and are using Django Waffle for that.  We usually try to develop new features that sit behind a waffle flag so that we can push  them to production once new things are finished and we have a QA team that can test them on  our development and staging systems and have our UX and UI team look at the features that  we implemented and have them give us the go when things look good again and then we just  have to flip the switch.  So that makes it much easier to push new code, make sure it runs on your development and  staging systems, also push that code to production without putting it live but also making sure  that you didn't break any other systems that you rely on.  And that makes it much easier to revert any broken deployments obviously because you only  have to flip back that switch that can be or is usually very easy but depends on your  feature and if you need any database migrations, for example.  So if you have that, you need to take some special care.  So another learning that we recently made or almost a year now, or made since almost  a year now is third party dependencies can be a huge pain.  Have been always since we moved from Django 1.6 to 1.7 and 1.8 with a huge or very old  database as ours and using many, many internal functionality of Django, that was a huge pain.  And the move from 1.6 to 1.8 alone cost us around three months of more or less full-time  development.  Also while dependencies and semi-automatic upgrades by PyApp or GreenKipper are awesome  and they usually work on the end server, we have often the problem on our front end that  we get swamped with JavaScript dependency upgrades every now and then and it's very hard to keep  on top of them.  So be aware of that.  You don't have to be on the bleeding edge.  But also make sure you're subscribed to the correct mailing lists and project bug trackers  so that you can be aware of any security issues or bug fixes that are potentially impacting  your site.  Also upgrading ancient dependencies can be very daunting and very time-consuming and  it's often hard to find the right balance between doing the upgrade and knowing how long  that will take.  Experience obviously helps.  But what also helps is a very time-boxed work window and experimental branch that you just  start up, give you like two hours or three hours and work on that, see what breaks and  make a list of things that you think will happen or must happen so that the upgrade can  happen.  It also helps adding new dependencies to your continuous integration system while working  on the system as well so that you have both running at the same time.  That makes sure, for example, if you're upgrading Django, that your code base can run, for example,  both versions.  And then essentially you only have to flip over to, for example, from Django 1.8 to 1.11  in your production system and essentially can remove the old waffle flags or flags and  code and work arounds from your 1.8 times.  That's something we didn't do for our upgrade from 1.8 to 1.11 and that bit us a bit and  it cost us many, many months.  We're actually a year of work during that upgrade because we ran into lots and lots of third-party  dependencies that were broken, un-maintained because we used many, or back in the days,  many of the dependencies that we used were written by Mozilla, maintained by Mozilla,  but over time the people changed positions and the projects are un-maintained now and  we had to upgrade them step by step or get rid of them.  So that's very time-consuming and daunting.  So make sure the dependencies that you're using are up-to-date, are well-maintained,  or also make sure that you're on top of them and know how to handle them.  And if they're un-maintained, make sure that you get rid of them or replace them or contribute  to them on time before you run into any problems.  I actually can't stress this enough.  A QA team actually saves lives.  While unit tests are awesome, automated Selenium UI tests are awesome as well, but having a  QA team that manually tests your system and also makes sure that edge cases are covered,  the whole new features that you add are working as expected is a huge time-saver.  Because often, if you're using, for example, MOCs, you can end up not actually testing  the right system, so having a QA team that has a clear list of things that need to be  tested helps pushing your features, upgrades, dependency upgrades, and even migrations makes  all that work much, much easier.  Also explaining what you did and how to test it to someone else makes it much easier to  reason over the code and the functionality that you implemented.  Makes a reason about that or makes yourself reason about that and allows you to see any  potential edge cases that you might have forgotten over time.  Implement features incrementally and flip the switch to that feature once they're ready.  It's not often as simple as that, but if you can do that, do that, because it helps having  that code in production sooner rather than later and catch any potential errors on time.  Also don't panic.  Usually you're there for the long run and you have to make sure the project works for  quite some time.  If you see some code that really doesn't look that great, there usually was a reason for  the code back when it was written like that and no one actually wanted to do something  bad because when you write your code, you use your best judgment and go forth with that.  That was my talk.  Thanks a lot for listening.  You can find me on GitHub, Twitter, and there's also the awesome mizela.org.  We can find a lot more technical information about AMO and the stuff that we do.  If you have any questions, just find me after the talk.  Thank you.  Thank you.
DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_0_5795.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_5796_9059.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_9060_9815.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_9816_9959.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_9960_10319.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_10320_10547.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_10548_10883.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_10884_11123.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_11124_11387.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_11388_11963.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_11964_12251.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_12252_13703.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_13704_16151.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_16152_16583.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_16584_17087.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_17088_17147.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_17148_17303.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_17304_18563.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_18564_18623.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_18624_18671.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_18672_19775.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_19776_19919.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_19920_21863.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_21864_22547.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_22548_24899.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_24900_26747.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_26748_30887.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_30888_31031.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_31032_31571.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_31572_32279.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_32280_32519.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_32520_37019.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_37020_37235.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_37236_37451.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_37452_37859.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_37860_38147.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_38148_38615.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_38616_38915.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_38916_39179.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_39180_39731.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_39732_39851.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_39852_40295.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_40296_40751.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_40752_42227.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_42228_42695.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_42696_43103.png DjangoCon_2018_-_Slow_Food_Digests_Better_-_or_how_to_maintain_a_8_43104_45575.png