Banking with Django - how to not lose your customer's money
CC Attribution - NoDerivatives 4.0 International:
You are free to use, copy, distribute and transmit the work or content in unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Kääriäinen, Anssi
How Holvi decided to pick Django as part of their core infrastructure. I'll go through both the business reasons for using Django for banking (5-10 minutes), and technical details of how we do reliable distributed software which keeps our and customer's money safe (10-20 minutes).
 Our next speaker is Ansi Kariyain. I hope I pronounced the last name correctly. He's one of the, he's a part of the Django team and he's brought a huge part of the ORM. And now he's going to talk to us today about how we lose all our customers, sorry, how we not lose our customers money. So my name is Ansi Kariyain. I'm here to talk about using Django in a banking setup and of course about making reliable banking with Django. First a bit about me, so I work at the whole of it as a back-end lead. I've been working over there for a couple of years. I enjoy it a lot. I'm also, have been doing some ORM work. So from 2012 I have been Django contributor, worked on the ORM a lot until a couple of years ago. Now I have a couple of small kids, so that's taking all of my time, but I'm hoping to get back to Django at some point. So first a quick introduction to what Hall of it does. So we offer business banking services to micro entrepreneurs in Finland, Germany, Austria and to eResidence of Estonia. The eResidence is actually a pretty fun setup. So anywhere in the world you can go to Estonian embassy, register as an eEstonian and then you can set up banking, you can set up a business like you were a real Estonian origin. And that way you can, without ever visiting Estonia, you can actually set up a business in Europe. Hall of it does, Hall of it has a payments account, so it's very much like a banking, a bank account, except for legal reasons. It's kind of different. But for practical purposes, it offers you a separate payment, it offers you a master card, it offers you online payments which you can use with the whole way online shop. On top of the bank account, we have some business tools. So we have invoices, you can create invoices, send them to somebody, collect money so you get a separate transfer in. We automatically match the payment to the invoice, we do the bookkeeping for you. We have an online shop so you can sell stuff online, it's for small shops, but it's really quick to set up so it takes a couple of minutes to start selling something online. Again, you get the money immediately when somebody buys something from you, you get the money immediately on your account and we do the bookkeeping, all of that for you. Finally, we have the expense handling, so when you're traveling you can go to somewhere with taxi, use your master card, you get a push notification when you pay, so you can do the bookkeeping easily. On the go, you take the picture to receive, you do the categorization, the VAT stuff, so you get the bookkeeping preparation done real time. So that is what Holbeidahs, it's an interesting offering, try it out if you're in one of the markets or if you want to be an Estonian. Our tech stack, nothing more special going on in there, but I'll go through it in any case so you understand what we are using on the backend side. So we have Django, of course, then we have Django rest framework using PostgreSQL, then we have a salary for asynchronous tasks for doing these scheduled tasks, using Redis behind that. We have Angular on the front-end side, I don't know that much about it, but it works really well for us. We have the native mobile apps, we have iOS, Android, and finally we are running all of this on Amazon, so we use S3, we use the EC2, we use a lot of services, RDS for example from Amazon, and it's working really well for us. Django for us has been a good choice. The ecosystem is really nice. For pretty much anything you want to do, you find something already existing, so that's a big plus when using Django. It's an overstatement to say that it's easy to find developers, but there are a lot of developers using Django. At the moment in Finland, of course, anybody who has seen a computer is immediately hired, so it's kind of not that easy to find the developers, but at least there are a lot of developers who know Django. And finally, it's reliable. It's proven software, it works really well for backing setup. We are using Django for 99% of the code we have, and it works really well. It's reliable. As all of the basic things you need for building something for banking. So my subtopic for today is that how not to lose your customers' money. There are two options. The first option, which we try to use, is reliable payments. You don't lose the payments. You always get them go through the system and everybody's happy. But in case there's an error somewhere, you lose a message, you have a bug somewhere. It's not that you lose your customers' money, you actually lose your own money. You need to use a lot of time for each error case to solve it out that, hey, is this really an error case? What's happening in the system? Where is the money? And in the worst case, for example, you can double execute a payment. You send 1,000 euros two times to somewhere and it's often impossible to get the money back. So there it went. You lost your own money. But you actually don't ever lose your customers' money because that's the way banks work. You take the losses for yourself instead of anything going for the customers. So I'm quickly going to introduce the payment flow. So just to have the mindset for how this thing works in a whole lot. This is a bit simplified, but it has all the major pieces in there. So it starts with customer verifying a payment, typing in the I-BAN and stuff like that. Then they receive a push notification with the security token. So we do two factor authentication with that. They type it in, click verify, the payment goes out. Then actually starts the payment processing inside whole. So first we record this in the front-end side, the customer facing side of the back-end. And then we send the message to core accounting, the next layer, which handles actually all of the money. We want to keep that at least conceptually separate from the customer facing features for security, for performance, for you can change the customer facing features quickly, but you want to keep the core accounting such that you don't change it too often. Then core accounting, we send a message to approvers. This is something that was kind of a surprise for me, that actually the approvers, how do you verify that this payment is OK? It's causing a lot of the issues when you are trying to build a banking setup. So you have a lot of compliance rules from the law, for example, that you don't send money to terrorists, you don't send money for money laundering purposes, and then you of course need to handle your own risk. So you check that, is everything OK with the payment? From kind of if somebody trying to empty the account illegally. Then somehow we approve the payment by some machine learning approach or manually. We get another message for core accounting that, OK, this money is now OK to move out of the side of Hallaby. Then you get the message to Bank Gateway. Bank Gateway is a system we use for interacting the external banking world. The external banking world, in many cases, is based on technologies from 80s. It's very much batch based. So in the Bank Gateway, we take all of the payments and build the batch of those, send an XML somewhere outside of our system. But inside Hallaby, everything is message based. Everything is real type, but when it gets to the external banking, then it turns into a batch. In the other direction, it's pretty much the same setup. So now you get the batch from external banking, then you get the payment message to core accounting from your Bank Gateway. You do the approvals, couple of messages going over in there. Then you get the message to the business tool, so the customer facing backend, where we do all of the bookkeeping and stuff like that. And finally, you try to automate the bookkeeping. And or depending on the case, you send a notification to the customer. Now you receive 100 euros of money. So the reliability in this case, for a startup, you might have 100 payments a day. It's low amount. At this point, you probably have a kind of a simple system still, so you just use couple of messages. You don't do the approvals by having a separate system somewhere, but you do it maybe by just using a Django admin and checking that it seems okay. Your reliability might not be that high, three nines. So one in thousand messages of these two messages is lost. This means that you get one case a week. It's fine. That's easily doable by hand, so no problem in that case. Actually an interesting story about Holvy. In the beginning, very beginning of Holvy, the system works such that when the user typed the payment out, they filled the form, clicked okay, pay this. What happened in the backend was that somebody copied the data from Django admin to an external bank by hand, and it worked for the very beginning of Holvy. So getting back to the sophistication we had in the morning by Daniel, so if you are a startup, you can do something very unsophisticated, and it works really well for the beginning, so you can prove the concept and then start improving it. Now Holvy is now in the mid-sized phase. These numbers are not real numbers from Holvy. We have a bit different numbers, but this kind of reflect what we have in Holvy. So 10,000 payments a day, five messages per payment. So now as you saw, the flows, the messaging amounts are now a bit higher. We have the approval system, we have the backend, or the customer-facing backend, we have the core accounting backend talking with each other. The reliability has gone up a bit, so you lose one message out of 10,000. This means that you get on average five cases a day. And if it takes you a couple of hours, if you haven't automated resolving the payments, if it takes a couple of hours per payment, failed payment, you need now a dedicated person, maybe even two dedicated persons to solve out these payments. And it's really not cost-effective. So at this point, you want to start to have reliability, and we are now building the reliability for the messaging. Finally, on the enterprise level, just as an example, if you get a million payments a day, 10 messages per payment, so now you're making it more microservice style, you have improved your reliability. You have invested a lot on your architecture, kind of the hardware, and now you have five nights reliability, which is really good, but you still get 100 cases a day, which means that you have a department of solving these cases. So when you scale up, you want to build more reliability into your system. So the thing that I see very often is that people build these systems, microservices, or other systems where you have messaging, but they don't build the reliability into the messaging. That's fine if you don't have that many messages or if it's fine to lose some messages. So if you, for example, to push notification we send to the customers, it's fine to lose one in 10,000. It's not a problem at all. But the payment messages, each time we lose it, pretty much each time, we will get a call from the customer that, hey, now my money is missing. Do something. And we get to do the work or we get to pay the money. So we want to build the messaging reliable. So the idea we are using is that we, in the origin system, for example, in the approval system, when we record that, hey, okay, this payment is now approved, at the same time, we record a message to the local database inside the same transaction. So if the approval gets committed to the database, also the message gets committed to the database. Then on commit, using Django's on commit, you can find more about it in the documentation, we send the message. So only after it has been committed to the database, we send the message to the next system. The idea over here is that if you send the message before it's committed to the system, you can have the hardest to debug fail case. So for example, if you have the customer verifying a payment, you first record that inside the transaction to the database. Then you send the message about that. And then for some reason, the transaction doesn't get committed. You have a message over in the core accounting that, okay, customers send 100 euros of money out of the account. But in the origin system, you see nothing about that. You might have something in the logs, but you have this phantom message. You don't see anything in the database. And it's really hard to debug these cases. So use on commit if you are doing messaging, because in the other direction, it's quite reliable, but when you get the error case, it's really hard to debug. Okay, now on commit is not guaranteed to run, or you can have some errors. Actually, we are using on commit. We are actually in a hallway. We are creating an asynchronous task, which we then execute because on commit, you don't want to run anything heavy. But you can kind of do lightweight HTTP requests, for example, in the on commit. But it might fail. You might have a network error. You might have the other system might be done. It might be overloaded. So if you get the batch of payments in, thousand payments coming in, and you fire as fast as you can to the other system, it will get overloaded. And then you need to retry. So the idea is that only on commit, you send the message from the local database, then you retry if it doesn't go first through. Finally, on the receiving side, when you are retrying, you need to de-doublek it. So you get this idempotency setup where basically the semantics are exactly once. This way, we have reliable messaging. This works. I have also been talking with some consultants from other companies, and they are using for banking, and they are using something very much like this. So it's a proven system. Some might say that it's maybe a bit heavyweight, but it's really reliable. Okay. Now, you can abstract this model. You can have this inbox-outbox model. So in the oraching system, you have the outbox where you record the messages. Basically you have an id. You have the payload. You have a couple of timestamps when the message was created, when was it processed or sent out from the system. Then you do the oncoming send and so on. On the receiving side, all of the messages from the outbox of the other system, you store them in an inbox. And you should have a unique constraint on the inbox so that the origin's id, you don't process the same message multiple times. So doing a de-douplication by having a post-create-square unique constraint, or, well, database unique constraint, doesn't need to be post-create-square. Basically you do reconciliation checks with the inbox and outbox. You check, okay, in the last hour, how many messages did I send from system A? How many messages do I have in the outbox? And in the receiving system, you check, okay, how many messages there are in the inbox? And do the count-match, if they don't, you can react fast. Also you can easily, if some message is lost somewhere, even with this setup, you can, from the outbox, you can just have an admin view and click some message that, okay, recent this. Okay, how do you actually transport the messages? For reliability, it turns out it's not important. You can use HTTP with REST. So now the simple setup is that you have one original system, you have one receiver system, and you send the messages just using the standard requests library and then REST framework on the other side. It works pretty well. Of course you can have these cases where you overload the other system, but it's fine. You have this free trying logic, so it will be eventually sent over to the other system. But if you want to do, for example, this PubSub type of thing, it's really nice to use something in between that does the PubSub, so you, from the origin, you still have the outbox. From the outbox on-commit, you send to Kafka. Kafka does it thing. It can do things like check which receivers are allowed to see which messages. You kind of get the rate limiting because the receivers can play the log as fast as they can, but if they fall behind, it's fine. They can just keep on doing their stuff until they catch up. And of course the receivers over here, they can be different systems. And so you get through PubSub, so you can do stuff like, okay, I got to payment in at the same time, send a push notification to the customer. At the same time, try to do some machine learning. Use some machine learning based approach on the categorization so we can check that, okay, how to push this message to bookkeeping. And so you can do the true PubSub with Kafka. So transport using Kafka, it's important or it's very useful when you are trying to get these benefits, but it's not actually that important for reliability. And finally, of course, you have the outbox inbox model over here still, so you can do the reconciliation of, okay, I have this message in actually being sent. So this was mostly about reliability considerations of messaging. So now you have a messaging setup that works really well. It turns out as you very likely know that the most likely reason for error is that somebody makes programming mistakes. So what we are doing is that we have this, of course, we are using pretty standard, uh, practices for developing our software. So we have this test-driven development approach. We have right test, we have continuous interactions, we do reviews in GitHub, so everything that goes in, it's actually from compliance point of view also important that you review the code so everything needs to be checked by somebody else. We react quickly to failures, so this is actually pretty important from the customer's point of view that if a payment is delayed for a couple of hours, it's fine if we react to it. If the customer needs to call us, you have a problem in this customer trust in all of it. And of course, you get to do a lot of more work because then you need to do the customer communications that, okay, yes, this payment was lost, we fixed it by doing something and now it's fine. So it costs you a lot of money. Always fix the original reason. So if you find that, hey, something strange is happening, try to find out what's the original reason. Do not just check that, okay, I made the payment now go through, I just clicked somewhere that, okay, try this. Try to find out what's wrong in the system because when you are scaling up, you want to get more and more reliability. Otherwise it's going to be really, really painful if you go for the end price level. And we are using monitoring and reconciliation. I said a bit about reconciliation of the inbox-outbox model, but there is also reconciliation you can do on kind of a business logical level. So you check in one system that how much money you have moved out, you check in other systems that do they match, do they amounts match. If you do this constantly, you will quickly see the cases where you have these error cases and you can fix them quickly without needing to wait for customers to call you. In many cases it happens actually if the customers call you, it takes a lot of time to react because I have seen a couple of cases where it has taken more than a week than when you get the initial call that something is broken. We lose one payment in 10,000 say. Something is broken and then you let it run for a while and you lose more payments and then it gets really painful to fix the data. The data is already in the accounting of the company and you need to somehow then fix past data which is something you really don't want to change the account statement of your bank account or the payment account. If you need to do that, it's going to cost you a lot in customer trust. For monitoring we are also using data docs on the infrastructure layer that works really well for us. So we get nice alerts when we have these error cases. Usually you actually want to check this error rate instead of individual errors when you have a bit more volume because it's kind of assumed that some of the cases are going to fail once and then you retry and then they go through. That was pretty much it. One thing I could say a bit more about is this sophistication. What I said in the startup slide is that when you have a small system, you don't need to have this reliability yet built in. But if you aim to scale or if you are now scaling, you will face this problem at some point. If you have these messages, if you're working with money, you can't lose. If you are working with messages, you can't lose. In that case, I really would recommend that you use this kind of setup where you have actual reliability on the messaging setup. So that was it. Thank you. I'd like to thank the organizers. This has been a really nice example of how to set up an inclusive, professional, well-organized event while still keeping it relaxed. Thank you very much. Thank you, Ansi. We got some time for questions before we go into lunch. So about the inbox-outbox model, can you talk a little bit about that in the sense of are there are two completely database clusters also? Are the systems separated for security reasons? Because you could as well just have the inbox-outbox model in one database or even the same table to do the reconciliation. If you have these two different systems, of course, then you need to have them in two different databases also. If you are using two different databases, we are actually doing, we have kind of a large monolith code base right now, but we are doing messaging from the same code base back to the same code base. So even in that setup, it's good to have this inbox-different model, outbox-different model where you can, the outbox records a bit different data. So in the outbox, you record that, OK, the inbox message ID was this. I received it at this time. The inbox origin, kind of inbox create time was this. In the outbox, you record basically just the data and when you sent it, when you created it, so then you can reconciliate on the create time of the outbox and inbox. And even if you are using it inside one system, one code base, I'd go for two different database models just for the reason of if you want it somewhat split it up, then it's easy to do. Also, when you are a small company, I'd go, not go for micro systems, but have one large code base, but have these messaging approaches, even if you are using a bigger code base. So that's when you scale, you can split it up if you need to. Do we have more questions? Yes, one to the microphone. Hey, I was wondering whether you considered, instead of this inbox and outbox model, to use only append-only data structures and keeping pointers and basically following the logs from one and copying to the other one. I mean, I'm asking this because I'm involved with a project. We do 100 million messages per day and it's not okay to lose a single one. And it's not really hard to do if you emulate the patterns of how Kafka does it internally, also in your own databases. You could, depending on the case, you could use Kafka directly. So instead of recording anything in the database, you first record the message and then you process it in the database. So you go kind of moving from Kafka as a transport for messages, you move for Kafka as the data storage of the messages. So that's one way. Then you use Kafka for these pointers and stuff like that. You could also do that in the database. I haven't considered how to do that, so not sure how it would work. The idea over here is that you would use the outbox model and send it to Kafka. So if you want this 100 million messages per day, so you get it reliably to Kafka. And then you can use Kafka for the kind of having the pointers and doing the replays and stuff like that. My question was because the outbox introduces a mutable intermediate storage, it introduces also the chance to lose the message. It's happened only the outbox. So you will delete after 30 days. So you just insert new messages over there, and the only update you do is that, okay, now this has been pushed to somewhere. But that's the only update. Okay. Other more questions? Going once? Going twice? Thank you, Ansi. Thank you. Thank you.
 Our next speaker is Ansi Kariyain.  I hope I pronounced the last name correctly.  He's one of the, he's a part of the Django team and he's brought a huge part of the  ORM.  And now he's going to talk to us today about how we lose all our customers, sorry, how  we not lose our customers money.  So my name is Ansi Kariyain.  I'm here to talk about using Django in a banking setup and of course about making reliable  banking with Django.  First a bit about me, so I work at the whole of it as a back-end lead.  I've been working over there for a couple of years.  I enjoy it a lot.  I'm also, have been doing some ORM work.  So from 2012 I have been Django contributor, worked on the ORM a lot until a couple of  years ago.  Now I have a couple of small kids, so that's taking all of my time, but I'm hoping to get  back to Django at some point.  So first a quick introduction to what Hall of it does.  So we offer business banking services to micro entrepreneurs in Finland, Germany, Austria  and to eResidence of Estonia.  The eResidence is actually a pretty fun setup.  So anywhere in the world you can go to Estonian embassy, register as an eEstonian and then  you can set up banking, you can set up a business like you were a real Estonian origin.  And that way you can, without ever visiting Estonia, you can actually set up a business  in Europe.  Hall of it does, Hall of it has a payments account, so it's very much like a banking,  a bank account, except for legal reasons.  It's kind of different.  But for practical purposes, it offers you a separate payment, it offers you a master  card, it offers you online payments which you can use with the whole way online shop.  On top of the bank account, we have some business tools.  So we have invoices, you can create invoices, send them to somebody, collect money so you  get a separate transfer in.  We automatically match the payment to the invoice, we do the bookkeeping for you.  We have an online shop so you can sell stuff online, it's for small shops, but it's really  quick to set up so it takes a couple of minutes to start selling something online.  Again, you get the money immediately when somebody buys something from you, you get  the money immediately on your account and we do the bookkeeping, all of that for you.  Finally, we have the expense handling, so when you're traveling you can go to somewhere  with taxi, use your master card, you get a push notification when you pay, so you can  do the bookkeeping easily.  On the go, you take the picture to receive, you do the categorization, the VAT stuff,  so you get the bookkeeping preparation done real time.  So that is what Holbeidahs, it's an interesting offering, try it out if you're in one of the  markets or if you want to be an Estonian.  Our tech stack, nothing more special going on in there, but I'll go through it in any  case so you understand what we are using on the backend side.  So we have Django, of course, then we have Django rest framework using PostgreSQL, then  we have a salary for asynchronous tasks for doing these scheduled tasks, using Redis behind  that.  We have Angular on the front-end side, I don't know that much about it, but it works really  well for us.  We have the native mobile apps, we have iOS, Android, and finally we are running all of  this on Amazon, so we use S3, we use the EC2, we use a lot of services, RDS for example  from Amazon, and it's working really well for us.  Django for us has been a good choice.  The ecosystem is really nice.  For pretty much anything you want to do, you find something already existing, so that's  a big plus when using Django.  It's an overstatement to say that it's easy to find developers, but there are a lot of  developers using Django.  At the moment in Finland, of course, anybody who has seen a computer is immediately hired,  so it's kind of not that easy to find the developers, but at least there are a lot of  developers who know Django.  And finally, it's reliable.  It's proven software, it works really well for backing setup.  We are using Django for 99% of the code we have, and it works really well.  It's reliable.  As all of the basic things you need for building something for banking.  So my subtopic for today is that how not to lose your customers' money.  There are two options.  The first option, which we try to use, is reliable payments.  You don't lose the payments.  You always get them go through the system and everybody's happy.  But in case there's an error somewhere, you lose a message, you have a bug somewhere.  It's not that you lose your customers' money, you actually lose your own money.  You need to use a lot of time for each error case to solve it out that, hey, is this really  an error case?  What's happening in the system?  Where is the money?  And in the worst case, for example, you can double execute a payment.  You send 1,000 euros two times to somewhere and it's often impossible to get the money  back.  So there it went.  You lost your own money.  But you actually don't ever lose your customers' money because that's the way banks work.  You take the losses for yourself instead of anything going for the customers.  So I'm quickly going to introduce the payment flow.  So just to have the mindset for how this thing works in a whole lot.  This is a bit simplified, but it has all the major pieces in there.  So it starts with customer verifying a payment, typing in the I-BAN and stuff like that.  Then they receive a push notification with the security token.  So we do two factor authentication with that.  They type it in, click verify, the payment goes out.  Then actually starts the payment processing inside whole.  So first we record this in the front-end side, the customer facing side of the back-end.  And then we send the message to core accounting, the next layer, which handles actually all  of the money.  We want to keep that at least conceptually separate from the customer facing features  for security, for performance, for you can change the customer facing features quickly,  but you want to keep the core accounting such that you don't change it too often.  Then core accounting, we send a message to approvers.  This is something that was kind of a surprise for me, that actually the approvers, how do  you verify that this payment is OK?  It's causing a lot of the issues when you are trying to build a banking setup.  So you have a lot of compliance rules from the law, for example, that you don't send  money to terrorists, you don't send money for money laundering purposes, and then you  of course need to handle your own risk.  So you check that, is everything OK with the payment?  From kind of if somebody trying to empty the account illegally.  Then somehow we approve the payment by some machine learning approach or manually.  We get another message for core accounting that, OK, this money is now OK to move out  of the side of Hallaby.  Then you get the message to Bank Gateway.  Bank Gateway is a system we use for interacting the external banking world.  The external banking world, in many cases, is based on technologies from 80s.  It's very much batch based.  So in the Bank Gateway, we take all of the payments and build the batch of those, send  an XML somewhere outside of our system.  But inside Hallaby, everything is message based.  Everything is real type, but when it gets to the external banking, then it turns into  a batch.  In the other direction, it's pretty much the same setup.  So now you get the batch from external banking, then you get the payment message to core accounting  from your Bank Gateway.  You do the approvals, couple of messages going over in there.  Then you get the message to the business tool, so the customer facing backend, where we do  all of the bookkeeping and stuff like that.  And finally, you try to automate the bookkeeping.  And or depending on the case, you send a notification to the customer.  Now you receive 100 euros of money.  So the reliability in this case, for a startup, you might have 100 payments a day.  It's low amount.  At this point, you probably have a kind of a simple system still, so you just use couple  of messages.  You don't do the approvals by having a separate system somewhere, but you do it maybe by just  using a Django admin and checking that it seems okay.  Your reliability might not be that high, three nines.  So one in thousand messages of these two messages is lost.  This means that you get one case a week.  It's fine.  That's easily doable by hand, so no problem in that case.  Actually an interesting story about Holvy.  In the beginning, very beginning of Holvy, the system works such that when the user typed  the payment out, they filled the form, clicked okay, pay this.  What happened in the backend was that somebody copied the data from Django admin to an external  bank by hand, and it worked for the very beginning of Holvy.  So getting back to the sophistication we had in the morning by Daniel, so if you are  a startup, you can do something very unsophisticated, and it works really well for the beginning,  so you can prove the concept and then start improving it.  Now Holvy is now in the mid-sized phase.  These numbers are not real numbers from Holvy.  We have a bit different numbers, but this kind of reflect what we have in Holvy.  So 10,000 payments a day, five messages per payment.  So now as you saw, the flows, the messaging amounts are now a bit higher.  We have the approval system, we have the backend, or the customer-facing backend, we have the  core accounting backend talking with each other.  The reliability has gone up a bit, so you lose one message out of 10,000.  This means that you get on average five cases a day.  And if it takes you a couple of hours, if you haven't automated resolving the payments,  if it takes a couple of hours per payment, failed payment, you need now a dedicated person,  maybe even two dedicated persons to solve out these payments.  And it's really not cost-effective.  So at this point, you want to start to have reliability, and we are now building the reliability  for the messaging.  Finally, on the enterprise level, just as an example, if you get a million payments  a day, 10 messages per payment, so now you're making it more microservice style, you have  improved your reliability.  You have invested a lot on your architecture, kind of the hardware, and now you have five  nights reliability, which is really good, but you still get 100 cases a day, which means  that you have a department of solving these cases.  So when you scale up, you want to build more reliability into your system.  So the thing that I see very often is that people build these systems, microservices,  or other systems where you have messaging, but they don't build the reliability into  the messaging.  That's fine if you don't have that many messages or if it's fine to lose some messages.  So if you, for example, to push notification we send to the customers, it's fine to lose  one in 10,000.  It's not a problem at all.  But the payment messages, each time we lose it, pretty much each time, we will get a call  from the customer that, hey, now my money is missing.  Do something.  And we get to do the work or we get to pay the money.  So we want to build the messaging reliable.  So the idea we are using is that we, in the origin system, for example, in the approval  system, when we record that, hey, okay, this payment is now approved, at the same time,  we record a message to the local database inside the same transaction.  So if the approval gets committed to the database, also the message gets committed to the database.  Then on commit, using Django's on commit, you can find more about it in the documentation,  we send the message.  So only after it has been committed to the database, we send the message to the next  system.  The idea over here is that if you send the message before it's committed to the system,  you can have the hardest to debug fail case.  So for example, if you have the customer verifying a payment, you first record that inside the  transaction to the database.  Then you send the message about that.  And then for some reason, the transaction doesn't get committed.  You have a message over in the core accounting that, okay, customers send 100 euros of money  out of the account.  But in the origin system, you see nothing about that.  You might have something in the logs, but you have this phantom message.  You don't see anything in the database.  And it's really hard to debug these cases.  So use on commit if you are doing messaging, because in the other direction, it's quite  reliable, but when you get the error case, it's really hard to debug.  Okay, now on commit is not guaranteed to run, or you can have some errors.  Actually, we are using on commit.  We are actually in a hallway.  We are creating an asynchronous task, which we then execute because on commit, you don't  want to run anything heavy.  But you can kind of do lightweight HTTP requests, for example, in the on commit.  But it might fail.  You might have a network error.  You might have the other system might be done.  It might be overloaded.  So if you get the batch of payments in, thousand payments coming in, and you fire as fast as  you can to the other system, it will get overloaded.  And then you need to retry.  So the idea is that only on commit, you send the message from the local database, then  you retry if it doesn't go first through.  Finally, on the receiving side, when you are retrying, you need to de-doublek it.  So you get this idempotency setup where basically the semantics are exactly once.  This way, we have reliable messaging.  This works.  I have also been talking with some consultants from other companies, and they are using for  banking, and they are using something very much like this.  So it's a proven system.  Some might say that it's maybe a bit heavyweight, but it's really reliable.  Okay.  Now, you can abstract this model.  You can have this inbox-outbox model.  So in the oraching system, you have the outbox where you record the messages.  Basically you have an id.  You have the payload.  You have a couple of timestamps when the message was created, when was it processed or sent  out from the system.  Then you do the oncoming send and so on.  On the receiving side, all of the messages from the outbox of the other system, you  store them in an inbox.  And you should have a unique constraint on the inbox so that the origin's id, you don't  process the same message multiple times.  So doing a de-douplication by having a post-create-square unique constraint, or, well, database unique  constraint, doesn't need to be post-create-square.  Basically you do reconciliation checks with the inbox and outbox.  You check, okay, in the last hour, how many messages did I send from system A?  How many messages do I have in the outbox?  And in the receiving system, you check, okay, how many messages there are in the inbox?  And do the count-match, if they don't, you can react fast.  Also you can easily, if some message is lost somewhere, even with this setup, you can,  from the outbox, you can just have an admin view and click some message that, okay, recent  this.  Okay, how do you actually transport the messages?  For reliability, it turns out it's not important.  You can use HTTP with REST.  So now the simple setup is that you have one original system, you have one receiver system,  and you send the messages just using the standard requests library and then REST framework on  the other side.  It works pretty well.  Of course you can have these cases where you overload the other system, but it's fine.  You have this free trying logic, so it will be eventually sent over to the other system.  But if you want to do, for example, this PubSub type of thing, it's really nice to use something  in between that does the PubSub, so you, from the origin, you still have the outbox.  From the outbox on-commit, you send to Kafka.  Kafka does it thing.  It can do things like check which receivers are allowed to see which messages.  You kind of get the rate limiting because the receivers can play the log as fast as  they can, but if they fall behind, it's fine.  They can just keep on doing their stuff until they catch up.  And of course the receivers over here, they can be different systems.  And so you get through PubSub, so you can do stuff like, okay, I got to payment in at  the same time, send a push notification to the customer.  At the same time, try to do some machine learning.  Use some machine learning based approach on the categorization so we can check that, okay,  how to push this message to bookkeeping.  And so you can do the true PubSub with Kafka.  So transport using Kafka, it's important or it's very useful when you are trying to get  these benefits, but it's not actually that important for reliability.  And finally, of course, you have the outbox inbox model over here still, so you can do  the reconciliation of, okay, I have this message in actually being sent.  So this was mostly about reliability considerations of messaging.  So now you have a messaging setup that works really well.  It turns out as you very likely know that the most likely reason for error is that somebody  makes programming mistakes.  So what we are doing is that we have this, of course, we are using pretty standard,  uh, practices for developing our software.  So we have this test-driven development approach.  We have right test, we have continuous interactions, we do reviews in GitHub, so everything that  goes in, it's actually from compliance point of view also important that you review the  code so everything needs to be checked by somebody else.  We react quickly to failures, so this is actually pretty important from the customer's point  of view that if a payment is delayed for a couple of hours, it's fine if we react to  it.  If the customer needs to call us, you have a problem in this customer trust in all of  it.  And of course, you get to do a lot of more work because then you need to do the customer  communications that, okay, yes, this payment was lost, we fixed it by doing something and  now it's fine.  So it costs you a lot of money.  Always fix the original reason.  So if you find that, hey, something strange is happening, try to find out what's the original  reason.  Do not just check that, okay, I made the payment now go through, I just clicked somewhere that,  okay, try this.  Try to find out what's wrong in the system because when you are scaling up, you want  to get more and more reliability.  Otherwise it's going to be really, really painful if you go for the end price level.  And we are using monitoring and reconciliation.  I said a bit about reconciliation of the inbox-outbox model, but there is also reconciliation you  can do on kind of a business logical level.  So you check in one system that how much money you have moved out, you check in other systems  that do they match, do they amounts match.  If you do this constantly, you will quickly see the cases where you have these error cases  and you can fix them quickly without needing to wait for customers to call you.  In many cases it happens actually if the customers call you, it takes a lot of time to react  because I have seen a couple of cases where it has taken more than a week than when you  get the initial call that something is broken.  We lose one payment in 10,000 say.  Something is broken and then you let it run for a while and you lose more payments and  then it gets really painful to fix the data.  The data is already in the accounting of the company and you need to somehow then fix past  data which is something you really don't want to change the account statement of your bank  account or the payment account.  If you need to do that, it's going to cost you a lot in customer trust.  For monitoring we are also using data docs on the infrastructure layer that works really  well for us.  So we get nice alerts when we have these error cases.  Usually you actually want to check this error rate instead of individual errors when you  have a bit more volume because it's kind of assumed that some of the cases are going to  fail once and then you retry and then they go through.  That was pretty much it.  One thing I could say a bit more about is this sophistication.  What I said in the startup slide is that when you have a small system, you don't need to  have this reliability yet built in.  But if you aim to scale or if you are now scaling, you will face this problem at some  point.  If you have these messages, if you're working with money, you can't lose.  If you are working with messages, you can't lose.  In that case, I really would recommend that you use this kind of setup where you have  actual reliability on the messaging setup.  So that was it.  Thank you.  I'd like to thank the organizers.  This has been a really nice example of how to set up an inclusive, professional, well-organized  event while still keeping it relaxed.  Thank you very much.  Thank you, Ansi.  We got some time for questions before we go into lunch.  So about the inbox-outbox model, can you talk a little bit about that in the sense of are  there are two completely database clusters also?  Are the systems separated for security reasons?  Because you could as well just have the inbox-outbox model in one database or even the same table  to do the reconciliation.  If you have these two different systems, of course, then you need to have them in two  different databases also.  If you are using two different databases, we are actually doing, we have kind of a large  monolith code base right now, but we are doing messaging from the same code base back to  the same code base.  So even in that setup, it's good to have this inbox-different model, outbox-different  model where you can, the outbox records a bit different data.  So in the outbox, you record that, OK, the inbox message ID was this.  I received it at this time.  The inbox origin, kind of inbox create time was this.  In the outbox, you record basically just the data and when you sent it, when you created  it, so then you can reconciliate on the create time of the outbox and inbox.  And even if you are using it inside one system, one code base, I'd go for two different database  models just for the reason of if you want it somewhat split it up, then it's easy to  do.  Also, when you are a small company, I'd go, not go for micro systems, but have one large  code base, but have these messaging approaches, even if you are using a bigger code base.  So that's when you scale, you can split it up if you need to.  Do we have more questions?  Yes, one to the microphone.  Hey, I was wondering whether you considered, instead of this inbox and outbox model, to  use only append-only data structures and keeping pointers and basically following the logs  from one and copying to the other one.  I mean, I'm asking this because I'm involved with a project.  We do 100 million messages per day and it's not okay to lose a single one.  And it's not really hard to do if you emulate the patterns of how Kafka does it internally,  also in your own databases.  You could, depending on the case, you could use Kafka directly.  So instead of recording anything in the database, you first record the message and then you  process it in the database.  So you go kind of moving from Kafka as a transport for messages, you move for Kafka as the data  storage of the messages.  So that's one way.  Then you use Kafka for these pointers and stuff like that.  You could also do that in the database.  I haven't considered how to do that, so not sure how it would work.  The idea over here is that you would use the outbox model and send it to Kafka.  So if you want this 100 million messages per day, so you get it reliably to Kafka.  And then you can use Kafka for the kind of having the pointers and doing the replays  and stuff like that.  My question was because the outbox introduces a mutable intermediate storage, it introduces  also the chance to lose the message.  It's happened only the outbox.  So you will delete after 30 days.  So you just insert new messages over there, and the only update you do is that, okay,  now this has been pushed to somewhere.  But that's the only update.  Okay.  Other more questions?  Going once?  Going twice?  Thank you, Ansi.  Thank you.  Thank you.
DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_0_5099.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_5100_5279.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_5280_5471.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_5472_9539.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_9540_12575.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_12576_12827.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_12828_19475.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_19476_19535.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_19536_22967.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_22968_23111.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_23112_23483.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_23484_24083.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_24084_24323.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_24324_24707.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_24708_25115.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_25116_30779.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_30780_30851.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_30852_30911.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_30912_35147.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_35148_36035.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_36036_36923.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_36924_37727.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_37728_38723.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_38724_39863.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_39864_40151.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_40152_40919.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_40920_41279.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_41280_41399.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_41400_41459.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_41460_41663.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_41664_43487.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_43488_43787.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_43788_43871.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_43872_44255.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_44256_44459.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_44460_44891.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_44892_45071.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_45072_45131.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_45132_45179.png DjangoCon_2018_-_Banking_with_Django_-_how_to_not_lose_your_customer_s_money_45180_45203.png