Distributed locks with Python and Redis
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Buczyński, Sebastian
Sebastian Buczyński - Distributed locks with Python and Redis Traditional methods of coping with concurrent programming problems are well-known and described in literature. Many programming languages, including Python, contain in their standard libraries tools and primitives such as semaphores and can spawn threads or subprocesses. However, in the face of increasing interest in service oriented architecture and building distributed systems, that span across many independent server nodes, emerges a need to adapt traditional solutions, so they can be applied in the new environment. In this talk I will share my experiences gathered during building a modern contact center - highly concurrent system, which requires certain resources to be accessed exclusively by several self-contained components.
 So, well, thank you for coming. Hola, everyone. Did you notice that there are free talks at the same time we've distributed in name? The Arup-item folks have some sense of humor, for sure. Well, first of all, I would like to introduce myself, as it was already did. So I'm a software developer and a maintenance guy at Focus Telecom Poland. My main toys are G-Event, Twisted, and Celery. What we are doing at the company is to providing a business-to-business telecommunications solutions. We host teleconferences, we sell Pebexes, but our main product is a contact center. So, nowadays, you have multiple ways to communicate with each other. There are social media, there are SMS text messages, well, there are emails, and there are phone calls. So a contact center is a way to unify all these, to gain a much better contact with your customer. So throughout this talk, I will show you the principles of implementing distributed logs in Redis using Python and show the usage in a real-life application. So does anyone know, or does anybody doesn't know what Redis is? Okay, there are some hands, so I will quickly refer to that. The Redis is an OSQL database. It's basic key value storage, but the key can be only a string, but the value can be more complicated data structure. For the sake of this talk, most important are string and list. They are pretty much like Python counterparts. And also Redis has a simple support for a message passing. So first of all, I will introduce some case study, then I will show you how to implement correctly distributed binary semaphore, so-called mutex, and later on semaphore. And finally, I will give you some tips, and there will be some time for questions. So first of all, this is a loose definition that I came up with a few days ago. The Gido on his talk yesterday gave a great example of locking and keeping safe the reference counters. The other case would be that we have a short database connection between threads in Python, and we would not like to other thread sending some SQL comments while the other thread was trying to carry out a transaction. So this is basically what lock is. It's to protect a shared resource. So the case study will concern a contact center. Hope nobody expected that. So in the contact center, work people, of course, and we refer to them as agents. So an agent can answer or originate a phone call, email messages, text messages, or carry out chat conversations, and this kind of stuff. Well, and my advantage of contact center is that it automates all of this kind of work. So it keeps the agents busy. So they are not looking at the floor or sailing. Okay. But to be more familiar, I have replaced agent with giant W that stands for a worker. And these are tasks. So you could say that this looks like a task queue. Who doesn't know any task queue solutions like Celery, RQ, Germans? Are there anyone? Who hasn't heard any of them? No. It's good. So basically, you know the concept. And this is the problem we are trying to solve. Yeah, it looks similar. You know the stuff. You can apply a working solution, because it will be great, right? Wrong. There are a couple of things that you have to worry about. First of all, a worker is an autonomous creature. And it can choose whether to work or not. He can simply log out of the system and he will not be available anymore. And this is a normal situation. This is not a failure of system or some kind. You can force somebody to stay logged in and work, at least not in Poland. Well, the tasks must be prioritized. And this is a heavily desired feature from our clients. So yeah, I don't know if any other distributed task queue is like Celery support this. As far as I know, in Celery, you can partially achieve this by routing to different queues. And that's all. So it's also not sufficient. And the last mostly mind-blowing is that one worker can handle multiple tasks simultaneously. And from... I don't mean concurrently, but simultaneously, so in parallel, at least from a point of view of task manager. So this will be viewed that way. Of course, Q1 cannot carry out two phone conversations at the same time. But why not write an email and speak to somebody? That's fine. So our solution depicted as a whole looks for some centralized task manager and its rating using Twisted. And the Redis will serve as a logs server and will hold them all. So the first thing I would like to say is about binary semaphore. It has only two states. It can be locked or unlocked. To lock it, I will refer to this as acquiring and to unlocking as releasing. This is the same as the... In standard Python library. Who has a background with multiprocessing or multithreading library? Yeah, that's quite a lot. And who was on the Monday talk on this? Yeah, that's quite a few things. So... And it will be stored as a Redis string. And all calls to acquire it or release will be non-blocking from Python. So the first use case is about keeping state of a worker. Whether he's busy or idle. So meaning that the lock is released, it's the same as idle and so on. And the lock will be acquired when a task manager will assign a task and it will be released afterwards. So it's over. And the second case is about preventing race conditions. The one I mentioned that a worker would like to log out sometimes and I don't know, go for a break or something. That he can do it using this mechanism because it can acquire a lock before a task manager do. Or it will simply do not release it after finishing its previous task. Okay. So since this is stored in this Redis string, we would use two Redis comments. A set to store a lock, to set a string, and a get to retrieve it. What we need is atomicity. Every single comment of Redis is an atomic one. So no one else can disrupt your data. So the first draft, we have a Chrome disk, something like that. We get in the log, check in if we can lock it actually, and then do it. But there's a problem because between get and set, there is some instruction running and this as a whole is not atomic. So someone else might have changed this in the meantime. This is a problem and it's not acceptable then. So we'll introduce some kind of Redis transactions. This is not a real transaction in a sense of SQL databases, but it allows you to execute a bunch of comments in an atomic way. So we issue the command watch and giving the key name. So this will be carefully watched during the transaction. And finally, if we cannot do anything, then we unwatch it. So this connection will be free of side effects later. The next step will be to introduce the transaction denotions, multi and exec. These actually will denote a transaction or the things that should be done conditionally only if a watched key has not been changed. So I guess that's quite simple stuff. Okay. The important part that task manager sometimes would like to know whether the state changed. When we lock the worker, he will find it eventually, but when we do not, when we are freeing it, it must know when the worker is freed. So we can actually perform some action. For example, look for the next task for it. And to receive the notifications, we issue a subscribe comment. But to be more precise, on the Python side, so this is a task manager, as I stated, it's written in Twisted. There is some kind of magic like ready subscriber will handle the subscribe and actions for them for you. And this is the event driven. You see the very ugly method name, message received, which is written camel case. By overloading this method, you actually define the behavior for this. So the second part will be about distributed semaphore. So the semaphore is different from lock in that way, that it maintains an internal counter. So it means that it can be released or acquired many times. Or if we define a semaphore with counter two, we can acquire it twice and also release it twice. But the original counter cannot be exceeded by further calls. And this will be stored as a list and the calls to it will be blocking. So the use case is that we can handle multiple tasks at the same time. As I stated before, the usual lock is not sufficient and the use of a bunch of binary semaphores would not be sufficient, too. So this is similar to string operations, but now we are using our push, which stands for right push, so it will append an element to a list. And the error pop, which stands for blocking right pop, so it will pop an element from a list. And our push will serve as a release operation and the blocking right pop as an acquiring operation. So when there's nothing there, and it means that semaphore has been acquired a maximum number of times, the command will block until there's something. Something there. So in the Python part, we do not change anything because it reacts only on events. Eventually you can implement your checking whether the semaphore is free or not. And there are cases when you might need a state of semaphore after changing it. You could use previously showed a multi-exec block, but that's not always the case. And alternative is to write a simple Lua script. You see that it's very much like Python in this case, except this word local, so it means it's allocating a local variable, so it will be garbage collected after this block. But this script only releases our log and return length that we have left. Okay. But there's a warning. Multi-exec blocks and Lua scripts all are atomic in Redis. The problem is that blocking invocations in atomic way has not sense this time. So it would mean that the Redis instance has to be blocked forever to serve this one. So making BurrPOP inside multi-exec will return you a new, so it's none in Python client. And invoking such a Lua script will, Redis will respond with an error. Alternatively you can use an air pop. So it's non-blocking feature. It will return you either nil if there's, if lock cannot be acquired or a value if it has. Some final remarks before you rush and implement these solutions in your software. Please care about starting conditions. Make sure that the logs are properly instantiated before using them. And study carefully control flow in your application to not introduce too many logs or too less. And also please work up a restoring state procedure. When something goes wrong, there might be cases when the state in Redis is not finally consistent. So you might want to fix that some way. For example, we persist a state in SQL database as some kind of history of activity of given worker. So we can restore a state properly after some kind of failure. So well, that's all for me. And now it's time for some questions if you have some, of course. Thank you. Hello. Have you considered using set nx for the log, set nx command? You mean set nx, right? Yes. Yes. For log. Yes. I'm not quite sure about this, and this is described also on Redis documentation as far as I know. But well, this is an expiring comment as far as I know, yeah? You can use it as a non expiring comment. You don't need to set an expiring. Yeah. But somehow it didn't fit our needs. I can't remember now why, but now it wasn't sufficient for this kind of locking. Yeah, but that's an alternative for sure. Okay. Have you tried to look at other implementations besides the built-in lock implementation in Redis client? You mean besides Redis, right? Yeah, besides the official Redis client. Well, ideally it would be to write your own locks server in Python, but the case is that some code base we have and the most is written in PHP, so it has to be something outside. Ah, I see, okay. Hi. When you're talking about distributed semaphores and locks, are we talking about, so are the locks in one location, so in one Redis instance, and the workers can be distributed, or are the locks in semaphores actually distributed? Yeah, that's a good question, and I hope that someone asked that. Well, basically we use a single Redis instance for serving our locks, due to different limitations, we do not need more resources, but distribution you can achieve is by either sharding Redis instances, and this is done automatically by using Redis cluster starting from Redis 3.0, or replication, which is also in this kind. But when I prepared this, I meant distribution by, yeah, the workers are distributed, and these are different kind of resources that might achieve a lock, sorry, acquire. Does this answer your question? Okay. Any more questions? Oh, wait. Have you tried, have you thought about using Zookeeper? Because I'm not entirely familiar with it, but my colleagues at work use it, and it's supposedly that, which is synchronizing few distributed instances of anything to only get one resource at a time. Well, we didn't, and I have read about something like Zookeeper just yesterday on OREALY stand, but I will surely consider that. So thanks for suggestion, but no, I don't have any experience with that. Me too, but it seems like a good idea. Yeah, if it would be simpler or better in some case, why not replace? Because it's used supposedly to select the name node in a Hadoop cluster, for example. So this is also a really distributed system, and only you have to think about it and synchronize it, so Zookeeper is supposedly for that. Yeah, I will investigate this for sure, and if I came up with something interesting, I will surely give a talk at next occasion, not this year, I think. Okay. Thank you. Any more questions? Well, I do have one there. Have you actually ever seen anyone taking on a phone and writing an email at the same time? Yeah, myself. Do you manage to actually make sense? Well, you know, there are some studies that shows that a human that is carrying out a few simultaneous tasks is completely reduced by its brain capabilities. So there was a comparison in the book I read that he behaves as if he had a brain of a six years old child. So, you know, I don't find personally that multiple tasks at the same time are a super killing feature, but that's what the business wants. Yeah, yeah, because they respond with some delay. Maybe. So any more questions? I guess we could then. Thank you again, Sebastian.
 So, well, thank you for coming.  Hola, everyone.  Did you notice that there are free talks at the same time we've distributed in name?  The Arup-item folks have some sense of humor, for sure.  Well, first of all, I would like to introduce myself, as it was already did.  So I'm a software developer and a maintenance guy at Focus Telecom Poland.  My main toys are G-Event, Twisted, and Celery.  What we are doing at the company is to providing a business-to-business telecommunications solutions.  We host teleconferences, we sell Pebexes, but our main product is a contact center.  So, nowadays, you have multiple ways to communicate with each other.  There are social media, there are SMS text messages, well, there are emails, and there  are phone calls.  So a contact center is a way to unify all these, to gain a much better contact with your customer.  So throughout this talk, I will show you the principles of implementing distributed logs  in Redis using Python and show the usage in a real-life application.  So does anyone know, or does anybody doesn't know what Redis is?  Okay, there are some hands, so I will quickly refer to that.  The Redis is an OSQL database.  It's basic key value storage, but the key can be only a string, but the value can be  more complicated data structure.  For the sake of this talk, most important are string and list.  They are pretty much like Python counterparts.  And also Redis has a simple support for a message passing.  So first of all, I will introduce some case study, then I will show you how to implement  correctly distributed binary semaphore, so-called mutex, and later on semaphore.  And finally, I will give you some tips, and there will be some time for questions.  So first of all, this is a loose definition that I came up with a few days ago.  The Gido on his talk yesterday gave a great example of locking and keeping safe the reference  counters.  The other case would be that we have a short database connection between threads in Python,  and we would not like to other thread sending some SQL comments while the other thread was  trying to carry out a transaction.  So this is basically what lock is.  It's to protect a shared resource.  So the case study will concern a contact center.  Hope nobody expected that.  So in the contact center, work people, of course, and we refer to them as agents.  So an agent can answer or originate a phone call, email messages, text messages, or carry  out chat conversations, and this kind of stuff.  Well, and my advantage of contact center is that it automates all of this kind of work.  So it keeps the agents busy.  So they are not looking at the floor or sailing.  Okay.  But to be more familiar, I have replaced agent with giant W that stands for a worker.  And these are tasks.  So you could say that this looks like a task queue.  Who doesn't know any task queue solutions like Celery, RQ, Germans?  Are there anyone?  Who hasn't heard any of them?  No.  It's good.  So basically, you know the concept.  And this is the problem we are trying to solve.  Yeah, it looks similar.  You know the stuff.  You can apply a working solution, because it will be great, right?  Wrong.  There are a couple of things that you have to worry about.  First of all, a worker is an autonomous creature.  And it can choose whether to work or not.  He can simply log out of the system and he will not be available anymore.  And this is a normal situation.  This is not a failure of system or some kind.  You can force somebody to stay logged in and work, at least not in Poland.  Well, the tasks must be prioritized.  And this is a heavily desired feature from our clients.  So yeah, I don't know if any other distributed task queue is like Celery support this.  As far as I know, in Celery, you can partially achieve this by routing to different queues.  And that's all.  So it's also not sufficient.  And the last mostly mind-blowing is that one worker can handle multiple tasks simultaneously.  And from...  I don't mean concurrently, but simultaneously, so in parallel, at least from a point of view  of task manager.  So this will be viewed that way.  Of course, Q1 cannot carry out two phone conversations at the same time.  But why not write an email and speak to somebody?  That's fine.  So our solution depicted as a whole looks for some centralized task manager and its  rating using Twisted.  And the Redis will serve as a logs server and will hold them all.  So the first thing I would like to say is about binary semaphore.  It has only two states.  It can be locked or unlocked.  To lock it, I will refer to this as acquiring and to unlocking as releasing.  This is the same as the...  In standard Python library.  Who has a background with multiprocessing or multithreading library?  Yeah, that's quite a lot.  And who was on the Monday talk on this?  Yeah, that's quite a few things.  So...  And it will be stored as a Redis string.  And all calls to acquire it or release will be non-blocking from Python.  So the first use case is about keeping state of a worker.  Whether he's busy or idle.  So meaning that the lock is released, it's the same as idle and so on.  And the lock will be acquired when a task manager will assign a task and it will be  released afterwards.  So it's over.  And the second case is about preventing race conditions.  The one I mentioned that a worker would like to log out sometimes and I don't know, go  for a break or something.  That he can do it using this mechanism because it can acquire a lock before a task manager  do.  Or it will simply do not release it after finishing its previous task.  Okay.  So since this is stored in this Redis string, we would use two Redis comments.  A set to store a lock, to set a string, and a get to retrieve it.  What we need is atomicity.  Every single comment of Redis is an atomic one.  So no one else can disrupt your data.  So the first draft, we have a Chrome disk, something like that.  We get in the log, check in if we can lock it actually, and then do it.  But there's a problem because between get and set, there is some instruction running  and this as a whole is not atomic.  So someone else might have changed this in the meantime.  This is a problem and it's not acceptable then.  So we'll introduce some kind of Redis transactions.  This is not a real transaction in a sense of SQL databases, but it allows you to execute  a bunch of comments in an atomic way.  So we issue the command watch and giving the key name.  So this will be carefully watched during the transaction.  And finally, if we cannot do anything, then we unwatch it.  So this connection will be free of side effects later.  The next step will be to introduce the transaction denotions, multi and exec.  These actually will denote a transaction or the things that should be done conditionally  only if a watched key has not been changed.  So I guess that's quite simple stuff.  Okay.  The important part that task manager sometimes would like to know whether the state changed.  When we lock the worker, he will find it eventually, but when we do not, when we are freeing it,  it must know when the worker is freed.  So we can actually perform some action.  For example, look for the next task for it.  And to receive the notifications, we issue a subscribe comment.  But to be more precise, on the Python side, so this is a task manager, as I stated, it's  written in Twisted.  There is some kind of magic like ready subscriber will handle the subscribe and actions for  them for you.  And this is the event driven.  You see the very ugly method name, message received, which is written camel case.  By overloading this method, you actually define the behavior for this.  So the second part will be about distributed semaphore.  So the semaphore is different from lock in that way, that it maintains an internal counter.  So it means that it can be released or acquired many times.  Or if we define a semaphore with counter two, we can acquire it twice and also release it  twice.  But the original counter cannot be exceeded by further calls.  And this will be stored as a list and the calls to it will be blocking.  So the use case is that we can handle multiple tasks at the same time.  As I stated before, the usual lock is not sufficient and the use of a bunch of binary  semaphores would not be sufficient, too.  So this is similar to string operations, but now we are using our push, which stands for  right push, so it will append an element to a list.  And the error pop, which stands for blocking right pop, so it will pop an element from  a list.  And our push will serve as a release operation and the blocking right pop as an acquiring  operation.  So when there's nothing there, and it means that semaphore has been acquired a maximum  number of times, the command will block until there's something.  Something there.  So in the Python part, we do not change anything because it reacts only on events.  Eventually you can implement your checking whether the semaphore is free or not.  And there are cases when you might need a state of semaphore after changing it.  You could use previously showed a multi-exec block, but that's not always the case.  And alternative is to write a simple Lua script.  You see that it's very much like Python in this case, except this word local, so it means  it's allocating a local variable, so it will be garbage collected after this block.  But this script only releases our log and return length that we have left.  Okay.  But there's a warning.  Multi-exec blocks and Lua scripts all are atomic in Redis.  The problem is that blocking invocations in atomic way has not sense this time.  So it would mean that the Redis instance has to be blocked forever to serve this one.  So making BurrPOP inside multi-exec will return you a new, so it's none in Python client.  And invoking such a Lua script will, Redis will respond with an error.  Alternatively you can use an air pop.  So it's non-blocking feature.  It will return you either nil if there's, if lock cannot be acquired or a value if it  has.  Some final remarks before you rush and implement these solutions in your software.  Please care about starting conditions.  Make sure that the logs are properly instantiated before using them.  And study carefully control flow in your application to not introduce too many logs or too less.  And also please work up a restoring state procedure.  When something goes wrong, there might be cases when the state in Redis is not finally  consistent.  So you might want to fix that some way.  For example, we persist a state in SQL database as some kind of history of activity of given  worker.  So we can restore a state properly after some kind of failure.  So well, that's all for me.  And now it's time for some questions if you have some, of course.  Thank you.  Hello.  Have you considered using set nx for the log, set nx command?  You mean set nx, right?  Yes.  Yes.  For log.  Yes.  I'm not quite sure about this, and this is described also on Redis documentation as far  as I know.  But well, this is an expiring comment as far as I know, yeah?  You can use it as a non expiring comment.  You don't need to set an expiring.  Yeah.  But somehow it didn't fit our needs.  I can't remember now why, but now it wasn't sufficient for this kind of locking.  Yeah, but that's an alternative for sure.  Okay.  Have you tried to look at other implementations besides the built-in lock implementation in  Redis client?  You mean besides Redis, right?  Yeah, besides the official Redis client.  Well, ideally it would be to write your own locks server in Python, but the case is that  some code base we have and the most is written in PHP, so it has to be something outside.  Ah, I see, okay.  Hi.  When you're talking about distributed semaphores and locks, are we talking about, so are the  locks in one location, so in one Redis instance, and the workers can be distributed, or are  the locks in semaphores actually distributed?  Yeah, that's a good question, and I hope that someone asked that.  Well, basically we use a single Redis instance for serving our locks, due to different limitations,  we do not need more resources, but distribution you can achieve is by either sharding Redis  instances, and this is done automatically by using Redis cluster starting from Redis  3.0, or replication, which is also in this kind.  But when I prepared this, I meant distribution by, yeah, the workers are distributed, and  these are different kind of resources that might achieve a lock, sorry, acquire.  Does this answer your question?  Okay.  Any more questions?  Oh, wait.  Have you tried, have you thought about using Zookeeper?  Because I'm not entirely familiar with it, but my colleagues at work use it, and it's  supposedly that, which is synchronizing few distributed instances of anything to only get  one resource at a time.  Well, we didn't, and I have read about something like Zookeeper just yesterday on OREALY stand,  but I will surely consider that.  So thanks for suggestion, but no, I don't have any experience with that.  Me too, but it seems like a good idea.  Yeah, if it would be simpler or better in some case, why not replace?  Because it's used supposedly to select the name node in a Hadoop cluster, for example.  So this is also a really distributed system, and only you have to think about it and synchronize  it, so Zookeeper is supposedly for that.  Yeah, I will investigate this for sure, and if I came up with something interesting, I  will surely give a talk at next occasion, not this year, I think.  Okay.  Thank you.  Any more questions?  Well, I do have one there.  Have you actually ever seen anyone taking on a phone and writing an email at the same  time?  Yeah, myself.  Do you manage to actually make sense?  Well, you know, there are some studies that shows that a human that is carrying out a  few simultaneous tasks is completely reduced by its brain capabilities.  So there was a comparison in the book I read that he behaves as if he had a brain of a  six years old child.  So, you know, I don't find personally that multiple tasks at the same time are a super  killing feature, but that's what the business wants.  Yeah, yeah, because they respond with some delay.  Maybe.  So any more questions?  I guess we could then.  Thank you again, Sebastian.
Sebastian_Buczynski_-_Distributed_locks_with_Python_and_Redis_0_13271.png Sebastian_Buczynski_-_Distributed_locks_with_Python_and_Redis_13272_33731.png