Use Python to process 12mil events per minute and still keep it simple (Talk)
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Dima, Teodor
Teodor Dima - Use Python to process 12mil events per minute and still keep it simple (Talk) Creating a large-scale event processing system can be a daunting task. Especially if you want it “stupid simple” and wrapped around each client’s needs. We built a straightforward solution for this using Python 3 and other open-source tools. Main issues to solve for a system that needs to be both performant and scalable: - handling a throughput of 1 million events per minute in a 4 cores AWS instance; - following the principle of least astonishment; - data aggregation and how Python's standard libraries and data structures can help; - failsafe and profiling mechanisms that can be applied to any Linux service in production; - addressing unexpected behaviors of Python’s Standard Library; like reading from a file while it is written; - tackling a sudden spectacular cloud instance failure; The alternative to this system would be to adopt existing technology stacks that might be too general, add more complexity, bloat, costs and which need extensive work to solve your specific problem. Moreover, our approach resulted in over 85% drop on hardware utilisation.
 So, hi, my name is Tau Dordimo. I am a developer at East Vision Systems, a company from Romania. And we do a lot of things there, but we mainly concern ourselves with ad tech, big data, video technologies and real-time bidding. For those that do not know what that means, this means that we must handle a lot of user events every second, which must be continuously persisted into the database. So what is an event? An event is represented by a simple get request, which is sent by a user from a browser, which views advertising. And that is sent to the event processing system that must analyze the data and then push it into the database. The amount of data that can be generated by a single user is quite big. A single user viewing a single ad can send about, at least, about 10 events of data. These events are not sent all at the same time, which means that the user must keep a continuous connection with the server. And that means that you have a lot of users, which send a lot of events from a lot of concurrent clients. This problem has been seen long before this talk. It is called C10K. And it basically means how can you handle 10,000 concurrent connections on the same machine? During my time in the company, we had averages of about 12 million requests per minute. That means about 200,000 requests per second that must be handled by a single server cluster. So the question is, how can you handle such a traffic? And moreover, how can you handle such a traffic by using as little resources as possible? So the main solution would be to implement a system that responds to each request and sends it directly to the database. This works for a small amount of data and is quite easy and fast to implement, but it's ultimately unmaintainable and consumes a lot of resources. Another alternative solution would be to use the Apache Trio, Kafka, Storm, and ZooKeeper, or some alternatives to that. However, if you want to configure and to tune the systems as good as you can into a coherent whole, that takes time. And it's often non-Pythonic, although some work has been put to make it a lot more Pythonic by the ones at the people at Parsley. So could those to them? Initially, when we had to implement such a system, we had to ship it. And that meant that we implemented a simple solution, the naive one, which handled streams of events and then simply sent them to the database. In order to check for data consistency, though, in order to be sure that we haven't dropped any event that reached the server but was not inserted into the database, we checked the access logs of the web server and we checked if data data corresponds with the data from the database. Because all the data that you need, all the events are there in the access log. So this led to a simple idea. Why not use the access log as a simple queue for an event processing system? The idea was that when the request reached the machine, they are received by our system by the EngineX web server. The EngineX web server solved the Ctnk problem, so that would solve a lot of problems. So between the access logs and the database, there had to be another service which would take the data, analyze it, transform it, and then push it into the database. So we began to think about the implementations of such a project and if it would be resilient and feasible enough for us to do it. After some prototypes and some new ideas, we came up with a clean structure and we came up with a service that we called LogBunker. Now this is a data flow diagram which shows a simplified scheme of the data flow through a single virtual machine in the cloud. As I said, a single HTML request is sent by the user, by a browser, through a load balancer and then to the EngineX web server. In order to ensure that the data was easily swappable between the virtual machines, we used the Amazon web server's EBS, Elastic Block Store service to store the access log data. Then the access log was read by our service by LogBunker and then it is inserted into the database. Inside the LogBunker service, there are actually three processes that work at the same time. So, two of these processes are the parser and the absorber. The parser reads the access log continuously, just like a tail, a unique tail, and then analyzes these events and caches them into an internal cache structure. This internal cache structure is using the Python standard types. They're actually quite beautiful to use and very simple to use. Whether it has cached this data for a fixed period of seconds, configurable, of course, it then takes this data and pushes it into a multiprocessing queue that makes the connection with the absorber. The absorber then pops the queue, takes the data from that, and then pushes it into the database. Besides pushing it into the database, the absorber actually logs into a special file, we call it the bin log file. Every event that it has actually pushed into the database and with the offset of the access log so that in the event of a crash, in the event that you actually want to reboot the system to restart it, the service would then know which point was the last point that was introduced into the database and could restart from that point on. The third and final process is the admin process. This process checks periodically if the other two processes are active. If they are not, it shuts down the whole system. Now why did we do that? Why didn't we just reboot the whole service? Well there is a tiny probability that if you have something that can crash a process, that can crash the data processing service, then you may have corrupted data. If you have data that does not reach the database but it is there somewhere in the system, then that's bad. But if you insert corrupt data into the database, then that's really much worse. So we try to avoid that at all costs. Another function of the admin process is to fsync the data files, the access file, and the bin log file. The fsync function is a function which synchronizes the virtual memory of a file with the actual disk content of that file, which means that the file is persisted after you make the fsync call. That is extremely important if you want data persistence. The last important function is to set as data at the configurable port. That was done with a single protocol. It just accepts every single request that comes and then serves the JSON status data. This status data is collected from the other two processes through some shared virtual memory that is controlled by a simple read-write log. So one of the first issues that we thought about was how stable would the service be when it stays the access log? So it turns out it's very easy and completely stable. It's very easy to implement. But there is one tiny problem. The offsets are hard to calculate efficiently. And like I said, we need those offsets to put them in the bin log file. So in order to do that, using buffered text files is almost useless because if you use buffered text files, you cannot know the exact offset of a single line of text. Unbuffered text files are really slow and could not help us. So it's actually easier to open the file in byte mode. Just read a number of bytes, add a number of bytes that are contained in a line, and there you go. You have the offsets. So between the parser and the absorber process, we have a multiprocessing queue. The queue is using a pipe in the background, a Unix pipe, and that pipe never corrupted data. We never had problems with it. But there is a problem with the data transfer speed. Because when you actually insert some data into the queue, it actually spawns a thread which then inserts the data gradually from the buffer, its internal buffer, into the pipe. If this thread does not have the gil, you are going to have data in the buffer which is not actually inserted in the pipe. So the connection between the parser and the absorber is broken. There are ways to minimize the damages, and I will talk about them in a later slide. So how could the catastrophic crash be handled securely and efficiently? Where if that happens, if you have a catastrophic crash, then there are two files, essential files, which can be corrupted or incomplete, the access log and the bin log. And you can manage that somewhat by just syncing them as often as you can. So then if you restart the machine or if you have a crash, then you can recover the data. You can just go on and then insert the data that you have there. So is Python fast enough to ingest this data? Well, yes. Actually with some performance optimizations, see Python could get about 20,000 requests per second on the same machine on a C4X large Amazon Web Service virtual machine. That means that you have this processing power on a machine with four virtual cores and eight gigabytes of RAM. However, after we inserted it into production, we re-implemented the data processing system in CIFA. That was quite easy to do. It just took a week, excluding testing, and without any prior CIFN experience, and it's doubled the performance. So CIFN way to go. There is another problem, though. Like I said, we called Fsync on the two essential data files. And we use the network file storage offered by AWS. The Fsync does affect the performance of the read from the files. And we had a problem at some point that's periodically, once every two days or so, the network lags in the Amazon Web Service. And that meant that the Fsync call took from beneath 0.1 seconds to almost 25 seconds, which meant that the service was blocked at that time. But it was actually pretty rare, and it did not affect the system too much. So during one of the testing phases, we observed a strange behavior with the Absurter. The parser was reading continuously from the access log file. It cached the data happily. The cache was full. And the Absurter was not having the same efficiency as it would have under normal conditions. So what happened there was that the parser thread was starving the feeder thread, starving the thread that had the purpose of taking the data and then inserted into the pipe. The only way we could manage until now to avoid this problem was to force the parser thread to sleep periodically. It could not be fully fixed, and it's just solved our problem right there. So how can you maintain, how easy is to maintain such a system? A single machine that answers to this amount of requests, like I said, 20,000 requests per second, contains only two essential processes, the engine service, which takes the requests and just needs a configuration file. The Python Demon log bunker, which also needs a single configuration file. Those machines are small. They only contain four cores, and they are quite cheap. They are commodity hardware. And you can serve with a server cluster 12 million requests per minute by just using 15 virtual machines. That means that there is no single point of failure. When you have machines that are this small, you have enough processing power to obtain such an improvement. And if you lose a machine, then the system will not be affected as much as, let's say, if you lose a gigantic 32 cores of a machine, with, God knows how many gigabytes of RAM. Another thing to say about maintenance is that these machines are not throttled to 100%. These machines are throttled at a maximum of 50% in a normal situation. Let's say 75% on peaks. This reduces the possibility of availability and hardware failure, because if you throttle a machine at 100%, then you will have some hardware failures. And even if the peaks reach 100% of system capacity, then you have an event queue that is represented by the access log file. That access log file will just keep it be appended to by the NGINX web server. The NGINX web server will happily continue to serve the requests. And if the Python process does not, does not, cannot pace with the NGINX web server, it doesn't really have to. It will have a delay from the point when the data reaches the virtual machine to when it is inserted into the database by about, let's say, a couple of minutes. But it will not be critical. You will never lose data. You will just continue pushing it into the database when the peak is gone. And the same thing applies to the database connection. The database can actually lag quite much. And if you have, at some point in time, some problems with the database, let's say that you have a cluster of database servers and one of them or several of them crash, then you will have less right performance on them. And if the database connection lags, then the absorber will just lag in getting the data from the queue. But what that means is that the data will actually add itself into the cache. The cache will just add the data more and more and more. And after the database is restored to its full power, it will just continue to insert it into the database and it will just reach its 100%. Okay. So that was it. Thank you for your attention. There are questions, please. Hi. One question, well, actually I have probably two, but the first one is why between your parser and your UPSetter, you don't use RabitMQ or any message queuing like, I don't know, 0MQ even would solve your problem of queue that you have in the multiprocessing. Between the parser and the absorber. Yeah. Well, firstly, because it was easier to implement and because it didn't not affect the performance as much as we thought. So even with that performance penalty, we still reached 20,000 requests per second. 20,000 requests, the limit of the NGINX web server in an Amazon C4X large instance is about 22, 24 requests per second. And we did not reach that limit. And like I said, we kept our servers to serve about 12,000 requests per second so that we kept low the possibility of a harder failure. So we could have implemented a message queuing service. We thought about implementing something that we read this and we may implement that in the future. We don't know, but for now it's sitting quite a bit. Well, just 0MQ is five lines and it solves your problem. That's it. Oh, okay. Thanks. Hey, I just want to ask about, you talked about how even if your log bunker process crashes or doesn't work, you're not going to lose any logs because they're stored in the access log. Yes. So you have NGINX on each virtual machine. Yes. So if you have a virtual machine crash or go down, what happens to the log messages that are in the access log on that machine which have yet to be stored in your database? They're gone. They're not gone forever because they are kept in the access log and that access log will be kept on the EBS network partition. So what that means is that temporarily while the virtual machine is down and you cannot have access to the EBS network partition, you will have some data that does not reach the database. But if your support team is ready, then it will reconnect to that EBS system to another log bunker instance maybe or another tool and just reinsert it. Thanks. I knew I had a second question. Why are you not using things like log stash or equivalent to do your parsing? Log stash between what components? Basically to parse your logs. What you do manually by re-implementing what those utilities do is just pick up the information directly from the logs via dedicated utilities. I mean, there are many already existing solutions to do that precise job, I think. I mean, I'm not expert in log stash, but I've heard many things about logging and log parsing and whatnot. To me, it seems that log stash or something equivalent would be a very good solution to extract information, meaningful information from your logs because it's structured information and probably you can inform log stash to read how to read text basically. So that means that we'll have a cluster of servers which have the engine X on each one of the machines. But it's the same thing. It's just you read your access log somehow and just instead of re-implementing something in Python, you use a standard utility that is well maintained by professionals. Yes, well, that means that it will be redirected to another cluster of servers which actually has log stash on them. Right? Well, you can install, I guess, log stash on the same machines as where you run your log bunker. Yes, that is true. However, I'm not really sure how is the performance on those services. We actually could implement this really quickly and had really good performance. And we thought that we could use this. So I don't know, ultimately, I don't know how the performance would be. Thank you. Any other questions? One more question? Okay. Thank you, Teyala. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you.
 So, hi, my name is Tau Dordimo.  I am a developer at East Vision Systems, a company from Romania.  And we do a lot of things there, but we mainly concern ourselves with ad tech, big data,  video technologies and real-time bidding.  For those that do not know what that means, this means that we must handle a lot of user  events every second, which must be continuously persisted into the database.  So what is an event?  An event is represented by a simple get request, which is sent by a user from a browser, which  views advertising.  And that is sent to the event processing system that must analyze the data and then push it  into the database.  The amount of data that can be generated by a single user is quite big.  A single user viewing a single ad can send about, at least, about 10 events of data.  These events are not sent all at the same time, which means that the user must keep  a continuous connection with the server.  And that means that you have a lot of users, which send a lot of events from a lot of concurrent  clients.  This problem has been seen long before this talk.  It is called C10K.  And it basically means how can you handle 10,000 concurrent connections on the same machine?  During my time in the company, we had averages of about 12 million requests per minute.  That means about 200,000 requests per second that must be handled by a single server cluster.  So the question is, how can you handle such a traffic?  And moreover, how can you handle such a traffic by using as little resources as possible?  So the main solution would be to implement a system that responds to each request and  sends it directly to the database.  This works for a small amount of data and is quite easy and fast to implement, but it's  ultimately unmaintainable and consumes a lot of resources.  Another alternative solution would be to use the Apache Trio, Kafka, Storm, and ZooKeeper,  or some alternatives to that.  However, if you want to configure and to tune the systems as good as you can into a coherent  whole, that takes time.  And it's often non-Pythonic, although some work has been put to make it a lot more Pythonic  by the ones at the people at Parsley.  So could those to them?  Initially, when we had to implement such a system, we had to ship it.  And that meant that we implemented a simple solution, the naive one, which handled streams  of events and then simply sent them to the database.  In order to check for data consistency, though, in order to be sure that we haven't dropped  any event that reached the server but was not inserted into the database, we checked  the access logs of the web server and we checked if data data corresponds with the  data from the database.  Because all the data that you need, all the events are there in the access log.  So this led to a simple idea.  Why not use the access log as a simple queue for an event processing system?  The idea was that when the request reached the machine, they are received by our system  by the EngineX web server.  The EngineX web server solved the Ctnk problem, so that would solve a lot of problems.  So between the access logs and the database, there had to be another service which would  take the data, analyze it, transform it, and then push it into the database.  So we began to think about the implementations of such a project and if it would be resilient  and feasible enough for us to do it.  After some prototypes and some new ideas, we came up with a clean structure and we  came up with a service that we called LogBunker.  Now this is a data flow diagram which shows a simplified scheme of the data flow through  a single virtual machine in the cloud.  As I said, a single HTML request is sent by the user, by a browser, through a load balancer  and then to the EngineX web server.  In order to ensure that the data was easily swappable between the virtual machines, we  used the Amazon web server's EBS, Elastic Block Store service to store the access log  data.  Then the access log was read by our service by LogBunker and then it is inserted into  the database.  Inside the LogBunker service, there are actually three processes that work at the same time.  So, two of these processes are the parser and the absorber.  The parser reads the access log continuously, just like a tail, a unique tail, and then  analyzes these events and caches them into an internal cache structure.  This internal cache structure is using the Python standard types.  They're actually quite beautiful to use and very simple to use.  Whether it has cached this data for a fixed period of seconds, configurable, of course,  it then takes this data and pushes it into a multiprocessing queue that makes the connection  with the absorber.  The absorber then pops the queue, takes the data from that, and then pushes it into the  database.  Besides pushing it into the database, the absorber actually logs into a special file,  we call it the bin log file.  Every event that it has actually pushed into the database and with the offset of the access  log so that in the event of a crash, in the event that you actually want to reboot the  system to restart it, the service would then know which point was the last point that was  introduced into the database and could restart from that point on.  The third and final process is the admin process.  This process checks periodically if the other two processes are active.  If they are not, it shuts down the whole system.  Now why did we do that?  Why didn't we just reboot the whole service?  Well there is a tiny probability that if you have something that can crash a process,  that can crash the data processing service, then you may have corrupted data.  If you have data that does not reach the database but it is there somewhere in the system, then  that's bad.  But if you insert corrupt data into the database, then that's really much worse.  So we try to avoid that at all costs.  Another function of the admin process is to fsync the data files, the access file, and  the bin log file.  The fsync function is a function which synchronizes the virtual memory of a file with the actual  disk content of that file, which means that the file is persisted after you make the fsync  call.  That is extremely important if you want data persistence.  The last important function is to set as data at the configurable port.  That was done with a single protocol.  It just accepts every single request that comes and then serves the JSON status data.  This status data is collected from the other two processes through some shared virtual memory  that is controlled by a simple read-write log.  So one of the first issues that we thought about was how stable would the service be  when it stays the access log?  So it turns out it's very easy and completely stable.  It's very easy to implement.  But there is one tiny problem.  The offsets are hard to calculate efficiently.  And like I said, we need those offsets to put them in the bin log file.  So in order to do that, using buffered text files is almost useless because if you use  buffered text files, you cannot know the exact offset of a single line of text.  Unbuffered text files are really slow and could not help us.  So it's actually easier to open the file in byte mode.  Just read a number of bytes, add a number of bytes that are contained in a line, and there  you go.  You have the offsets.  So between the parser and the absorber process, we have a multiprocessing queue.  The queue is using a pipe in the background, a Unix pipe, and that pipe never corrupted  data.  We never had problems with it.  But there is a problem with the data transfer speed.  Because when you actually insert some data into the queue, it actually spawns a thread  which then inserts the data gradually from the buffer, its internal buffer, into the  pipe.  If this thread does not have the gil, you are going to have data in the buffer which  is not actually inserted in the pipe.  So the connection between the parser and the absorber is broken.  There are ways to minimize the damages, and I will talk about them in a later slide.  So how could the catastrophic crash be handled securely and efficiently?  Where if that happens, if you have a catastrophic crash, then there are two files, essential  files, which can be corrupted or incomplete, the access log and the bin log.  And you can manage that somewhat by just syncing them as often as you can.  So then if you restart the machine or if you have a crash, then you can recover the data.  You can just go on and then insert the data that you have there.  So is Python fast enough to ingest this data?  Well, yes.  Actually with some performance optimizations, see Python could get about 20,000 requests  per second on the same machine on a C4X large Amazon Web Service virtual machine.  That means that you have this processing power on a machine with four virtual cores and eight  gigabytes of RAM.  However, after we inserted it into production, we re-implemented the data processing system  in CIFA.  That was quite easy to do.  It just took a week, excluding testing, and without any prior CIFN experience, and it's  doubled the performance.  So CIFN way to go.  There is another problem, though.  Like I said, we called Fsync on the two essential data files.  And we use the network file storage offered by AWS.  The Fsync does affect the performance of the read from the files.  And we had a problem at some point that's periodically, once every two days or so, the  network lags in the Amazon Web Service.  And that meant that the Fsync call took from beneath 0.1 seconds to almost 25 seconds,  which meant that the service was blocked at that time.  But it was actually pretty rare, and it did not affect the system too much.  So during one of the testing phases, we observed a strange behavior with the Absurter.  The parser was reading continuously from the access log file.  It cached the data happily.  The cache was full.  And the Absurter was not having the same efficiency as it would have under normal conditions.  So what happened there was that the parser thread was starving the feeder thread, starving  the thread that had the purpose of taking the data and then inserted into the pipe.  The only way we could manage until now to avoid this problem was to force the parser  thread to sleep periodically.  It could not be fully fixed, and it's just solved our problem right there.  So how can you maintain, how easy is to maintain such a system?  A single machine that answers to this amount of requests, like I said, 20,000 requests  per second, contains only two essential processes, the engine service, which takes the requests  and just needs a configuration file.  The Python Demon log bunker, which also needs a single configuration file.  Those machines are small.  They only contain four cores, and they are quite cheap.  They are commodity hardware.  And you can serve with a server cluster 12 million requests per minute by just using  15 virtual machines.  That means that there is no single point of failure.  When you have machines that are this small, you have enough processing power to obtain  such an improvement.  And if you lose a machine, then the system will not be affected as much as, let's say,  if you lose a gigantic 32 cores of a machine, with, God knows how many gigabytes of RAM.  Another thing to say about maintenance is that these machines are not throttled to 100%.  These machines are throttled at a maximum of 50% in a normal situation.  Let's say 75% on peaks.  This reduces the possibility of availability and hardware failure, because if you throttle  a machine at 100%, then you will have some hardware failures.  And even if the peaks reach 100% of system capacity, then you have an event queue that  is represented by the access log file.  That access log file will just keep it be appended to by the NGINX web server.  The NGINX web server will happily continue to serve the requests.  And if the Python process does not, does not, cannot pace with the NGINX web server, it  doesn't really have to.  It will have a delay from the point when the data reaches the virtual machine to when it  is inserted into the database by about, let's say, a couple of minutes.  But it will not be critical.  You will never lose data.  You will just continue pushing it into the database when the peak is gone.  And the same thing applies to the database connection.  The database can actually lag quite much.  And if you have, at some point in time, some problems with the database, let's say that  you have a cluster of database servers and one of them or several of them crash, then  you will have less right performance on them.  And if the database connection lags, then the absorber will just lag in getting the  data from the queue.  But what that means is that the data will actually add itself into the cache.  The cache will just add the data more and more and more.  And after the database is restored to its full power, it will just continue to insert  it into the database and it will just reach its 100%.  Okay.  So that was it.  Thank you for your attention.  There are questions, please.  Hi.  One question, well, actually I have probably two, but the first one is why between your  parser and your UPSetter, you don't use RabitMQ or any message queuing like, I don't know,  0MQ even would solve your problem of queue that you have in the multiprocessing.  Between the parser and the absorber.  Yeah.  Well, firstly, because it was easier to implement and because it didn't not affect the performance  as much as we thought.  So even with that performance penalty, we still reached 20,000 requests per second.  20,000 requests, the limit of the NGINX web server in an Amazon C4X large instance is  about 22, 24 requests per second.  And we did not reach that limit.  And like I said, we kept our servers to serve about 12,000 requests per second so that we  kept low the possibility of a harder failure.  So we could have implemented a message queuing service.  We thought about implementing something that we read this and we may implement that in  the future.  We don't know, but for now it's sitting quite a bit.  Well, just 0MQ is five lines and it solves your problem.  That's it.  Oh, okay.  Thanks.  Hey, I just want to ask about, you talked about how even if your log bunker process crashes  or doesn't work, you're not going to lose any logs because they're stored in the access  log.  Yes.  So you have NGINX on each virtual machine.  Yes.  So if you have a virtual machine crash or go down, what happens to the log messages that  are in the access log on that machine which have yet to be stored in your database?  They're gone.  They're not gone forever because they are kept in the access log and that access log will  be kept on the EBS network partition.  So what that means is that temporarily while the virtual machine is down and you cannot  have access to the EBS network partition, you will have some data that does not reach  the database.  But if your support team is ready, then it will reconnect to that EBS system to another  log bunker instance maybe or another tool and just reinsert it.  Thanks.  I knew I had a second question.  Why are you not using things like log stash or equivalent to do your parsing?  Log stash between what components?  Basically to parse your logs.  What you do manually by re-implementing what those utilities do is just pick up the information  directly from the logs via dedicated utilities.  I mean, there are many already existing solutions to do that precise job, I think.  I mean, I'm not expert in log stash, but I've heard many things about logging and log parsing  and whatnot.  To me, it seems that log stash or something equivalent would be a very good solution to  extract information, meaningful information from your logs because it's structured information  and probably you can inform log stash to read how to read text basically.  So that means that we'll have a cluster of servers which have the engine X on each one  of the machines.  But it's the same thing.  It's just you read your access log somehow and just instead of re-implementing something  in Python, you use a standard utility that is well maintained by professionals.  Yes, well, that means that it will be redirected to another cluster of servers which actually  has log stash on them.  Right?  Well, you can install, I guess, log stash on the same machines as where you run your  log bunker.  Yes, that is true.  However, I'm not really sure how is the performance on those services.  We actually could implement this really quickly and had really good performance.  And we thought that we could use this.  So I don't know, ultimately, I don't know how the performance would be.  Thank you.  Any other questions?  One more question?  Okay.  Thank you, Teyala.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.
Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__0_143.png Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__144_1847.png Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__1848_2819.png Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__2820_3587.png Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__3588_3839.png Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__3840_4619.png Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__4620_4871.png Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__4872_4991.png Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__4992_5123.png Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__5124_5891.png Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__5892_6407.png Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__6408_13631.png Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__13632_13847.png Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__13848_14723.png Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__14724_15359.png Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__15360_15551.png Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__15552_16451.png Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__16452_20951.png Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__20952_24515.png Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__24516_24611.png Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__24612_24719.png Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__24720_24815.png Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__24816_24959.png Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__24960_25499.png Teodor_Dima_-_Use_Python_to_process_12mil_events_per_minute_and_still_keep_it_simple__Talk__25500_31850.png