Guillotina Kafka: an awesome guillotina addon
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Leeb, Andy
This talk will describe integrating Apache Kafka with Guillotina in an addon.
 Hi, I'm Andy Lee and I'm going to talk to you about Gillatina Kafka. It's a story about an add-on. I'll talk a little bit about the reasons why we developed it, you know, where I work, who I am, and we'll get into the use case for why we built this software. I'll talk a little bit also about the software that we built and the company that I work for and so the use case and then I'll talk a little bit about Gillatina and Kafka themselves and sort of give you a little bit of an overview about that. So I'm Andy Leib. I work at Ona. I'm a platform engineer back in Python developer. I've worked with Plone for quite a while now, probably around 10 years and so I've served on the Plone Foundation board as well. So anyway, enough about me and so I work at a company called Ona and we use some Plone technologies, in particular we use Gillatina as one of our linchpins to our software stack and we have built out a fairly significant complex platform that we use in the legal service and so I'm going to describe a little bit about what we do at Ona and this will help sort of give a general idea of why we ended up having to work with Kafka. So Ona and a nutshell, we gather cloud data sources that you're authorized to see and make it searchable and so it sounds very simple but we have a fairly deep microservices stack that we use and so we can gather data from cloud sources like G Suite and Gmail and Dropbox and Slack and stuff like that and so we leverage the public APIs for these integration agents so that we can pull in the data automatically, they're long running processes and then when we get the data it's analyzed and indexed through a series of processing steps and all this allows us to search the data, allows you to search your data in one place and so typically the software is used by in-house corporate law firms and they use it for discovery and for legal holds and for risk management and so this allows them to perhaps implement things like data loss prevention or to make sure that corporate stuff isn't leaking out but there are other use cases as well that as time goes on our clients tell us about. But so anyway, we have a platform that we've built and it's based off of microservices and so we have, as I said before, we have these integration agents that go out and pull the data in, we do document processing, we do search indexing, we have a front end application that does the front end stuff, we have, Guillotine itself is our REST API server, we have a series of Guillotine-based applications as well. In addition, we also have stuff like AIO HTTP apps, we have a RabbitMQ Kafka, of course, Elasticsearch, TaskQs, Asyncios, I mean we have a lot of stuff going on and it's all orchestrated through Kubernetes, we put things in Docker containers and use Kubernetes to deploy it. We have a pretty significant sort of deep microservices stack that we have to do and so across the board, our API understands JSON and so we're fairly typical in that sense. So we use Guillotine as our REST API server. So I think that everyone's fairly familiar with it so I'm probably not going to spend too much more time on it but as you know it's built with AsyncIO, it's the ZCA reimagined so it's familiar to anyone with a ZOP or PLOM background. We use it to store our data basically and we validate our data and all that stuff with interfaces and JSON definitions of things and so we can talk more about Guillotine if we need to later. But more, probably a little bit more germane to this is a little Kafka overview and so we recently introduced Kafka into our software stack and we've had a little bit of a learning curve with it but so anyway, Kafka is essentially a PubSub message broker and it will guarantee delivery of messages in a specific order. So LinkedIn wrote it and then donated it to the Apache Foundation and so they used it essentially to record website activity and I think in the process of doing that, I think the more use cases were realized. So anyway, it's implemented in Java and Scala and it is a PubSub based messaging system used to exchange data between processes, applications and servers. So like I said, its use cases include but definitely are not limited to a distributed commit log, like a website activity tracking which is what it was originally built for but it can also be used for log aggregation and there's even more use cases for it that we were using as a buffer for example but anyway, so it runs in a cluster and the cluster itself requires a manager called ZooKeeper which keeps track of the status of the Kafka cluster, the nodes in it and make sure that the stuff that's going on in Kafka, like particularly the internal stuff like topics and partitions and stuff like that are kept in order and so it's a fairly complex piece of software but it's really robust and it's really performant and essentially once you configure it and tune it, it generally does not require a whole lot of hands on but getting to that point does take quite a bit of effort. So a little bit more about Kafka, just some terms that will probably come in handy is a producer which is what sends messages to a topic in Kafka and a consumer receives messages from that topic and then does something with it and that's really what we're focused on but a topic is a named entity within Kafka that stores a message and then a little bit smaller than that is a topic partition which topics are split into partitions which allows you to split data across multiple brokers or across a distributed cluster essentially. So what does all this really mean? So essentially what happens is you have a producer that writes to a topic in the Kafka cluster and then the consumer reads from that and does something with it and so this is all sort of done in a guaranteed order and so Kafka can keep track of where the offset is for the last thing it read and so you can set a retention policy on a topic that allows you to say if you wanted to hold on to that data indefinitely and something happened to the thing that the consumer was doing to the data, you could go back in time like to offset zero and replay everything and you would never lose any data. So essentially that's really one of the main reasons why it's so robust and so great but anyway so the data that's exchanged through Kafka, through the producer and consumer interactions are typically called messages and we use JSON but really any data format works. It can be whatever it's easiest for you, whatever is easiest for you to validate and like I say for us it was JSON because that's what our API was already doing but so why then did we need to have Kafka if we had all these other services running and things seemed to be running along pretty way, was it just because we like added complexity, do we like things that are more complicated, do we want to sound cool, more knobs, all those things are probably true but really what we needed is that reliability of each message being guaranteed to be delivered and so as I said before you can replay these messages if you need to so that way you have a guarantee even beyond the guarantee that the software already provides that you will always have that data so you won't lose anything. Another big reason for it is that we realized that with our Elasticsearch that it was fairly easy to overwhelm it with too many requests and so when we're gathering data, you know we're pushing a lot of stuff through the processing pipeline and it all ends up in an index in Elasticsearch and if you give it too much to do it might silently drop things and so this seemed to be a way to guarantee that we would always have data that would always be correct that we wouldn't have any syncing problems between our object representation and our search engine. Another reason why we decided to do it was also for user activity stuff which is one of the express use cases for Kafka but we wanted to be able to keep track, not just keep track of what people are doing but sort of have an audit log that would allow our clients to go back through and make sure that they knew what activities happened on the platform while they were using it and what the users are doing and so this is actually really great for us because since we're dealing a lot with lawyers, you know, they have specific requirements for how long the data needs to be kept and so we can then have a specific retention policy for a topic that the data gets written to and read from. Another thing that we ended up finding out through, just a little bit through trial and error but more through observation is that sometimes we would notice that not all the logs that we expected were showing up in our logging platform and so we wanted to ensure that all application logs were getting written somewhere that, you know, we could guarantee that they would be there and it wasn't like we were losing a lot of stuff but, you know, we realized that perhaps we could be doing a little bit better so we also implemented Kafka as a logging buffer. So, you know, those are some of the use cases for why we decided to implement Kafka. You know, another thing that's kind of nice about Kafka is that it's not, it's platform agnostic and so you can pretty easily write a simple producer and consumer and not have to worry about, you know, what the other services are working with but we thought that it would probably be better for us to be able to leverage Guillotina's ability to have add-ons. So essentially what we did is we did, and we did so and so we built it using the fairly standard template for Guillotina add-ons. So as we built this, we, you know, we really wanted to accomplish a couple things and one thing that was really good is that we could leverage Guillotina and the SoComponent architecture and so that allowed us to, you know, have a common vocabulary of, you know, code and reuse that we use. The other main reason was that we realized that the majority of the time was going to be spent building producers and consumers and we wanted to alleviate some of the, you know, burden of having to create them and then integrate them into every single package that might need it. And so, you know, we figured that since we already have Guillotina deployed that, you know, in our, you know, software stack that we could leverage that and then just have this add-on that was, you know, fairly easy to, that would allow us to easily create these producers and consumers. So what the add-on did is it provided two APIs to us, one for the consumer and one for the producer. And like I say, we mostly leverage it for multiple consumers. It uses, like I say, it uses ECA concepts like utilities for singletons and adapters for, you know, adding functionality and interfaces. You know, obviously it integrates with Guillotina because it's Guillotina add-on. You know, one of the other advantages is that it's based on AIO Kafka, so it's async native. It plays well with Kubernetes, which is very important for us. And so, you know, with, essentially with this, the other thing that we really needed to make sure that we could do is be able to scale up a number of consumers because as the load on our processes increased and we had more requests, we wanted to make sure that we could handle all that data, which was, you know, essentially the whole point in having Kafka introduced into our stack. And so, you know, before we wrote that add-on, we were doing one-off producers and consumers. And, you know, that's okay, but it's a little bit too much maintenance. And then furthermore, you know, it's kind of hard to scale because then you have to continue to put those same, you know, more consumer pods up in your Kubernetes cluster. So one of the things that, you know, we did to, you know, sort of as the evolution of this went on is that we made it the ability to have multiple consumers in a pod. And so what that meant is that we could have a single package with multiple producers in it and then deploy it into our, you know, Kubernetes stuff and then we could scale that way up if we needed to. You know, it's good in the sense that it allows us to not have a lot of individual consumer packages, which, you know, is added maintenance and, you know, there's more pods and stuff. And so we can have, you know, multiple consumers in a single pod and then we can scale that up and down as needed. But the other thing that, you know, we sort of noticed about this whole thing is that, you know, most of the consumers, you know, essentially do the same thing where they read from a topic and then put it into the next component, you know, a lot of the times it's putting it into, you know, a specific, you know, part of Elasticsearch. And so, you know, multiple consumers in a single package is actually really very powerful for, you know, for us. Did I mention that it makes it much easier to scale because that was really something that we found that we needed? So anyway, you know, what did we learn about all of this? You know, we learned that it's, you know, helpful to have it a way to easily create producers and consumers in sort of a predictable way. And it's, the add-on also provided just enough framework so that we could easily customize what the consumer does, you know. We did learn that, of course, scaling is important because it always is. But you know, one sort of even bigger lesson that we learned from all of this is that, you know, just being able to create a producer and a consumer easily and being able to customize it is only a small part of it that, you know, there was a significant part, you know, a significant amount of time spent tuning Kafka itself and making sure that, you know, it was behaving the way that we wanted to in a predictable fashion, that it wasn't taking too many resources and, you know, that we could, you know, that we had it tuned in a way that it could easily handle all the load. So, you know, that was, you know, a big part of it. It wasn't necessarily part of the add-on. It just so happened that it, you know, it all sort of happened at around the same time. And so one, you know, one other thing that we sort of learned from it is that we could probably do a better implementation that maybe wasn't a guillotine add-on, that was maybe just a library that we could easily integrate into, you know, one place in our stack instead of necessarily having it as an add-on. So, you know, we did sort of look at, you know, what the next steps would be with it. And so, you know, like I say, we took looking at a package called FOST, which does, you know, essentially a similar thing, but is a lot more overblown than what we need. But looking at the way that they did their sort of Python Kafka integration stuff, you know, we could probably do it with decorators instead of having to have an add-on. You know, and we could, like I say, we could build it as a library instead of an add-on, but, you know, not to worry, we're probably going to continue working with the add-on. So that's pretty much all the things that we did with guillotine and Kafka in, you know, in a very quick overview. So do we have any questions? Hi, it was nice to see, like, this integration. As far as understood, you collected data inside of guillotine, right? Yeah. So did you talk about using, like, a CDC solution, like, to capture this from the database chains or why it's better to do this way or not for your user case? Well, we found that this was a little bit better than what we had before because the way that our software works is that we do store the data and the object representation in guillotine and then we would sync that with Elasticsearch. And the job that did that, you know, was kind of process-intensive, process-intensive, and so we figured that this is probably a little bit better than having to run a background process. And so this, you know, happens a little bit more closer to real-time, and so we felt that the results also in the search were improving a little bit with this. So what is Kafka and you just can... Related to that, Elasticsearch, for example, it's really important to batch update and reduce the amount of concurrent updates to things. So Elasticsearch, we can't handle all the load that guillotine is pushing into it. But I did have a question for you. Did you want to talk about monitoring Kafka and some of the things that we had to do in order to see where we were on the Kafka offset in terms of processing? Yeah, I mean, I can touch on that a little bit. So, you know, full disclosure, Nathan works with me and had done quite a bit of the engineering on, you know, a lot of this Kafka stuff as well. So we did, you know, have to monitor what Kafka was doing in order to get some idea of the performance that we expected from it. And essentially, we needed to have a little bit better integration into our main application in order to have that visibility. And so in order for us to, you know, essentially, like, we weren't able to customize the add-on in order to get, you know, some of this, like, figuring out where the offset was, that had to actually be done as a customization to the producer or the consumer in the application code. And so that was, you know, a little bit about on the customization that we allowed with the framework that we had. So I mean, it's a little bit of a non-answer, but that's, you know, essentially the level of detail that I feel like I could provide at this point. Yes, I find it very cool to know, again, Kafka, I went to a meetup local where I come from a few weeks ago, and there I got a whole presentation on Kafka, on banking accounts, and also, and one of the things that came up there, that if you want to replay all the messages that have come into a topic and you have to go back, Kafka stores the messages. So when I hear you, you say, look, we want to go back to zero, then you now have two and a half places where you store it. You store it in Guillotine in a database, then it goes to Kafka, and Kafka has to have a copy of the thing. And then, yeah, okay, elastic search is not really storage, but it's a huge index, of course. But now you effectively double the storage you need, right? Yeah. Yeah, essentially you do. And so, you know, it depends on how long you want to retain those messages. And in some cases, you know, you want forever, maybe. And in some cases, you may only want a minute. So it all depends. And since we consider the source of truth to be in our Guillotine application, that's more important for us, for a lot of these things. And so there are certain clever things that you could do, depending on what your data is, and the messages that you send to Kafka in order to trigger whatever to be re-indexed. But the sort of like the backing, the offset up to zero and replaying everything would be done if there was a catastrophe, right? Like if you lost an index, or suppose you wanted to do a migration, and you had a new version of elastic search or something like that, and you wanted to, you know, more easily like not try to move in between versions, and you could just take the data from, you know, day zero and run it into there. So that's more the use cases that we had in mind, but you are correct in that, you know, it requires additional storage, particularly if you want to retain the data for a long period of time.
 Hi, I'm Andy Lee and I'm going to talk to you about Gillatina Kafka.  It's a story about an add-on.  I'll talk a little bit about the reasons why we developed it, you know, where I work,  who I am, and we'll get into the use case for why we built this software.  I'll talk a little bit also about the software that we built and the company that I work  for and so the use case and then I'll talk a little bit about Gillatina and Kafka themselves  and sort of give you a little bit of an overview about that.  So I'm Andy Leib.  I work at Ona.  I'm a platform engineer back in Python developer.  I've worked with Plone for quite a while now, probably around 10 years and so I've served  on the Plone Foundation board as well.  So anyway, enough about me and so I work at a company called Ona and we use some Plone  technologies, in particular we use Gillatina as one of our linchpins to our software stack  and we have built out a fairly significant complex platform that we use in the legal  service and so I'm going to describe a little bit about what we do at Ona and this will  help sort of give a general idea of why we ended up having to work with Kafka.  So Ona and a nutshell, we gather cloud data sources that you're authorized to see and  make it searchable and so it sounds very simple but we have a fairly deep microservices stack  that we use and so we can gather data from cloud sources like G Suite and Gmail and  Dropbox and Slack and stuff like that and so we leverage the public APIs for these integration  agents so that we can pull in the data automatically, they're long running processes and then when  we get the data it's analyzed and indexed through a series of processing steps and all  this allows us to search the data, allows you to search your data in one place and so typically  the software is used by in-house corporate law firms and they use it for discovery and  for legal holds and for risk management and so this allows them to perhaps implement things  like data loss prevention or to make sure that corporate stuff isn't leaking out but  there are other use cases as well that as time goes on our clients tell us about.  But so anyway, we have a platform that we've built and it's based off of microservices and  so we have, as I said before, we have these integration agents that go out and pull the  data in, we do document processing, we do search indexing, we have a front end application  that does the front end stuff, we have, Guillotine itself is our REST API server, we have a series  of Guillotine-based applications as well.  In addition, we also have stuff like AIO HTTP apps, we have a RabbitMQ Kafka, of course,  Elasticsearch, TaskQs, Asyncios, I mean we have a lot of stuff going on and it's all  orchestrated through Kubernetes, we put things in Docker containers and use Kubernetes to  deploy it.  We have a pretty significant sort of deep microservices stack that we have to do and  so across the board, our API understands JSON and so we're fairly typical in that sense.  So we use Guillotine as our REST API server.  So I think that everyone's fairly familiar with it so I'm probably not going to spend  too much more time on it but as you know it's built with AsyncIO, it's the ZCA reimagined  so it's familiar to anyone with a ZOP or PLOM background.  We use it to store our data basically and we validate our data and all that stuff with  interfaces and JSON definitions of things and so we can talk more about Guillotine if  we need to later.  But more, probably a little bit more germane to this is a little Kafka overview and so  we recently introduced Kafka into our software stack and we've had a little bit of a learning  curve with it but so anyway, Kafka is essentially a PubSub message broker and it will guarantee  delivery of messages in a specific order.  So LinkedIn wrote it and then donated it to the Apache Foundation and so they used it  essentially to record website activity and I think in the process of doing that, I think  the more use cases were realized.  So anyway, it's implemented in Java and Scala and it is a PubSub based messaging system  used to exchange data between processes, applications and servers.  So like I said, its use cases include but definitely are not limited to a distributed  commit log, like a website activity tracking which is what it was originally built for  but it can also be used for log aggregation and there's even more use cases for it that  we were using as a buffer for example but anyway, so it runs in a cluster and the cluster  itself requires a manager called ZooKeeper which keeps track of the status of the Kafka  cluster, the nodes in it and make sure that the stuff that's going on in Kafka, like particularly  the internal stuff like topics and partitions and stuff like that are kept in order and  so it's a fairly complex piece of software but it's really robust and it's really performant  and essentially once you configure it and tune it, it generally does not require a whole  lot of hands on but getting to that point does take quite a bit of effort.  So a little bit more about Kafka, just some terms that will probably come in handy is  a producer which is what sends messages to a topic in Kafka and a consumer receives messages  from that topic and then does something with it and that's really what we're focused on  but a topic is a named entity within Kafka that stores a message and then a little bit  smaller than that is a topic partition which topics are split into partitions which allows  you to split data across multiple brokers or across a distributed cluster essentially.  So what does all this really mean?  So essentially what happens is you have a producer that writes to a topic in the Kafka  cluster and then the consumer reads from that and does something with it and so this is all  sort of done in a guaranteed order and so Kafka can keep track of where the offset is  for the last thing it read and so you can set a retention policy on a topic that allows  you to say if you wanted to hold on to that data indefinitely and something happened to  the thing that the consumer was doing to the data, you could go back in time like to offset  zero and replay everything and you would never lose any data.  So essentially that's really one of the main reasons why it's so robust and so great but  anyway so the data that's exchanged through Kafka, through the producer and consumer interactions  are typically called messages and we use JSON but really any data format works.  It can be whatever it's easiest for you, whatever is easiest for you to validate and like I  say for us it was JSON because that's what our API was already doing but so why then  did we need to have Kafka if we had all these other services running and things seemed to  be running along pretty way, was it just because we like added complexity, do we like things  that are more complicated, do we want to sound cool, more knobs, all those things are probably  true but really what we needed is that reliability of each message being guaranteed to be delivered  and so as I said before you can replay these messages if you need to so that way you have  a guarantee even beyond the guarantee that the software already provides that you will  always have that data so you won't lose anything.  Another big reason for it is that we realized that with our Elasticsearch that it was fairly  easy to overwhelm it with too many requests and so when we're gathering data, you know  we're pushing a lot of stuff through the processing pipeline and it all ends up in an index in  Elasticsearch and if you give it too much to do it might silently drop things and so  this seemed to be a way to guarantee that we would always have data that would always  be correct that we wouldn't have any syncing problems between our object representation  and our search engine.  Another reason why we decided to do it was also for user activity stuff which is one  of the express use cases for Kafka but we wanted to be able to keep track, not just keep track  of what people are doing but sort of have an audit log that would allow our clients  to go back through and make sure that they knew what activities happened on the platform  while they were using it and what the users are doing and so this is actually really  great for us because since we're dealing a lot with lawyers, you know, they have specific  requirements for how long the data needs to be kept and so we can then have a specific  retention policy for a topic that the data gets written to and read from.  Another thing that we ended up finding out through, just a little bit through trial and  error but more through observation is that sometimes we would notice that not all the  logs that we expected were showing up in our logging platform and so we wanted to ensure  that all application logs were getting written somewhere that, you know, we could guarantee  that they would be there and it wasn't like we were losing a lot of stuff but, you know,  we realized that perhaps we could be doing a little bit better so we also implemented  Kafka as a logging buffer.  So, you know, those are some of the use cases for why we decided to implement Kafka.  You know, another thing that's kind of nice about Kafka is that it's not, it's platform  agnostic and so you can pretty easily write a simple producer and consumer and not have  to worry about, you know, what the other services are working with but we thought that it would  probably be better for us to be able to leverage Guillotina's ability to have add-ons.  So essentially what we did is we did, and we did so and so we built it using the fairly  standard template for Guillotina add-ons.  So as we built this, we, you know, we really wanted to accomplish a couple things and one  thing that was really good is that we could leverage Guillotina and the SoComponent architecture  and so that allowed us to, you know, have a common vocabulary of, you know, code and  reuse that we use.  The other main reason was that we realized that the majority of the time was going to  be spent building producers and consumers and we wanted to alleviate some of the, you  know, burden of having to create them and then integrate them into every single package  that might need it.  And so, you know, we figured that since we already have Guillotina deployed that, you  know, in our, you know, software stack that we could leverage that and then just have this  add-on that was, you know, fairly easy to, that would allow us to easily create these  producers and consumers.  So what the add-on did is it provided two APIs to us, one for the consumer and one for  the producer.  And like I say, we mostly leverage it for multiple consumers.  It uses, like I say, it uses ECA concepts like utilities for singletons and adapters  for, you know, adding functionality and interfaces.  You know, obviously it integrates with Guillotina because it's Guillotina add-on.  You know, one of the other advantages is that it's based on AIO Kafka, so it's async native.  It plays well with Kubernetes, which is very important for us.  And so, you know, with, essentially with this, the other thing that we really needed to make  sure that we could do is be able to scale up a number of consumers because as the load  on our processes increased and we had more requests, we wanted to make sure that we could  handle all that data, which was, you know, essentially the whole point in having Kafka  introduced into our stack.  And so, you know, before we wrote that add-on, we were doing one-off producers and consumers.  And, you know, that's okay, but it's a little bit too much maintenance.  And then furthermore, you know, it's kind of hard to scale because then you have to  continue to put those same, you know, more consumer pods up in your Kubernetes cluster.  So one of the things that, you know, we did to, you know, sort of as the evolution of  this went on is that we made it the ability to have multiple consumers in a pod.  And so what that meant is that we could have a single package with multiple producers in  it and then deploy it into our, you know, Kubernetes stuff and then we could scale that  way up if we needed to.  You know, it's good in the sense that it allows us to not have a lot of individual consumer  packages, which, you know, is added maintenance and, you know, there's more pods and stuff.  And so we can have, you know, multiple consumers in a single pod and then we can scale that  up and down as needed.  But the other thing that, you know, we sort of noticed about this whole thing is that,  you know, most of the consumers, you know, essentially do the same thing where they read  from a topic and then put it into the next component, you know, a lot of the times it's  putting it into, you know, a specific, you know, part of Elasticsearch.  And so, you know, multiple consumers in a single package is actually really very powerful  for, you know, for us.  Did I mention that it makes it much easier to scale because that was really something  that we found that we needed?  So anyway, you know, what did we learn about all of this?  You know, we learned that it's, you know, helpful to have it a way to easily create  producers and consumers in sort of a predictable way.  And it's, the add-on also provided just enough framework so that we could easily customize  what the consumer does, you know.  We did learn that, of course, scaling is important because it always is.  But you know, one sort of even bigger lesson that we learned from all of this is that,  you know, just being able to create a producer and a consumer easily and being able to customize  it is only a small part of it that, you know, there was a significant part, you know, a  significant amount of time spent tuning Kafka itself and making sure that, you know, it  was behaving the way that we wanted to in a predictable fashion, that it wasn't taking  too many resources and, you know, that we could, you know, that we had it tuned in a  way that it could easily handle all the load.  So, you know, that was, you know, a big part of it.  It wasn't necessarily part of the add-on.  It just so happened that it, you know, it all sort of happened at around the same time.  And so one, you know, one other thing that we sort of learned from it is that we could  probably do a better implementation that maybe wasn't a guillotine add-on, that was maybe  just a library that we could easily integrate into, you know, one place in our stack instead  of necessarily having it as an add-on.  So, you know, we did sort of look at, you know, what the next steps would be with it.  And so, you know, like I say, we took looking at a package called FOST, which does, you  know, essentially a similar thing, but is a lot more overblown than what we need.  But looking at the way that they did their sort of Python Kafka integration stuff, you  know, we could probably do it with decorators instead of having to have an add-on.  You know, and we could, like I say, we could build it as a library instead of an add-on,  but, you know, not to worry, we're probably going to continue working with the add-on.  So that's pretty much all the things that we did with guillotine and Kafka in, you know,  in a very quick overview.  So do we have any questions?  Hi, it was nice to see, like, this integration.  As far as understood, you collected data inside of guillotine, right?  Yeah.  So did you talk about using, like, a CDC solution, like, to capture this from the database chains  or why it's better to do this way or not for your user case?  Well, we found that this was a little bit better than what we had before because the  way that our software works is that we do store the data and the object representation  in guillotine and then we would sync that with Elasticsearch.  And the job that did that, you know, was kind of process-intensive, process-intensive, and  so we figured that this is probably a little bit better than having to run a background  process.  And so this, you know, happens a little bit more closer to real-time, and so we felt that  the results also in the search were improving a little bit with this.  So what is Kafka and you just can...  Related to that, Elasticsearch, for example, it's really important to batch update and  reduce the amount of concurrent updates to things.  So Elasticsearch, we can't handle all the load that guillotine is pushing into it.  But I did have a question for you.  Did you want to talk about monitoring Kafka and some of the things that we had to do in  order to see where we were on the Kafka offset in terms of processing?  Yeah, I mean, I can touch on that a little bit.  So, you know, full disclosure, Nathan works with me and had done quite a bit of the engineering  on, you know, a lot of this Kafka stuff as well.  So we did, you know, have to monitor what Kafka was doing in order to get some idea of  the performance that we expected from it.  And essentially, we needed to have a little bit better integration into our main application  in order to have that visibility.  And so in order for us to, you know, essentially, like, we weren't able to customize the add-on  in order to get, you know, some of this, like, figuring out where the offset was, that had  to actually be done as a customization to the producer or the consumer in the application  code.  And so that was, you know, a little bit about on the customization that we allowed with  the framework that we had.  So I mean, it's a little bit of a non-answer, but that's, you know, essentially the level  of detail that I feel like I could provide at this point.  Yes, I find it very cool to know, again, Kafka, I went to a meetup local where I come from  a few weeks ago, and there I got a whole presentation on Kafka, on banking accounts, and also, and  one of the things that came up there, that if you want to replay all the messages that  have come into a topic and you have to go back, Kafka stores the messages.  So when I hear you, you say, look, we want to go back to zero, then you now have two  and a half places where you store it.  You store it in Guillotine in a database, then it goes to Kafka, and Kafka has to have  a copy of the thing.  And then, yeah, okay, elastic search is not really storage, but it's a huge index, of  course.  But now you effectively double the storage you need, right?  Yeah.  Yeah, essentially you do.  And so, you know, it depends on how long you want to retain those messages.  And in some cases, you know, you want forever, maybe.  And in some cases, you may only want a minute.  So it all depends.  And since we consider the source of truth to be in our Guillotine application, that's  more important for us, for a lot of these things.  And so there are certain clever things that you could do, depending on what your data is,  and the messages that you send to Kafka in order to trigger whatever to be re-indexed.  But the sort of like the backing, the offset up to zero and replaying everything would  be done if there was a catastrophe, right?  Like if you lost an index, or suppose you wanted to do a migration, and you had a new  version of elastic search or something like that, and you wanted to, you know, more easily  like not try to move in between versions, and you could just take the data from, you  know, day zero and run it into there.  So that's more the use cases that we had in mind, but you are correct in that, you know,  it requires additional storage, particularly if you want to retain the data for a long  period of time.
Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_0_239.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_240_311.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_312_3863.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_3864_5411.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_5412_6131.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_6132_9467.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_9468_9515.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_9516_12791.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_12792_12911.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_12912_13355.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_13356_14063.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_14064_14507.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_14508_14831.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_14832_14915.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_14916_15731.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_15732_17063.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_17064_27395.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_27396_27947.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_27948_30635.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_30636_30719.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_30720_33551.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_33552_34415.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_34416_34511.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_34512_34559.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_34560_34691.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_34692_34763.png Andy_Leeb_-_Guillotina_Kafka__an_awesome_guillotin_zw5jOHXrBPQ_34764_34836.png