Taking the Hipster out of Streaming
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Heider, Andreas
Wall, Robert
Taking the Hipster out of Streaming Winton ingests data continually from the world's financial markets. We track millions of individual timeseries, with divergent formats, from disparate time zones, and whose frequencies vary from months to milliseconds. We go beyond simply reading and storing it - we stitch distinct and vast data sets together and subject them to intricate calculations in real-time. This talk will focus on the way we use Python to achieve these ends, and how we are creating tools to further commoditise streaming as a service
 Yeah, thanks a lot for coming, everyone. Special greets to everyone watching this stream at the moment. Today, we want to talk a little bit around our experiences and story of how we've applied stream processing over the years, what we've learned, and towards the end, we're going to have an exciting announcement of how we think we can make stream processing with Python quite a lot better. So, in short, who are we? We are both two developers at Wynton, which is a global investment management and data science company. You might have already talked to us quite a lot at our stand downstairs, and we've been in the game for quite a while now for 20 years. And our core thing really is that we believe that the scientific method can be applied profitably to financial markets, which means we run a lot of data processing on financial and non-financial data and try to make a profit out of that. And as part of that, we apply a lot of technology and data science, and this talk is really about one part of that of how we do real-time stream processing. I'm interested, so quick hands up, please, if you've ever heard of Kafka or done some stream processing. Oh, very good. That's good. Okay. Then I'll go with this a bit shorter. Basically, what stream processing is, is that you, instead of doing the classic batch processing where you would go at the end of the day, and you collect all of your data throughout the day, write it into a big file, and then run, for example, a huge Spark job or something on that on it, you work on all your data as a stream, which means data comes in throughout the entire day, and you process it as it comes in, which is quite nice because then you get real-time and interactive data. You can put it on a nice dashboard, and often it also makes thinking about your processing quite a lot simpler because you don't have those random interruptions where your one batch stops and the next one starts. And one example for the stream processing in finance, we've actually been doing stream processing for quite a long time now, like this is just the natural thing you do. For example, if you buy and sell stocks at an exchange, you can subscribe to their market data feed. And basically what they do is every time someone buys something or sells something, you get a little message that tells you, for example, someone at 10, 15 has bought some stock in Apple and they've bought 10 shares at a price of $144, for example. And you really get this flowing in, you get a massive amount of messages every second, and then you do various things with it. One of the things we do, for example, is we show it in various dashboards and it feeds into various dragging algorithms, and they don't necessarily always care about every single trade that happens, but like in the question for the talk before, often we want to have like a little bit of an aggregate, so we want, for example, to know every minute how many Apple has been traded in total for what average price, for example, and that's something that one of our stream processors would do. So we take a nice input stream of high frequency events, run a bidding process, and then group it down into slower streaming events that are closer to what we really care about. And then our downstream consumers don't really need to do that much work if they're not interested in all the details. And over the time we found the streaming model really works well, so now it's expanded from just handling market data, which is the core financial data of the stock has been traded, to alternative data, which means we run some web scraping, we look at news, we look at various other not really core financial data. We've nowadays got an entire office in San Francisco focusing on this. The other thing that we found that's really useful is just normal business events, so something happens on the day and you want to make sure that once that happens, this other job runs or you want to update some monitoring page. And there we've really found the combination of stream processing where we would send, for example, this job has run on an event bus in combination with a more traditional database is really useful. So you would write your normal data into the database and then you write a little notification onto your event stream and that then can tell various applications to the right thing. For example, we run monitoring on it, we do our risk management based off those events, we do our trading decisions and there's various other things. And what I really want to focus on today is kind of the thing in the middle where we get various raw events in and then we process it a little bit via various transformations into the format that we really care about in the end. And if you're talking about stream processing nowadays, what you really mean usually is Apache Kafka, which is the project which has won the before mentioned streaming wars really. Apache Kafka is a message broker, which means it's a little bit like a database, so you install it on a server and it provides you with something called topics where you would write various data that belongs together on a single topic from a producer and then various consumers can pick up that data and do things with it. And one important thing around it is the topics are subdivided into different partitions and that gives you parallelism. So for example, if you have trades, you would send some of your trades on partition one, some on partition two, and then you can run multiple consumers, each consumer gets a petition and then they can share their work a little bit. And the other nice thing around Kafka is it makes it really easy to, or it has a really nice underlying model of how it deals with data. So within a partition, every message is ordered and you get it out in exactly the same order and it's really quite beautiful and elegant how it's implemented. And because it's so popular, it has huge support around the big data ecosystem. So the talk before we heard their system can interact with Kafka. There's a bunch of others which can do it. So there's a lot of projects that have sprung up lately in the last couple of years that try to make it easier for you to do stream processing on those events. We've looked at pretty much all of them and we weren't really too happy with them, basically for two main reasons. If you're interested in details, come talk to me later. I can talk about this for hours. But really the two core reasons where we had issues with those projects is that usually they're pretty heavyweight. So kind of if you want to use Spark Streaming to do your stream processing, you need to run and maintain an entire Spark cluster and then your code runs inside the Spark cluster and there's quite a bit of overhead in that, just installing it, setting it up, configuring it correctly and then doing things like scaling up. And the other thing is pretty much all of those projects are written for the JVM, which means Python is a bit of a second-class citizen and then doesn't really fit in or you talk with the actual system via this text JSON input output, which isn't really great for performance. There's one nice outline. Pretty much one year ago, Confluent and the creators of Apache Kafka introduced a new API for Kafka that does very similar things with these other stream processing frameworks do, but does it as just a simple library, which means it turns around the relationship between your code and the framework. So in the others, you would submit your code, you need to package it up and then it runs somewhere inside the Storm server and what you can do with Kafka Streams is it's just a library, which means the master of everything is still just your normal Python process. You can run it any way you want. You can run it on AWS. If you auto-scale it, you just auto-scale it completely the normal way because it's just a normal process. It's also got proper real event-by-event processing, which means unlike, for example, Spark streaming, you don't need to batch events up so you can get pretty good latency and it helps you a lot with all those really tricky things around stream processing that gets difficult if you want to do some stateful processing, for example, for the binning or if you want to join different streams together, if you want to do some aggregations and also for tolerance and distributed processing is solved quite nicely. And it's now part of the main Apache Kafka client, but the big drop back for us is so far it's Java only and it's not that easy to write bindings to that Java interface. And that's a bit of a problem for us because at Rintten we are big Python users. We are around 450 people in total and really at least half of them know some Python. They're not all developers, but we have lots of data scientists, researchers, operations people. And if you want to know a little bit more about our Python usage, we had a talk last year by some of our colleagues that showed a little bit what we do with Python. Yep. And what we really want is to give all those people who know Python the ability to do stream processing nicely and easily. And what we've done so far is kind of this hipster programming, which means every time you do it, you roll it on your own. You just grab a Kafka client, there's various good ones, and you connect to your cluster. And in four lines you can connect to it, you subscribe to messages, you get a callback for every single message. And to get started with, this is actually a really good idea. And we've been running quite a lot just like this, various processes to update monitoring and so on. If you want to go down that route, there's two clients I would recommend. There is one written by Donna Powers. Thanks a lot for writing that, which is a native Python client. They've written all of the protocol interactions in Python, and they especially have a really nice Python interface that feels very natural to use it. It's really nice and robust. It handles, for example, your cluster scaling correctly. And the other recommendation I would have is the Confluent Kafka client, which is a wrapper around the C library, which means it's really, really high performance. So we've pumped something like a million messages per second through it pretty easily. And yeah, they're both good. Since we had Parsley talking earlier, thanks to them as well. They've also built a Python client, but I haven't really used it that much. I thought I should mention it. So what we've done is we've wrote our own streaming framework quite a couple of times. Every time we start with those 10 lines or four lines, maybe, you start building it, and then you run it in production. In your demo setup, it all looks fine, and you let it lose and things slightly break or are slightly wrong because streaming is actually really, really hard. For example, if you want to do the spinning from earlier, where you say you group some traits together, then you do an aggregation over them and calculate some average or total. You really need to be careful that you handle every message exactly once because if you double count them, your aggregation is just wrong. And that's really something that people have actually struggled with. Stateful processing is difficult. If you want to do joins or if you want to cache like the last bits, you kind of run into the same problems that if you keep a local database as a lookup, you need to be really, really careful that you don't apply things twice. And that's hard. Then distributing load can be quite difficult because while the client's supported, usually what you have to do is you need to handle certain callbacks that get triggered when the rebalancing happens correctly and that can be quite tricky. It's fine if you've been doing it for a while, but if you're really just a data scientist who wants to write some algorithm and some code and run that, it can be quite difficult. And then if you want performance out of your streaming, you usually want to do micro-batching, which means that you buffer up. You don't do like a database write on every single message you do, but you group them together and after a while, like for example, every second, then you process all the messages you got as one chunk and that gives you quite a lot of speed up. And because we want all of that, we've decided we've started an open source project on our new open source organization to basically bring Kafka streams, which we think has solved all of those problems really, really nicely for Java and we're doing a re-implementation of that just in Python, pure pyphonic, which means you don't have any dependencies over and above a simple Kafka server. And my colleague Rob will now show you a little demo of how it really looks like and what it can do. Hi. Very excited, a little nervous to do a live demo, but it was a... So I'm going to start... No. It's good. Sorry. Now, I'm going to start up all the components required for the application, starting with ZooKeeper. So Kafka uses ZooKeeper in the background for cluster management, for storing the topics, leadership elections, for example. So let's start up our ZooKeeper. Start a Kafka server. Now, I'm going to start just one instance. In production, you really would want, say, three instances, three replicas for extra resilience and so on. So let's go ahead and start up Kafka. That's now running. I'm going to start up a consumer. And this is just the default console consumer that comes with Kafka. It'll take whatever is in the topic, bin prices. Sorry. Can you see this? Is it big enough? Great. So it will write out whatever finds in bin prices out to the console. There's currently nothing in there. We're not running our application just yet. So let's... Now, just before I... Let's start up the application. There it is. It started. We can see that it's listening to the prices topic. So I think it was at the fourth, the fifth line down. It says that it's listening to the prices topic. It will write out to the bin prices topic. We've currently got no data in the prices topic, though. I'm going to start one more consumer first. That's here in my Jupyter Notebook. So let's do the imports, set up Bokeh, set up our consumer. This is very similar to the lines code you just saw, Andreas. Show you. And our plot. So there's our plot ready. We've got one more thing to do, and that's just to start up this loop down here, which will loop on the consumer to read in the prices from the input topic. So let's start it. Now, again, like the previous consumer, there's no output yet because there are no inputs in there. So one last thing to do, let's get some inputs. Right. We've got this generator script. And the generator script is going to generate some prices, just random prices that normally distributed returns for the prices. So let's start generating some prices. Go back to our application. We can see it's already begun to process those. Go to this consumer here. We see there's nothing there yet. There's nothing there because we're binning the prices. Our definition of a bin is when we reach the end of a minute. For the purpose of the example, in fact, two minutes, we will push the last value on the bin, the last two values, sorry, the last two minutes out to the output topic. So let's have a look inside Chrome at our plot. We can see that our input prices are being parsed or being read in. Our bin prices will appear as these red dots here. So once we reach two minutes, I have sped this up. You can run the generator in approximately real time. So one moment more. We should see the dots. There we go. So as I say, in production, if we were running similar code, we would obviously be interested in pushing out the bin price every minute. For the purpose of the example here, we've just waited for two prices to be available before pushing them out. This will just carry on plotting. We'll just leave it go there. And we'll move back here. You can see in the consumer, the console consumer here, it's also found the two prices. So these are two perfectly independent applications. Neither knows about the other, but both are consuming the same data. So what does the code behind this look like? Here we go. So the first thing you probably want to do is to write a topology. Our topology comprises at least one source, one sync. The source is your input topic. So prices, our sync is the output topic, bin prices. And you will want at least one processor in between. The processor is where the meter your calculation really takes place. So in here in the binning class is the processor. So we've got some initialization, fairly standard stuff. And we've had this process function, process method. The process method will be called for every value that is read in from the input topic and then we convert it to the output. When we find two, so you can see down here when our store contains two outputs, so we've obviously transitioned into the third minute, say, we will call punctuate. Punctuate simply forwards the values, the bin values out to the output topic. So that's the flow from input to output and what we saw in the bokeh plot in Chrome. So we've seen the application, we've seen some code. What comes next? So the first thing and most importantly really is to finish our implementation. We're at a very early stage. We're at a very early stage but we absolutely encourage contributions. We'd love you to check out our code, start on GitHub, file some issues, write tests, write documentation, whatever you like. Just get involved. We'd really like to experiment with a more pythonic API, in particular for the DSL. DSL is the domain specific language that sits on top of the streams API in Kafka. We feel that there is very good opportunity to leverage Python's strengths to make it even easier to set up a topology using the domain specific language. Optimize, Python is a great language. We all love it. We wouldn't be here if we didn't. But we're under no illusions. It does have, it can hit limits. And when we hit limits we will work on optimizing the performance further. This could include batching, it could include using some of the great Python libraries, NumPy, Pandas are out there, or even leveraging some new projects like Apache arrow. Lastly, well, lastly on this slide, but really the roadmap is longer, there are many more advanced features of Kafka. The 011 release recently introduced exactly one semantics, for example. So yes, as I said before, and I'll repeat again, get in touch. There's the GitHub page. I can't seem to click on that. So it is there, the GitHub page. Get involved, check it out. And are there any questions? Hello, hello, Aya. Thank you very much. Very interesting. I waited for this for half a year now. So thank you. I will check it out just now. I haven't found it. So if you Google for this one, it doesn't show up. So we have to do some search engine optimization. It was out of just today. Okay. Very good. But I already tweeted it. So. Excellent. Thank you. Questions? Hi. So in the last presentation, we saw Apache storm and how supposedly nice it is with tracing the workflows and sync errors and stuff like that. And can you have that easily out of the box with Kafka streams and your library? It's a bit more lightweight. So parts of them you can have, especially Confluent, is happy to sell you an entire seat of tools. On the other side, because it's just a library, it's pretty lightweight and doesn't do like spinning up of your processes, which means that if you already have existing infrastructure to do all of that, for example, we run a lot of code on Kubernetes nowadays, you can just do it on there and it's integrated with what you already have. Thanks. Another one was over there. Thanks for the way you talk. Have you tried some kinds of tools for the stream processing part in particular? Not yet. But I would love to talk to you about that, how you think this could work. Because this is exactly something that we would really like to make nice and pyphonic. So please come to our booth maybe later. I would say that if you look at the current code base, it follows the Java code base reasonably closely, which we felt was a reasonable way to get started, get to learn the code behind the Java implementation. But yes, it would be fantastic to introduce new code where we can leverage, as I said, before Python's strengths. I worked one comment. So if someone is asking yourself, should I use streaming at all? There's a talk from him tomorrow why you should use streaming. Thank you. And it's really great to see the project you started, because I would have also said that this is missing in Python at the moment, so I'm very glad to try this out. As I looked also at Kafka streams, and one thing I found particularly interesting is the scaling approach and the state management approach, is that already included in your approach or is that on the roadmap? The scaling, we do have working in a simple fashion. So basically, you can run two different Python processes and they will share the load between them and balance it. The state processing is a bit further out. So before we want to do that, basically that's what we're working on next. It's really tricky to implement, which is why we want to put it in this library. Yeah, it's great. But it's coming. So what's the time? I think we are already over, isn't it? No? So first, it's good that you're making this open source. I'm not familiar with Kafka streams, but I use Kafka Hector. So I kind of like feel the pain you were describing. And my question is, I tried to combine RX, Python, these reactive extensions with Kafka. And it sort of works, but you have to write a lot of code. I'm not familiar with how Kafka streams work. So it's similar to that concept of having operation of the streaming or the events that are coming through Kafka and windowing and stuff like that. Yeah, it's basically exactly this. So it's the same. You get your stream input object and then you can map a function onto it or you can group by and you can split it up. It's basically exactly that. We have time for maybe two more questions. So do you use some binary protocol to get events from Kafka? Like is it performant? Is it fast? Okay, so to talk to, so basically what this is, this sits on top of an existing Kafka client and basically takes away all the boilerplate that you have to write. And beneath it, at the moment, we're using the Confluent Kafka client. This one, the second one. And basically that's talking just a normal binary Kafka protocol, but this highly optimized C implementation is really, really fast. Where I think the current bottleneck would be is where it goes from C implementation, where you get one callback for every single message into Python code. And basically what we've seen is that we really need to include some batching that, for example, you get a lumpy array of all your events back because if you do processing with Python of a million records per second, if you look at the profiler, really every single if shows up and it's getting a bit hard to optimize there. But yeah, in short, yeah, we're using a really optimized Kafka client underneath and it talks a binary protocol. Okay, one last question. Thank you. We recently implemented the exact same solution at my company, but for Amazon Kinesis, which is very similar to Kafka. Yeah. And I was wondering if you found any way to test your applications using this library without having the need of a full Kafka installation in your local development environment. Okay. So I think this library makes it a bit easier because you aren't so low level, so it's relatively if you write a processor, you have a really nicely defined Python interface and basically we will test against that. But I feel your pain around if you stitch more of them together at some point where we've managed or we've mostly gone for integration tests because there's so many small things that are really hard to unit test correctly. I decode Andreas's comments. I would just say though that the code that I showed the binning.py was the full application. So it should be not necessarily easy, but not very difficult to abstract that in a way to make it more amenable to testing because that is everything. Okay. So thank you very much for your talk and for all the things. So please start it and review it.
 Yeah, thanks a lot for coming, everyone. Special greets to everyone watching this stream at  the moment. Today, we want to talk a little bit around our experiences and story of how  we've applied stream processing over the years, what we've learned, and towards the  end, we're going to have an exciting announcement of how we think we can make stream processing  with Python quite a lot better. So, in short, who are we? We are both two developers at  Wynton, which is a global investment management and data science company. You might have already  talked to us quite a lot at our stand downstairs, and we've been in the game for quite a while  now for 20 years. And our core thing really is that we believe that the scientific method  can be applied profitably to financial markets, which means we run a lot of data processing  on financial and non-financial data and try to make a profit out of that. And as part  of that, we apply a lot of technology and data science, and this talk is really about  one part of that of how we do real-time stream processing. I'm interested, so quick hands  up, please, if you've ever heard of Kafka or done some stream processing. Oh, very good.  That's good. Okay. Then I'll go with this a bit shorter. Basically, what stream processing  is, is that you, instead of doing the classic batch processing where you would go at the  end of the day, and you collect all of your data throughout the day, write it into a big  file, and then run, for example, a huge Spark job or something on that on it, you work on  all your data as a stream, which means data comes in throughout the entire day, and you  process it as it comes in, which is quite nice because then you get real-time and interactive  data. You can put it on a nice dashboard, and often it also makes thinking about your  processing quite a lot simpler because you don't have those random interruptions where  your one batch stops and the next one starts. And one example for the stream processing  in finance, we've actually been doing stream processing for quite a long time now, like  this is just the natural thing you do. For example, if you buy and sell stocks at an  exchange, you can subscribe to their market data feed. And basically what they do is every  time someone buys something or sells something, you get a little message that tells you, for  example, someone at 10, 15 has bought some stock in Apple and they've bought 10 shares  at a price of $144, for example. And you really get this flowing in, you get a massive amount  of messages every second, and then you do various things with it. One of the things  we do, for example, is we show it in various dashboards and it feeds into various dragging  algorithms, and they don't necessarily always care about every single trade that happens,  but like in the question for the talk before, often we want to have like a little bit of  an aggregate, so we want, for example, to know every minute how many Apple has been  traded in total for what average price, for example, and that's something that one of  our stream processors would do. So we take a nice input stream of high frequency events,  run a bidding process, and then group it down into slower streaming events that are closer  to what we really care about. And then our downstream consumers don't really need to do  that much work if they're not interested in all the details. And over the time we found  the streaming model really works well, so now it's expanded from just handling market  data, which is the core financial data of the stock has been traded, to alternative  data, which means we run some web scraping, we look at news, we look at various other  not really core financial data. We've nowadays got an entire office in San Francisco focusing  on this. The other thing that we found that's really useful is just normal business events,  so something happens on the day and you want to make sure that once that happens, this  other job runs or you want to update some monitoring page. And there we've really found  the combination of stream processing where we would send, for example, this job has run  on an event bus in combination with a more traditional database is really useful. So  you would write your normal data into the database and then you write a little notification  onto your event stream and that then can tell various applications to the right thing. For  example, we run monitoring on it, we do our risk management based off those events, we  do our trading decisions and there's various other things. And what I really want to focus  on today is kind of the thing in the middle where we get various raw events in and then  we process it a little bit via various transformations into the format that we really care about  in the end. And if you're talking about stream processing nowadays, what you really mean  usually is Apache Kafka, which is the project which has won the before mentioned streaming  wars really. Apache Kafka is a message broker, which means it's a little bit like a database,  so you install it on a server and it provides you with something called topics where you  would write various data that belongs together on a single topic from a producer and then  various consumers can pick up that data and do things with it. And one important thing  around it is the topics are subdivided into different partitions and that gives you parallelism.  So for example, if you have trades, you would send some of your trades on partition one,  some on partition two, and then you can run multiple consumers, each consumer gets a petition  and then they can share their work a little bit. And the other nice thing around Kafka  is it makes it really easy to, or it has a really nice underlying model of how it deals  with data. So within a partition, every message is ordered and you get it out in exactly the  same order and it's really quite beautiful and elegant how it's implemented. And because  it's so popular, it has huge support around the big data ecosystem. So the talk before  we heard their system can interact with Kafka. There's a bunch of others which can do it.  So there's a lot of projects that have sprung up lately in the last couple of years that  try to make it easier for you to do stream processing on those events. We've looked at  pretty much all of them and we weren't really too happy with them, basically for two main  reasons. If you're interested in details, come talk to me later. I can talk about this  for hours. But really the two core reasons where we had issues with those projects is  that usually they're pretty heavyweight. So kind of if you want to use Spark Streaming  to do your stream processing, you need to run and maintain an entire Spark cluster and  then your code runs inside the Spark cluster and there's quite a bit of overhead in that,  just installing it, setting it up, configuring it correctly and then doing things like scaling  up. And the other thing is pretty much all of those projects are written for the JVM,  which means Python is a bit of a second-class citizen and then doesn't really fit in or  you talk with the actual system via this text JSON input output, which isn't really great  for performance. There's one nice outline. Pretty much one year ago, Confluent and the  creators of Apache Kafka introduced a new API for Kafka that does very similar things  with these other stream processing frameworks do, but does it as just a simple library,  which means it turns around the relationship between your code and the framework. So in  the others, you would submit your code, you need to package it up and then it runs somewhere  inside the Storm server and what you can do with Kafka Streams is it's just a library,  which means the master of everything is still just your normal Python process. You can run  it any way you want. You can run it on AWS. If you auto-scale it, you just auto-scale  it completely the normal way because it's just a normal process. It's also got proper  real event-by-event processing, which means unlike, for example, Spark streaming, you  don't need to batch events up so you can get pretty good latency and it helps you a lot  with all those really tricky things around stream processing that gets difficult if you  want to do some stateful processing, for example, for the binning or if you want to join different  streams together, if you want to do some aggregations and also for tolerance and distributed processing  is solved quite nicely. And it's now part of the main Apache Kafka client, but the big  drop back for us is so far it's Java only and it's not that easy to write bindings to  that Java interface. And that's a bit of a problem for us because at Rintten we are  big Python users. We are around 450 people in total and really at least half of them  know some Python. They're not all developers, but we have lots of data scientists, researchers,  operations people. And if you want to know a little bit more about our Python usage,  we had a talk last year by some of our colleagues that showed a little bit what we do with Python.  Yep. And what we really want is to give all those people who know Python the ability to  do stream processing nicely and easily. And what we've done so far is kind of this hipster  programming, which means every time you do it, you roll it on your own. You just grab  a Kafka client, there's various good ones, and you connect to your cluster. And in four  lines you can connect to it, you subscribe to messages, you get a callback for every  single message. And to get started with, this is actually a really good idea. And we've  been running quite a lot just like this, various processes to update monitoring and so on.  If you want to go down that route, there's two clients I would recommend. There is one  written by Donna Powers. Thanks a lot for writing that, which is a native Python client.  They've written all of the protocol interactions in Python, and they especially have a really  nice Python interface that feels very natural to use it. It's really nice and robust. It  handles, for example, your cluster scaling correctly. And the other recommendation I  would have is the Confluent Kafka client, which is a wrapper around the C library, which  means it's really, really high performance. So we've pumped something like a million  messages per second through it pretty easily. And yeah, they're both good. Since we had  Parsley talking earlier, thanks to them as well. They've also built a Python client,  but I haven't really used it that much. I thought I should mention it. So what we've  done is we've wrote our own streaming framework quite a couple of times. Every time we start  with those 10 lines or four lines, maybe, you start building it, and then you run it  in production. In your demo setup, it all looks fine, and you let it lose and things  slightly break or are slightly wrong because streaming is actually really, really hard.  For example, if you want to do the spinning from earlier, where you say you group some  traits together, then you do an aggregation over them and calculate some average or total.  You really need to be careful that you handle every message exactly once because if you  double count them, your aggregation is just wrong. And that's really something that people  have actually struggled with. Stateful processing is difficult. If you want to do joins or if  you want to cache like the last bits, you kind of run into the same problems that if  you keep a local database as a lookup, you need to be really, really careful that you  don't apply things twice. And that's hard. Then distributing load can be quite difficult  because while the client's supported, usually what you have to do is you need to handle  certain callbacks that get triggered when the rebalancing happens correctly and that  can be quite tricky. It's fine if you've been doing it for a while, but if you're really  just a data scientist who wants to write some algorithm and some code and run that, it can  be quite difficult. And then if you want performance out of your streaming, you usually want to  do micro-batching, which means that you buffer up. You don't do like a database write on every  single message you do, but you group them together and after a while, like for example,  every second, then you process all the messages you got as one chunk and that gives you quite  a lot of speed up. And because we want all of that, we've decided we've started an open  source project on our new open source organization to basically bring Kafka streams, which we  think has solved all of those problems really, really nicely for Java and we're doing a  re-implementation of that just in Python, pure pyphonic, which means you don't have  any dependencies over and above a simple Kafka server. And my colleague Rob will now show  you a little demo of how it really looks like and what it can do.  Hi. Very excited, a little nervous to do a live demo, but it was a... So I'm going to  start...  No. It's good. Sorry.  Now, I'm going to start up all the components required for the application, starting with  ZooKeeper. So Kafka uses ZooKeeper in the background for cluster management, for storing  the topics, leadership elections, for example. So let's start up our ZooKeeper. Start a Kafka  server. Now, I'm going to start just one instance. In production, you really would want, say,  three instances, three replicas for extra resilience and so on. So let's go ahead and  start up Kafka. That's now running. I'm going to start up a consumer. And this is just the  default console consumer that comes with Kafka. It'll take whatever is in the topic,  bin prices. Sorry. Can you see this? Is it big enough?  Great. So it will write out whatever finds in bin prices out to the console. There's  currently nothing in there. We're not running our application just yet. So let's... Now,  just before I... Let's start up the application. There it is. It started. We can see that it's  listening to the prices topic. So I think it was at the fourth, the fifth line down.  It says that it's listening to the prices topic. It will write out to the bin prices  topic. We've currently got no data in the prices topic, though. I'm going to start one more  consumer first. That's here in my Jupyter Notebook. So let's do the imports, set up  Bokeh, set up our consumer. This is very similar to the lines code you just saw, Andreas. Show  you. And our plot. So there's our plot ready. We've got one more thing to do, and that's  just to start up this loop down here, which will loop on the consumer to read in the prices  from the input topic. So let's start it. Now, again, like the previous consumer, there's  no output yet because there are no inputs in there. So one last thing to do, let's get  some inputs. Right. We've got this generator script. And the generator script is going  to generate some prices, just random prices that normally distributed returns for the  prices. So let's start generating some prices. Go back to our application. We can see it's  already begun to process those. Go to this consumer here. We see there's nothing there  yet. There's nothing there because we're binning the prices. Our definition of a bin is when  we reach the end of a minute. For the purpose of the example, in fact, two minutes, we will  push the last value on the bin, the last two values, sorry, the last two minutes out to  the output topic. So let's have a look inside Chrome at our plot. We can see that our input  prices are being parsed or being read in. Our bin prices will appear as these red dots  here. So once we reach two minutes, I have sped this up. You can run the generator in  approximately real time. So one moment more. We should see the dots. There we go. So as  I say, in production, if we were running similar code, we would obviously be interested in  pushing out the bin price every minute. For the purpose of the example here, we've just  waited for two prices to be available before pushing them out. This will just carry on  plotting. We'll just leave it go there. And we'll move back here. You can see in the consumer,  the console consumer here, it's also found the two prices. So these are two perfectly  independent applications. Neither knows about the other, but both are consuming the same  data. So what does the code behind this look like? Here we go. So the first thing you probably  want to do is to write a topology. Our topology comprises at least one source, one sync. The  source is your input topic. So prices, our sync is the output topic, bin prices. And  you will want at least one processor in between. The processor is where the meter your calculation  really takes place. So in here in the binning class is the processor. So we've got some  initialization, fairly standard stuff. And we've had this process function, process method.  The process method will be called for every value that is read in from the input topic  and then we convert it to the output. When we find two, so you can see down here when  our store contains two outputs, so we've obviously transitioned into the third minute, say, we  will call punctuate. Punctuate simply forwards the values, the bin values out to the output  topic. So that's the flow from input to output and what we saw in the bokeh plot in Chrome.  So we've seen the application, we've seen some code. What comes next? So the first thing  and most importantly really is to finish our implementation. We're at a very early stage.  We're at a very early stage but we absolutely encourage contributions. We'd love you to  check out our code, start on GitHub, file some issues, write tests, write documentation,  whatever you like. Just get involved. We'd really like to experiment with a more pythonic  API, in particular for the DSL. DSL is the domain specific language that sits on top  of the streams API in Kafka. We feel that there is very good opportunity to leverage  Python's strengths to make it even easier to set up a topology using the domain specific  language. Optimize, Python is a great language. We all love it. We wouldn't be here if we  didn't. But we're under no illusions. It does have, it can hit limits. And when we hit limits  we will work on optimizing the performance further. This could include batching, it could  include using some of the great Python libraries, NumPy, Pandas are out there, or even leveraging  some new projects like Apache arrow. Lastly, well, lastly on this slide, but really the  roadmap is longer, there are many more advanced features of Kafka. The 011 release recently  introduced exactly one semantics, for example. So yes, as I said before, and I'll repeat  again, get in touch. There's the GitHub page. I can't seem to click on that. So it is there,  the GitHub page. Get involved, check it out. And are there any questions?  Hello, hello, Aya. Thank you very much. Very interesting. I waited for this for half a year  now. So thank you. I will check it out just now. I haven't found it. So if you Google  for this one, it doesn't show up. So we have to do some search engine optimization.  It was out of just today. Okay. Very good. But I already tweeted it. So.  Excellent. Thank you. Questions? Hi. So in the last presentation, we saw Apache storm  and how supposedly nice it is with tracing the workflows and sync errors and stuff like  that. And can you have that easily out of the box with Kafka streams and your library?  It's a bit more lightweight. So parts of them you can have, especially Confluent, is happy  to sell you an entire seat of tools. On the other side, because it's just a library, it's  pretty lightweight and doesn't do like spinning up of your processes, which means that if you  already have existing infrastructure to do all of that, for example, we run a lot of code  on Kubernetes nowadays, you can just do it on there and it's integrated with what you  already have. Thanks. Another one was over there.  Thanks for the way you talk. Have you tried some kinds of tools for the stream processing  part in particular? Not yet. But I would love to talk to you about  that, how you think this could work. Because this is exactly something that we would really  like to make nice and pyphonic. So please come to our booth maybe later.  I would say that if you look at the current code base, it follows the Java code base reasonably  closely, which we felt was a reasonable way to get started, get to learn the code behind  the Java implementation. But yes, it would be fantastic to introduce  new code where we can leverage, as I said, before Python's strengths.  I worked one comment. So if someone is asking yourself, should I use streaming at all? There's  a talk from him tomorrow why you should use streaming.  Thank you. And it's really great to see the project you started, because I would have  also said that this is missing in Python at the moment, so I'm very glad to try this out.  As I looked also at Kafka streams, and one thing I found particularly interesting is  the scaling approach and the state management approach, is that already included in your  approach or is that on the roadmap? The scaling, we do have working in a simple  fashion. So basically, you can run two different Python processes and they will share the  load between them and balance it. The state processing is a bit further out.  So before we want to do that, basically that's what we're working on next.  It's really tricky to implement, which is why we want to put it in this library.  Yeah, it's great.  But it's coming.  So what's the time? I think we are already over, isn't it? No?  So first, it's good that you're making this open source. I'm not familiar with Kafka streams,  but I use Kafka Hector. So I kind of like feel the pain you were describing.  And my question is, I tried to combine RX, Python, these reactive extensions with Kafka.  And it sort of works, but you have to write a lot of code.  I'm not familiar with how Kafka streams work. So it's similar to that concept of having  operation of the streaming or the events that are coming through Kafka and windowing and  stuff like that.  Yeah, it's basically exactly this. So it's the same. You get your stream input object  and then you can map a function onto it or you can group by and you can split it up.  It's basically exactly that.  We have time for maybe two more questions.  So do you use some binary protocol to get events from Kafka? Like is it performant? Is  it fast?  Okay, so to talk to, so basically what this is, this sits on top of an existing Kafka  client and basically takes away all the boilerplate that you have to write. And beneath it, at  the moment, we're using the Confluent Kafka client.  This one, the second one. And basically that's talking just a normal binary Kafka protocol,  but this highly optimized C implementation is really, really fast.  Where I think the current bottleneck would be is where it goes from C implementation,  where you get one callback for every single message into Python code. And basically what  we've seen is that we really need to include some batching that, for example, you get a  lumpy array of all your events back because if you do processing with Python of a million  records per second, if you look at the profiler, really every single if shows up and it's getting  a bit hard to optimize there.  But yeah, in short, yeah, we're using a really optimized Kafka client underneath and it talks  a binary protocol.  Okay, one last question.  Thank you.  We recently implemented the exact same solution at my company, but for Amazon Kinesis, which  is very similar to Kafka.  Yeah.  And I was wondering if you found any way to test your applications using this library  without having the need of a full Kafka installation in your local development environment.  Okay. So I think this library makes it a bit easier because you aren't so low level, so  it's relatively if you write a processor, you have a really nicely defined Python interface  and basically we will test against that.  But I feel your pain around if you stitch more of them together at some point where we've  managed or we've mostly gone for integration tests because there's so many small things  that are really hard to unit test correctly.  I decode Andreas's comments.  I would just say though that the code that I showed the binning.py was the full application.  So it should be not necessarily easy, but not very difficult to abstract that in a way  to make it more amenable to testing because that is everything.  Okay. So thank you very much for your talk and for all the things.  So please start it and review it.
Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_0_1775.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_1776_9983.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_9984_12179.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_12180_12347.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_12348_12551.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_12552_13703.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_13704_15359.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_15360_16931.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_16932_18647.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_18648_18803.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_18804_18851.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_18852_18947.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_18948_19715.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_19716_21575.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_21576_22847.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_22848_23387.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_23388_25067.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_25068_26195.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_26196_27551.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_27552_28883.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_28884_31295.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_31296_33431.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_33432_36923.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_36924_37055.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_37056_37271.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_37272_37391.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_37392_37463.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_37464_38555.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_38556_38627.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_38628_39167.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_39168_39287.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_39288_41435.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_41436_41963.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_41964_42047.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_42048_42131.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_42132_42263.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_42264_42311.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_42312_42383.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_42384_44183.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_44184_44231.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_44232_44843.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_44844_44903.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_44904_44999.png Andreas_Heider__Robert_Wall_-_Taking_the_Hipster_out_of_Streaming_YWobIPRRnbU_45000_46319.png