Building Data Workflows with Luigi and Kubernetes
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Chhantyal, Nar Kumar
This talk will focus on how one can build complex data pipelines in Python. I will introduce Luigi and show how it solves problems while running multiple chain of batch jobs like dependency resolution, workflow management, visualisation, failure handling etc. After that, I will present how to package Luigi pipelines as Docker image for easier testing and deployment. Finally, I will go through way to deploy them on Kubernetes cluster, thus making it possible to scale Big Data pipelines on-demand and reduce infrastructure costs. I will also give tips and tricks to make Luigi Scheduler play well with Kubernetes batch execution feature. This talk will be accompanied by demo project. It will be very beneficial for audience who have some experience in running batch jobs (not necessarily in Python), typically people who work in Big Data sphere like data scientists, data engineers, BI devs and software developers. Familiarity with Python is helpful but not needed.
 Hi, hello. Namaste. I'm originally from Nepal, so I'm currently living in Germany and working in Germany. Today I'm going to talk about building data workflows with Luigi and Kubernetes. Before we start, a few things about me. My name is Narkumar. I'm currently working at Brininger. Brininger is traditional fashion house, mostly popular in South and West Germany. And currently, Brininger is expanding in online e-commerce space quite rapidly. I'm working at Brininger with a data team, data lake team. And we use Python, Luigi with Kubernetes, running on Google Cloud. I was a web dev in past life, and then companies came to me and then they wanted me to do data, so I moved to the more data engineering role. You can find me on the internet as my last name. So before I start, can I ask one question? Do you work with data engineering? It's kind of a new term. How many of are you using in data engineering? Okay, okay, slowly, yeah, slowly is growing. Quite a lot more than I expected. How many of are you using Luigi? Whoa. How many of are you using Airflow? Okay, pretty decent. So today, I'll just introduce Luigi because Luigi is pretty small. It's really lightweight. I think if you just see the read me of the Luigi, you'll pretty much get the idea. Then I'll talk about Kubernetes, how you can learn Luigi in Kubernetes. And I think that part is pretty interesting for this crowd. Okay, so just few things about Luigi. It's a workflow pipeline tool. If you are using Airflow, pretty similar to Airflow. It was open source by Spotify. It's already pretty mature. It's pretty old. And you can write basically the data flow or basically pipelines as a normal Python code. So that's always a plus point. It's really lightweight and it comes with basic web UI to basically to see the jobs and stuff like that. It has tons of packages. You can run Hadoop jobs using this as our orchestration tool. You can run BigQuery. You can run AWS stuff if you need to upload some file or if you need to query. Let's say the stuff like this, it already has tons of content packages. So you can do almost everything with really few lines of code. And it doesn't have scheduler, by the way. I'll come back to this point later in my talk. So just to demonstrate, let's make a use case. Let's use case. Let's say you work in a company where they have ice cream franchise. They have a lot of ice cream shops. And the manager or your colleagues or data analysts want to see the daily sales of yesterday. Every morning they come to work and then they want to see what happened yesterday. How was the company sold? So we need to do a few things to achieve this. You have a broad database where all the transactions happen. You want to dump the broad database to somewhere because you don't want to do aggregation on broad database. Otherwise, you can kill the broad database. And then you want to inject somewhere in analytics database. It could be Redshift. It could be BigQuery or anything. It could be PostgreSQL. And then you want to run this analytics database and then update the dashboard and send out to everyone. So since we are in a Python conference, and I know all of you are really good Python developers, so you wrote an awesome Python script and settled using Chrome. So dump sales data, ingest analytics database, and then aggregate data and then cool. Maybe profit. So it looks pretty great. I mean, runs works. Maybe not. We have a couple of issues with this implementation. What happens when your Boston fails? So if you see this here, we settled hourly because we think maybe the dumping database takes a bit of time and then ingest some text bit of time. So we kind of managed our Chrome to start one after another, assuming that the Boston finishes within one hour, second on finishes within one hour and stuff like this. And one hour because everybody is doing big data. I mean, we have big data. So yeah, we have a couple of problems here. What happens when the Boston fails? What if Boston takes longer than one hour? Or if you have to run this same reporting for the last five days or last one month or last one year, which can happen. And how do you see if these jobs are all successful? And somehow if you mistakenly run multiple times, what happens to your dashboard? Is it broken? Or what happens? So since now we know how to use case and we saw the Python implementation, I want to show you the loose implementation of this. I have to open my Python. So yeah, so this is my loose implementation. I put everything in one file. So yeah, I already hear some laugh. I put everything in one file and you can see SQL queries, plain SQL queries. So don't worry about that. What I want to point out is how you can implement some kind of task or some kind of job in loose. I can increase the font size. Is this good enough? So the first thing I wrote is a DOM database task and then I have load to analytics DB task and then I have aggregate task. So the one important thing you see here is load to analytics DB task depends on the previous task. So the DOM database task. And if you see the aggregate task, this depends on the previous one. And the last one depends on the previous one. So this is how you can change the series of bad jobs together in loose. And running this is pretty easy. So I'm currently using PPNB, so I have to trigger like this. So I have to go to that directory and then I can just run. So the output looks pretty a lot of stuff in there, but you can see here we run like four tasks and then we have the report, right? Yeah, so looks pretty okay. And if you see in the UI, this is how it will look like. So first I DOM the database and then loaded to our analytics database and then the aggregation and self-reported study. So we kind of solve our problems that I mentioned earlier here. I'm sorry. What was it? Here. So we solve some of the problems. What happens when Boston fails? They are saying together. So the next one will just stay in waiting state. And we also solve the second one that if Boston takes longer than that, it will just keep waiting, right? Then I'll come to the other points later. So we saw that you can do kind of changing of multiple batch scripts or bad jobs. And then you can run with simple command line use. And since this is a reporting thing that has to be in production, you have to somehow run it. So as I said earlier, Luigi doesn't have scheduler. So you have to use Chrome. Usually Chrome is used. So you need some way to trigger the task. So if I run with Luigi, it would be, oh, where is mine? What? So if I have to run with Chrome, it would look like this. So I was pretty surprised when I saw that Luigi doesn't have its own scheduler because it's pretty common to have scheduler in and this kind of pipeline tool. But it turns out that it's actually not bad. It's a design decision made by Luigi team. Not having scheduler means you are really flexible to do whatever you like. You're really flexible to run from different places. And one of them is from Kubernetes. So that's what I'm going to talk about in the next. So what is Kubernetes or Chrome job in Kubernetes? You can run a lot of stuff in Kubernetes. You can run normal services like web apps, which run 24-7. You can run jobs, which do a particular thing. And then they are done. And then Chrome job is basically jobs scheduled with the Chrome. So the jobs are known also called as run to completion. So Kubernetes runs it and then when there is error or failure, then it reschedules them. So it always tries to complete the job. So we saw earlier that it was pretty easy to run from local, but what about Kubernetes? So I have a simple Kubernetes setup on my local machine. I installed a mini-cube. So I have a mini-cube cluster running. And if you see here, so this was my Luigi code. So what I did is I have built a Docker image out of it, which is pretty easy. Nine lines of code and then you have Docker image. I have uploaded this image on Docker registry and deployed to Kubernetes, which is mini-cube right now on my local. And Kubernetes deployment looks like this. And then running on Kubernetes, you can see the command run Python and Luigi stuff like that. So this is the setup I have here. You can see here I have one Chrome job. And then I also deployed the Luigi demo also on Kubernetes. So you can see here deployment Luigi and the UI of the Luigi. You see this is the UI of the Luigi that I deployed. So all this setup I already did beforehand because we don't have much time. But if you are interested on setup, I have a pros and get ups so you can easily follow it. So let's see. I have one Chrome job here. And I want to run it. So this is scheduled from 7 to 16. So right now it's 16, 19, so it will not run. But I can also manually trigger this task using Chrome, a cube Chrome command line tool. So let's see what happens. So I should see a pod which is in running state. And you see here. And then let's see in our Luigi UI. So you see there are a couple of tasks in pending state. Some of them are already done and there is one running. The aggregate task is in running state. And this is how you can track the progress of the jobs. Some of them are done. And if there is error, you could see here. Other has just waiting here. So looks pretty cool. So we are able to run on Kubernetes and Kubernetes has this Chrome job which creates a job, which creates pods. So it sounds like a bit like a russin doll, but it's pretty easy once you get into this. So that's what the Kubernetes did here. I had deployed the Chrome job and then it created a couple of jobs, actually one job. And then which created a pod, which is still running. So here we are able to run Luigi. Same way I ran on local, but this has quite a bit of benefits. The main benefit being the scalability of your pipeline. So since it's on Kubernetes, you could scale in Leslie. In our case, we use Google Kubernetes engine. That means at midnight or after midnight around two, we have hundreds of jobs running which are automatically scaled up. Kubernetes automatically scales up because the load is really high during that time. And then after all the jobs are done, then all the Kubernetes nodes go down. So we only have there you get two nodes, rest are dynamic scaling. So this is, I personally find really powerful. But apart from this, you get all the benefits of Kubernetes from containerization to easier deployment to flexibility with the infrastructure and stuff like this. So that was it. It's a bit rushed, but you can follow with the example on my GitHub. And then you can, if you're interested, just try this. So I personally find it very powerful, Sarah, that Luigi being lightweight is really easy to containerize and deploy on Kubernetes. As a result, you can build complex batch processes which are easy to scale and maintain. And you get the benefit of both. The pipeline to Luigi and then the infrastructure side of Kubernetes like horizontally scaling and the deployment like I said earlier. So in short, that was it. I have one hidden slide which is my team is hiding. If you like Python, if you like Kubernetes, those kind of stuff, please feel free to talk to me. We are just based in Stuttgart. It's just two hours from here. And we are really looking for developers. With that, any questions? Thanks, Lars. Thanks for the talk. Great. Awesome. Question, if you have users or customers in multiple time zones, do you have an experience like what's the best substitute for Chrome tab to manage tasks in multiple time zones? I personally didn't really have to deal with that because our developers or our users are all in same time zone. So yeah, I have unfortunately no right answer for that. Why gentlemen? I have a question about here. I'm here. You are looking in the wrong direction. So does Luigi support rescuing of jobs? Meaning if the rescuing of jobs. So as a workflow management system, if you are in the middle of your job and something failed, can you go and fix it and just start from where you failed? Yeah, Luigi handles that. In my setup, the setup with the Kubernetes I showed you, Kubernetes actually handles it. But not using the Luigi feature. So whenever something fails, Kubernetes will run again. So we are using that part. But Luigi can also do that. Hi. Thank you for the talk. I was wondering, have you encountered situations where you need different Kubernetes pods to have different rights and how do you manage that? For example, you want one job to run with production rights, but maybe another job to run with dev rights, not the same rights. Yeah. So usually we use service accounts to run jobs. So one job has one service account, the other job has other service accounts. And the service accounts have limited permissions. So if you need a storage account access, you only give a storage account access to this service account. So that's how we do. Thank you. Okay. Thank you for your talk. We are currently in the stage where we are evaluating Luigi versus Airflow. I have to admit, yeah. So I mean, I'm looking for recommendation, you know. So what we saw is that it was really pretty hard to write a code that express nested steps and the complex like ERRG of tasks. We found a way to make it with Airflow. And I guess maybe there is a way to do it with Luigi and we just missed it. Do you know how to do that? Yeah. With Luigi, it's a bit complicated. The branching model is supported, but it's a bit hard. Airflow has definitely better mechanism in this case. And Luigi, we actually don't do that kind of super complicated branching or nested jobs. Okay. And really, really quick. Did you work on a use case where you have a NFS mount into your Kubernetes image? Sorry. I didn't. Did you mount a disk to your Docker image? Instead of the disk, did you mount a disk? Like we have a problem, which is accessing large quantity of data. And we want to move to Docker instead of accessing the local disk. And one barrier that we have is to mount NFS data to the image and access it. My question is, did you have a similar use case? Not really. So with Luigi, we usually only process the smaller data set. And Luigi is mostly for orchestration part. So all the heavy lifting we actually do with other tools like BigQuery and stuff like that. Okay. Thank you. Hi. So I was one of the few hands that were raised when you asked who was using Airflow. So I was just wondering what your reason was for using Luigi versus Airflow, if there's a particular reason. Yeah. So the Luigi and Airflow are both Python thing, but they really have a different use case. Now Airflow is really big and it comes with everything you need. The one main drawback was the Kubernetes support is still not so good. They recently added the Kubernetes support where each task can run on a Kubernetes part. But this is not so stabilized yet. And this is one of the main reasons that we prefer Luigi. Other than that, Airflow is also like if you have your pipelines, you have to copy in some directory. So you cannot really build Docker images and deploy like I did with Luigi. So the packaging and running, deploying Luigi is a lot easier and flexible. And Kubernetes, sorry, Airflow, you are always in the Airflow way of doing things. So it's really heavy, heavy thing. It gives a lot of things, but it's also not that flexible. Hi. Question. Thanks for the talk. How do you handle the logs? So when integrating with Kubernetes, do you get the logs from the pod then in Luigi? So since we use a Google Kubernetes engine, the logs are sent to Stackdriver by default. So yeah, so if you saw the running, Luigi will basically give the yesterday out and which is sent to the Stackdriver. Okay. And then how do you handle the secrets? So if the secrets are pushed to the logs, do you... We use Kubernetes secrets so they are not in the logs. Okay. Okay. Thank you. Is it possible to configure Luigi in a way to run jobs not just based on the Chrome tab, but also, I don't know, some redis message queues or a file being dropped in a directory or something like that? And how does that work? Because you were like starting a Docker container for every run, but I think something would be there for all of the time. Yeah, that's actually a really interesting use case. We are also thinking about that. There is no built-in way to do it, so you have to build something yourself. If you upload a file to yesterday and then you want to trigger Luigi, then you have to build this part yourself. So like I said earlier, running Luigi is just a command line thing, right? So you would run this command when the file is uploaded. So this part you have to implement yourself. There is no default way to do it. Yeah. Thank you for your talk. When you showed your example with the code and the dependencies between the tasks, it looks like the code is kind of coupled strongly to the code that is running. So the Luigi task definition part and the stuff that is running. Is this very flexible or how strong is the window locking if someone decides to use Luigi and then wants to switch to something? Is it complicated to do that? As far as airflow also has similar kind of contrips, so Luigi has really a lot of contrips. So I mean it's an open source, so it's not really a vendor locking thing. That's not really what I meant. It's about how deep is integration of the managing of the task and definition of the dependencies. And the code itself is coupled a lot or are the stuff that you want to do, the scripts that you're running, separate things and you just define the tasks separately and can edit for example some command line options for what you want to run. So easily. Yeah, I think I got your point. So what I do is we usually build one pipeline for certain stuff. Like building one report is one pipeline. So this is one image, Docker image for us. So we have like hundreds of images to handle all the workloads for the company. Does that answer your question? I'm getting a little bit different direction but maybe you take this offline. Yeah, we can also talk later. Okay, we could probably have one more short and quick question. Yes, please. And probably last one. How do you test your little pipelines? Is that integrated with CSUN pipelines? Yes, actually testing Luigi is really, really easy because it's really lightweight. So we use PyTest to test it. We have some extension built to support Luigi but generally it's really, really easy. Just like any other Python code. So thanks for joining me. I really liked it. There was a lot of questions. I was a bit afraid when I submitted the talk thinking that there might not be enough interest but I'm really excited that there were a lot of questions and if you use Python and data engineering I really want to talk to you. Please come to me. Thank you. Thank you.
 Hi, hello.  Namaste.  I'm originally from Nepal, so I'm currently living in Germany and working in Germany.  Today I'm going to talk about building data workflows with Luigi and Kubernetes.  Before we start, a few things about me.  My name is Narkumar.  I'm currently working at Brininger.  Brininger is traditional fashion house, mostly popular in South and West Germany.  And currently, Brininger is expanding in online e-commerce space quite rapidly.  I'm working at Brininger with a data team, data lake team.  And we use Python, Luigi with Kubernetes, running on Google Cloud.  I was a web dev in past life, and then companies came to me and then they wanted me to do data,  so I moved to the more data engineering role.  You can find me on the internet as my last name.  So before I start, can I ask one question?  Do you work with data engineering?  It's kind of a new term.  How many of are you using in data engineering?  Okay, okay, slowly, yeah, slowly is growing.  Quite a lot more than I expected.  How many of are you using Luigi?  Whoa.  How many of are you using Airflow?  Okay, pretty decent.  So today, I'll just introduce Luigi because Luigi is pretty small.  It's really lightweight.  I think if you just see the read me of the Luigi, you'll pretty much get the idea.  Then I'll talk about Kubernetes, how you can learn Luigi in Kubernetes.  And I think that part is pretty interesting for this crowd.  Okay, so just few things about Luigi.  It's a workflow pipeline tool.  If you are using Airflow, pretty similar to Airflow.  It was open source by Spotify.  It's already pretty mature.  It's pretty old.  And you can write basically the data flow or basically pipelines as a normal Python code.  So that's always a plus point.  It's really lightweight and it comes with basic web UI to basically to see the jobs and stuff  like that.  It has tons of packages.  You can run Hadoop jobs using this as our orchestration tool.  You can run BigQuery.  You can run AWS stuff if you need to upload some file or if you need to query.  Let's say the stuff like this, it already has tons of content packages.  So you can do almost everything with really few lines of code.  And it doesn't have scheduler, by the way.  I'll come back to this point later in my talk.  So just to demonstrate, let's make a use case.  Let's use case.  Let's say you work in a company where they have ice cream franchise.  They have a lot of ice cream shops.  And the manager or your colleagues or data analysts want to see the daily sales of yesterday.  Every morning they come to work and then they want to see what happened yesterday.  How was the company sold?  So we need to do a few things to achieve this.  You have a broad database where all the transactions happen.  You want to dump the broad database to somewhere because you don't want to do aggregation on  broad database.  Otherwise, you can kill the broad database.  And then you want to inject somewhere in analytics database.  It could be Redshift.  It could be BigQuery or anything.  It could be PostgreSQL.  And then you want to run this analytics database and then update the dashboard and send out  to everyone.  So since we are in a Python conference, and I know all of you are really good Python developers,  so you wrote an awesome Python script and settled using Chrome.  So dump sales data, ingest analytics database, and then aggregate data and then cool.  Maybe profit.  So it looks pretty great.  I mean, runs works.  Maybe not.  We have a couple of issues with this implementation.  What happens when your Boston fails?  So if you see this here, we settled hourly because we think maybe the dumping database  takes a bit of time and then ingest some text bit of time.  So we kind of managed our Chrome to start one after another, assuming that the Boston  finishes within one hour, second on finishes within one hour and stuff like this.  And one hour because everybody is doing big data.  I mean, we have big data.  So yeah, we have a couple of problems here.  What happens when the Boston fails?  What if Boston takes longer than one hour?  Or if you have to run this same reporting for the last five days or last one month or  last one year, which can happen.  And how do you see if these jobs are all successful?  And somehow if you mistakenly run multiple times, what happens to your dashboard?  Is it broken?  Or what happens?  So since now we know how to use case and we saw the Python implementation, I want to  show you the loose implementation of this.  I have to open my Python.  So yeah, so this is my loose implementation.  I put everything in one file.  So yeah, I already hear some laugh.  I put everything in one file and you can see SQL queries, plain SQL queries.  So don't worry about that.  What I want to point out is how you can implement some kind of task or some kind of job in loose.  I can increase the font size.  Is this good enough?  So the first thing I wrote is a DOM database task and then I have load to analytics DB task  and then I have aggregate task.  So the one important thing you see here is load to analytics DB task depends on the previous  task.  So the DOM database task.  And if you see the aggregate task, this depends on the previous one.  And the last one depends on the previous one.  So this is how you can change the series of bad jobs together in loose.  And running this is pretty easy.  So I'm currently using PPNB, so I have to trigger like this.  So I have to go to that directory and then I can just run.  So the output looks pretty a lot of stuff in there, but you can see here we run like  four tasks and then we have the report, right?  Yeah, so looks pretty okay.  And if you see in the UI, this is how it will look like.  So first I DOM the database and then loaded to our analytics database and then the aggregation  and self-reported study.  So we kind of solve our problems that I mentioned earlier here.  I'm sorry.  What was it?  Here.  So we solve some of the problems.  What happens when Boston fails?  They are saying together.  So the next one will just stay in waiting state.  And we also solve the second one that if Boston takes longer than that, it will just keep  waiting, right?  Then I'll come to the other points later.  So we saw that you can do kind of changing of multiple batch scripts or bad jobs.  And then you can run with simple command line use.  And since this is a reporting thing that has to be in production, you have to somehow run  it.  So as I said earlier, Luigi doesn't have scheduler.  So you have to use Chrome.  Usually Chrome is used.  So you need some way to trigger the task.  So if I run with Luigi, it would be, oh, where is mine?  What?  So if I have to run with Chrome, it would look like this.  So I was pretty surprised when I saw that Luigi doesn't have its own scheduler because  it's pretty common to have scheduler in and this kind of pipeline tool.  But it turns out that it's actually not bad.  It's a design decision made by Luigi team.  Not having scheduler means you are really flexible to do whatever you like.  You're really flexible to run from different places.  And one of them is from Kubernetes.  So that's what I'm going to talk about in the next.  So what is Kubernetes or Chrome job in Kubernetes?  You can run a lot of stuff in Kubernetes.  You can run normal services like web apps, which run 24-7.  You can run jobs, which do a particular thing.  And then they are done.  And then Chrome job is basically jobs scheduled with the Chrome.  So the jobs are known also called as run to completion.  So Kubernetes runs it and then when there is error or failure, then it reschedules them.  So it always tries to complete the job.  So we saw earlier that it was pretty easy to run from local, but what about Kubernetes?  So I have a simple Kubernetes setup on my local machine.  I installed a mini-cube.  So I have a mini-cube cluster running.  And if you see here, so this was my Luigi code.  So what I did is I have built a Docker image out of it, which is pretty easy.  Nine lines of code and then you have Docker image.  I have uploaded this image on Docker registry and deployed to Kubernetes, which is mini-cube  right now on my local.  And Kubernetes deployment looks like this.  And then running on Kubernetes, you can see the command run Python and Luigi stuff like  that.  So this is the setup I have here.  You can see here I have one Chrome job.  And then I also deployed the Luigi demo also on Kubernetes.  So you can see here deployment Luigi and the UI of the Luigi.  You see this is the UI of the Luigi that I deployed.  So all this setup I already did beforehand because we don't have much time.  But if you are interested on setup, I have a pros and get ups so you can easily follow  it.  So let's see.  I have one Chrome job here.  And I want to run it.  So this is scheduled from 7 to 16.  So right now it's 16, 19, so it will not run.  But I can also manually trigger this task using Chrome, a cube Chrome command line tool.  So let's see what happens.  So I should see a pod which is in running state.  And you see here.  And then let's see in our Luigi UI.  So you see there are a couple of tasks in pending state.  Some of them are already done and there is one running.  The aggregate task is in running state.  And this is how you can track the progress of the jobs.  Some of them are done.  And if there is error, you could see here.  Other has just waiting here.  So looks pretty cool.  So we are able to run on Kubernetes and Kubernetes has this Chrome job which creates a job, which  creates pods.  So it sounds like a bit like a russin doll, but it's pretty easy once you get into this.  So that's what the Kubernetes did here.  I had deployed the Chrome job and then it created a couple of jobs, actually one job.  And then which created a pod, which is still running.  So here we are able to run Luigi.  Same way I ran on local, but this has quite a bit of benefits.  The main benefit being the scalability of your pipeline.  So since it's on Kubernetes, you could scale in Leslie.  In our case, we use Google Kubernetes engine.  That means at midnight or after midnight around two, we have hundreds of jobs running which  are automatically scaled up.  Kubernetes automatically scales up because the load is really high during that time.  And then after all the jobs are done, then all the Kubernetes nodes go down.  So we only have there you get two nodes, rest are dynamic scaling.  So this is, I personally find really powerful.  But apart from this, you get all the benefits of Kubernetes from containerization to easier  deployment to flexibility with the infrastructure and stuff like this.  So that was it.  It's a bit rushed, but you can follow with the example on my GitHub.  And then you can, if you're interested, just try this.  So I personally find it very powerful, Sarah, that Luigi being lightweight is really easy  to containerize and deploy on Kubernetes.  As a result, you can build complex batch processes which are easy to scale and maintain.  And you get the benefit of both.  The pipeline to Luigi and then the infrastructure side of Kubernetes like horizontally scaling  and the deployment like I said earlier.  So in short, that was it.  I have one hidden slide which is my team is hiding.  If you like Python, if you like Kubernetes, those kind of stuff, please feel free to talk  to me.  We are just based in Stuttgart.  It's just two hours from here.  And we are really looking for developers.  With that, any questions?  Thanks, Lars.  Thanks for the talk.  Great.  Awesome.  Question, if you have users or customers in multiple time zones, do you have an experience  like what's the best substitute for Chrome tab to manage tasks in multiple time zones?  I personally didn't really have to deal with that because our developers or our users are  all in same time zone.  So yeah, I have unfortunately no right answer for that.  Why gentlemen?  I have a question about here.  I'm here.  You are looking in the wrong direction.  So does Luigi support rescuing of jobs?  Meaning if the rescuing of jobs.  So as a workflow management system, if you are in the middle of your job and something  failed, can you go and fix it and just start from where you failed?  Yeah, Luigi handles that.  In my setup, the setup with the Kubernetes I showed you, Kubernetes actually handles  it.  But not using the Luigi feature.  So whenever something fails, Kubernetes will run again.  So we are using that part.  But Luigi can also do that.  Hi.  Thank you for the talk.  I was wondering, have you encountered situations where you need different Kubernetes pods to  have different rights and how do you manage that?  For example, you want one job to run with production rights, but maybe another job to  run with dev rights, not the same rights.  Yeah.  So usually we use service accounts to run jobs.  So one job has one service account, the other job has other service accounts.  And the service accounts have limited permissions.  So if you need a storage account access, you only give a storage account access to this  service account.  So that's how we do.  Thank you.  Okay.  Thank you for your talk.  We are currently in the stage where we are evaluating Luigi versus Airflow.  I have to admit, yeah.  So I mean, I'm looking for recommendation, you know.  So what we saw is that it was really pretty hard to write a code that express nested steps  and the complex like ERRG of tasks.  We found a way to make it with Airflow.  And I guess maybe there is a way to do it with Luigi and we just missed it.  Do you know how to do that?  Yeah.  With Luigi, it's a bit complicated.  The branching model is supported, but it's a bit hard.  Airflow has definitely better mechanism in this case.  And Luigi, we actually don't do that kind of super complicated branching or nested jobs.  Okay.  And really, really quick.  Did you work on a use case where you have a NFS mount into your Kubernetes image?  Sorry.  I didn't.  Did you mount a disk to your Docker image?  Instead of the disk, did you mount a disk?  Like we have a problem, which is accessing large quantity of data.  And we want to move to Docker instead of accessing the local disk.  And one barrier that we have is to mount NFS data to the image and access it.  My question is, did you have a similar use case?  Not really.  So with Luigi, we usually only process the smaller data set.  And Luigi is mostly for orchestration part.  So all the heavy lifting we actually do with other tools like BigQuery and stuff like that.  Okay.  Thank you.  Hi.  So I was one of the few hands that were raised when you asked who was using Airflow.  So I was just wondering what your reason was for using Luigi versus Airflow, if there's  a particular reason.  Yeah.  So the Luigi and Airflow are both Python thing, but they really have a different use case.  Now Airflow is really big and it comes with everything you need.  The one main drawback was the Kubernetes support is still not so good.  They recently added the Kubernetes support where each task can run on a Kubernetes part.  But this is not so stabilized yet.  And this is one of the main reasons that we prefer Luigi.  Other than that, Airflow is also like if you have your pipelines, you have to copy in some  directory.  So you cannot really build Docker images and deploy like I did with Luigi.  So the packaging and running, deploying Luigi is a lot easier and flexible.  And Kubernetes, sorry, Airflow, you are always in the Airflow way of doing things.  So it's really heavy, heavy thing.  It gives a lot of things, but it's also not that flexible.  Hi.  Question.  Thanks for the talk.  How do you handle the logs?  So when integrating with Kubernetes, do you get the logs from the pod then in Luigi?  So since we use a Google Kubernetes engine, the logs are sent to Stackdriver by default.  So yeah, so if you saw the running, Luigi will basically give the yesterday out and  which is sent to the Stackdriver.  Okay.  And then how do you handle the secrets?  So if the secrets are pushed to the logs, do you...  We use Kubernetes secrets so they are not in the logs.  Okay.  Okay.  Thank you.  Is it possible to configure Luigi in a way to run jobs not just based on the Chrome  tab, but also, I don't know, some redis message queues or a file being dropped in a directory  or something like that?  And how does that work?  Because you were like starting a Docker container for every run, but I think something would  be there for all of the time.  Yeah, that's actually a really interesting use case.  We are also thinking about that.  There is no built-in way to do it, so you have to build something yourself.  If you upload a file to yesterday and then you want to trigger Luigi, then you have to  build this part yourself.  So like I said earlier, running Luigi is just a command line thing, right?  So you would run this command when the file is uploaded.  So this part you have to implement yourself.  There is no default way to do it.  Yeah.  Thank you for your talk.  When you showed your example with the code and the dependencies between the tasks, it  looks like the code is kind of coupled strongly to the code that is running.  So the Luigi task definition part and the stuff that is running.  Is this very flexible or how strong is the window locking if someone decides to use Luigi  and then wants to switch to something?  Is it complicated to do that?  As far as airflow also has similar kind of contrips, so Luigi has really a lot of contrips.  So I mean it's an open source, so it's not really a vendor locking thing.  That's not really what I meant.  It's about how deep is integration of the managing of the task and definition of the  dependencies.  And the code itself is coupled a lot or are the stuff that you want to do, the scripts  that you're running, separate things and you just define the tasks separately and can edit  for example some command line options for what you want to run.  So easily.  Yeah, I think I got your point.  So what I do is we usually build one pipeline for certain stuff.  Like building one report is one pipeline.  So this is one image, Docker image for us.  So we have like hundreds of images to handle all the workloads for the company.  Does that answer your question?  I'm getting a little bit different direction but maybe you take this offline.  Yeah, we can also talk later.  Okay, we could probably have one more short and quick question.  Yes, please.  And probably last one.  How do you test your little pipelines?  Is that integrated with CSUN pipelines?  Yes, actually testing Luigi is really, really easy because it's really lightweight.  So we use PyTest to test it.  We have some extension built to support Luigi but generally it's really, really easy.  Just like any other Python code.  So thanks for joining me.  I really liked it.  There was a lot of questions.  I was a bit afraid when I submitted the talk thinking that there might not be enough interest  but I'm really excited that there were a lot of questions and if you use Python and data  engineering I really want to talk to you.  Please come to me.  Thank you.  Thank you.
Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_0_623.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_624_10307.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_10308_10355.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_10356_10523.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_10524_10619.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_10620_12263.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_12264_12935.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_12936_13079.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_13080_13499.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_13500_13931.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_13932_14903.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_14904_14951.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_14952_15887.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_15888_16247.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_16248_16295.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_16296_18815.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_18816_18863.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_18864_19967.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_19968_20831.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_20832_21143.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_21144_21383.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_21384_21599.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_21600_25703.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_25704_25751.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_25752_28223.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_28224_28331.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_28332_29063.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_29064_30059.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_30060_40439.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_40440_41471.png Nar_Kumar_Chhantyal_-_Building_Data_Workflows_with_Luigi_and_Kuberne_41472_44103.png