Monitoring Kubernetes with OMD Labs Edition and Prometheus
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Kraus, Michael
Prometheus is an open source monitoring tool - unlike traditional tools likeNagios, Prometheus implements a white-box monitoring approach: Applicationsactively provide metrics, these metrics are stored in a time-series database,the time-series data is used as a source for generating alerts. Prometheuscomes with a powerful query language allowing for statistical evaluation ofmetrics. Many modern infrastructure components have Prometheus metrics built-in, like Docker's cAdvisor, Kubernetes, or Konsul. Prometheus' servicediscovery mechanisms make monitoring these components very easy. The OMD Labs Edition is a collection of proven monitoring tools like Nagios orIcinga together with newcomers like InfluxDB and Grafana to provide a best-practice monitoring platform with advanced features like multi-tenancy. OMDLabs Edition bundles exclusively open-source tools and is itself developedopen-source. By adding Prometheus to its stack, OMD Labs Edition is also ready for cloud-native applications. As such it eases the integration of traditionalmonitoring with the needs of dynamic cloud monitoring. This talk gives an introduction on how to integrate Kubernetes monitoringusing Prometheus into the OMD Labs Edition.
 Hello, next up is Michael who is going to talk about monitoring Kubernetes using Kubernetes. Hello everyone. As Kyle already said, I want to show you now one possible way to monitor Kubernetes with Prometheus and OMB Labs Edition. You probably heard of already. You get to know what OMB Labs Edition is in a couple of minutes during the talk. First, let me introduce me. I'm Michael. I'm doing monitoring now for about 12 years, mainly using Plano at Nagas. As we learned today in the morning, this kind of old-fashioned Nagas, but it still works some way, sometimes. But one focus is open-source monitoring. I do only open-source monitoring with the current tool set. I'm a senior monitoring consultant at Munich-based console. Just to drop the logo here. They paid my expenses today. And now for the real thing, to the background. I'm a Nagas guy. What should I do with Kubernetes monitoring? Obviously, Nagas doesn't work in such an environment. But our DevOps team at console had the chance to implement a proof of concept of a Kubernetes environment at a customer to be more precise. It is an open-shift environment, but from a monitoring point of view, it's quite comparable at some unknown Munich car manufacturer. So it was a quite nice chance for us. And we have already monitoring instances there running. So it was obvious that we should try to monitor Kubernetes there, but we did not have any clue about how to do that, especially with Nagas, the way to do. So here is where Prometheus comes in. Prometheus seems to be a natural choice when doing Kubernetes monitoring. It has an integrated service discovery for Kubernetes. And it has another example. It retains labels between Kubernetes and Prometheus. For example, when you define a label or an annotation in a Kubernetes definition, you have a good chance to have it retained in your Prometheus metrics. And another reason why using Prometheus is there are a lot of resources, tutorials, blog posts available on the net. For example, Brides Robust Perception blog or Fabian's Coroes blog. And there are tons of examples on GitHub where you can browse through. So I want to start with the actual implementation. We tried. The first building block of this implementation is, as I already told, the Prometheus integrated service discovery of Kubernetes. A very simplified configuration would look like this in your scrape targets definition within Prometheus. Basically, in my demo environment, I used the Prometheus Kubernetes YAML example directly from Prometheus GitHub account. Just left some parts out which are not necessary for the demo. So basically, this is like that. In this diagram, this is a Kubernetes cluster running with some cluster nodes. There is a Prometheus instance deployed within, and this instance talks to the Kubernetes API and gets some metrics, for example, from the advisor or from the API itself. For example, God of Metrics contains AP service data, container CPU statuses, Qubelet, etc. This is the first part of this implementation. The second part is the node exporter. We already heard about node exporter. This is an exporter from Prometheus itself to export hardware and OS metrics, for example, provided by the kernel. In our Kubernetes development here, this node exporter is deployed as a demon set. As a demon set, it runs pretty fast in every cluster node available. If someone decides to add another cluster node, it will get automatically started there. With this annotation, Prometheus will automatically find this new node exporter. When adding new cluster nodes, new cluster instances, no one would have to change something in the configuration. It would simply appear in Prometheus. Here are the node exporter instances as demons that started on the nodes, and running Prometheus discovers these and scrapes its metrics. This provides us with a lot of additional metrics, for example, CPU metrics, disk file system, and so on, from the separate cluster nodes. One third component is CubeSlate metrics. It's a project I found on Kubernetes itself. This one is focused on the health of the API, not the API, but the C advisor metrics that Prometheus already scrapes. It exposes metrics, for example, deployment, health, node health, pot health, all of these internals of a Kubernetes or OpenChift deployment. Here as well, I set the annotation script true so that it gets discovered automatically. This is the diagram for that component. It's deployed as well and gets automatically discovered. Here is again some of the metrics that are now exposed from this. Now we have three types of ingestion, the Kubernetes services discovery itself, the node exporter for the node metrics of the cluster nodes, and the CubeSlate metrics. In a little demo environment here, I can show you these metrics. The environment is based on Minicube. It's a subproject of Kubernetes for running sample Kubernetes on your laptop. The sample config you can get on my GitHub account, links and presentation slides. By the way, you will find on the FOSTEM schedule event page for my talk here when you need it. So try it out. So I have here Minicube running. Stages, for example. It is running. I have already opened the dashboard where you see I have two deployments, CubeSlate metrics, Prometheus as a deployment, and a demon set for the node exporters. In Minicube there is only one cluster node. Therefore, for the pods, I have only one running node exporter pod and one pod for the CubeSlate metrics and one pod for Prometheus itself. So let's have a look at Prometheus. This is the Prometheus instance running on Kubernetes on the Minicube demo environment right now. As you see, it already ingests a lot of metrics of this running Minicube. To be honest, I don't know what to do with this bunch of metrics, but the operations team of Kubernetes probably will know what to do with it. I'm fine with providing these metrics for someone else to know what to do with it. So the list is pretty, pretty large. So you see here from the targets configuration, I have the scrape targets, Kubernetes AP servers, nodes, Kubernetes pods, service endpoints, and Prometheus itself. As already said, this configuration is basically the same as available as an example on the Prometheus GitHub account. So if trying that out or using my list of services. Second. You get some deployment files for your Kubernetes, 01207. This is on my GitHub account, you get three example dashboards, the JSON files for Prefana, and later then the definition for the OMG. For example, in the Prometheus config map is included the configuration for running Prometheus inside after the official semi-official examples. So now we have a Powerpoint list. Now we have a Prometheus running in just a metrics. But we are at a point where I don't know what to do with it now. Because we would need many other components to have a running monitoring system and a user monitoring system. We would need persistent storage for the time series database, for example, for configuration files when needed. We would have to separate instance of an alert manager with its configurations. Obviously we would need Grafana, maybe we would need a push gate reset, typically in the instance. This would look like that in a Kubernetes environment. So deploying all that inside a Kubernetes, I thought that's not my kind of stuff. As well, we already have that. We already have many of these components here in our classical OMD Labs edition monitoring. The OMD Labs edition is basically monitoring in one package. OMD stands for open source monitoring distribution, but distribution is kind of misleading. It's a separate Linux distribution, for example. It's a platform where we bundle stuff that has proven useful for us consultants to get a monitoring up and running in almost no time at a customer. One big point is, of course, it's completely open source. It's based upon the experiences around Nagios monitoring. Nagios is in this package as well as Asinga or Asinga 2 or Neiman as a monitoring course. All the forks are available. It bundles, as I already said, best practices of many years of consulting experience. So we simply wanted to do the same stuff over and over again at a customer. You don't need no root access to run a monitoring site. You only need this for installation of that monitoring package. It's the muster-growsung, the sample solution at this Munich car manufacturer for monitoring projects there. The OMD Labs edition, as the name implies, is developed by Console Labs. Console Labs, you may have noticed it, is a platform from console employees that do open source stuff in their work and their part time. But we are already free for other contributions who want to work with us, who want to do something great. And OMD is not only the only project here. For example, Citrus, Asakoli for end-to-end testing is another project. There's Fabia and stuff, for example, is already as well available in OMD Labs. But it's not that OMD is only a single package for monitoring that would be okay for a fixed starting point. It has already as well in one super nice feature, I think. Sorry for that. You see what we here include. It's a small part of what we ship with OMD. We recognize Asinga, truck as an alternative user interface, influx DB, as well as Prometheus and Grafana is also built in there. And also some monitoring plugins, proven useful, and also we shipping also Ansible, because we regularly need it to do some monitoring tasks. So we decided to put it into one package to minimize configuration needed. And here's the other feature, the killer feature, I think, besides. The site from an OMD is a separate monitoring instance that runs off the same binaries on the file system. Basically, you don't only create my site and have an environment where you can set, for example, your monitoring core to Nagios or to Asinga or to Neiman. You can enable Prometheus or Grafana per instance. You can also copy, for example, the prod instance to stage when you want to test larger changes in your monitoring. Or if you want to do version upgrades of OMD, you can deploy several versions of OMD in parallel. On your server and can selectively upgrade every site to a specific version. So, for example, you could do OMD copy prod to stage, update this new site, test if all of your monitoring stuff still works. And if it does, you could set this one or remove stage production. It's also one command and disable or delete the old production environment. So you're quite safe when doing upgrades, when doing tests. And as already said, you have a possibility to run different configurations on one machine or different custom installations. For example, customer one, customer two, and they are separated completely. So you don't have to care about if customer one sees monitoring information of customer two. More information about OMD Labs Edition you find on labs.console.de. So, back to the implementation. When setting up OMD, we already have Prometheus there. So we could scrape the Prometheus Kubernetes metrics from the OMD. I've decided not to do so. Why? It's quite hard to access pods inside Kubernetes. The SDN makes it quite hard to access some non-privileged port as I found out. And it's also hard to access the API from outside Kubernetes. This is especially true for OpenShift. So you have to be a cluster admin and a cluster admin role to access the API and exposing that from outside with TLS certificates and tokens. It's quite complicated to get this information outside. So I decided to use Federation for that purpose. I will come to that later. Federation basically exposes the metrics from the inside Prometheus. And with a Ragex matching, an outside Prometheus can scrape these metrics. I thought this is a great idea to get something like that running. We have this Kubernetes environment with the inner Prometheus and the outer Prometheus in the OMD environment, and it's generating all Kubernetes metrics from the inside to the outside. I found it worked. So we can see here. I start an OMD environment. So my OMD is now running. So someone will kind of have a flashback. It's a little bit more modern interface for Nagios, but it should be very familiar. So in this example, I have some monitored hosts with Nagios measurements, for example. And then I can access the OMD Prometheus. And here I have running the Kubernetes Federation and the metrics of the inner Prometheus. Two scrape targets set up. They are running straight up and they are delivering metrics. So you should see the same huge list of metrics in the outside. Prometheus. There we are with Prometheus. We also have Grafana here. No. That one. Grafana. This has access to the OMD Prometheus. For example, I've added a node dashboard, which I got from Grafana.net, which shows you the metrics and health state of the cluster node, the node exporter gathered metrics, system load, memory, test usage. Another dashboard here is the Kubernetes dashboard, which shows the running ports, the CPU usage. You see Prometheus is quite high here, but I found out it still works. Memory usage, NetRoute.io. And I bundled as well an extended dashboard where more metrics are provided, for example, system service, and two container CPU usage, for example. These three dashboards are also available on the GitHub repository with the demo data for you to play with. So, at first glance, this looks nice. We have Prometheus. We have data in there. We have a Prometheus running inside Kubernetes where we don't have to care much about. Simply deploy it. It's simply running. There's no storage there. We have an outside Prometheus running within the OMD site, which garters its metrics via federation. At first glance, it looks like everything is working. But there are some issues. Yeah, two weeks ago, at a blog post with Brian, who told me, no, this is not the purpose of federation. One should do aggregation and minimize data when doing aggregation, federation, for example, or putting only relevant metrics out of other instance of Prometheus, pulling all that metrics could cause, for example, race conditions or something like that. So, I decided to try it anyway. It runs for a couple of months, to be honest, in our proof of concept, as well as in our internal testing environment. And I didn't notice any problems. So... Yes. Yes. Let's see. Another issue we're facing, especially this customer, is that the inner Prometheus is now open to be accessed by everyone. This is not allowed in production, in a proof of concept. We can do this, but not in the production environment. So, I still have to figure out how to secure this inner Prometheus and be still able to get metrics out of it. So, this is an open point. I have not looked into it up to now, but we know we cannot go live with this issue. Another issue combining the traditional OMD and Prometheus is, should Nogius alert, should Alert Manager alert when we mix them? How do we get rid of an overhead of configuration? So, when using Nogius, should we route through Alert Manager? Or the other way around, should we gather metrics through Prometheus and create Nogius alerts out of it? So, this is kind of an open task too. As we learned, open long-term storage for Prometheus is in the making. I think Nogius works on influx to be exporting. I think I read something about it. I don't know. I thought it. I don't know. So, our customer wants long-term storage for at least some of its metrics. We could try to do it with a federation of aggregated metrics. Therefore, the customer needs to know what he wants to store for a long time. But that could be a viable way using federation here for one long-term storage for Prometheus for the time being until another solution exists. And the last issue with it is the coverage. In our internal environment, our DevOps team had a severe outage last week. And simply no ports, no deployments worked, nothing worked anymore. And they did not see anything until the Kubernetes cluster was up and running again because all the internal Prometheus was down. No metrics were delivered. So, they decided they wanted to additionally have an external monitoring of crucial concepts, co-components. For example, moving out the node exporter from inside Kubernetes to the machine itself or monitoring machine health traditionally using Nagios. We have not yet decided what to do. They also wanted to have important services like ADCD monitored as well as some main API queries that indicate the overall health state of a Kubernetes cluster. This is the main point which is from the lessons learned here. I think you need some kind of external monitoring. The approach to have an inside Prometheus and gathering data there is nice for the customers, for example, of your Kubernetes environment, where they need to know what their deployments do, if their service is running correctly, but from the point of view from the operation side from the Kubernetes cluster. I think the approach of an inner Prometheus is not really sufficient. You need an external monitoring as well. Be it Prometheus, be it traditional Nagios, that doesn't matter here for us, or indeed, it consists of all of these components. So, it doesn't matter really. So, thanks for watching. So, questions? So, the system is running right now as a proof of concept still? This is running as a proof of concept and in our internal testing environment, yes. It's nothing yet deployed in production. And your external Prometheus server is just one server right now? You're running on one external Prometheus server? Yes. Our OMD environment runs normally at most of our customers on a virtual machine. This is the fact here also. The OMD runs fine on VMware or something like that. Next question? Why do we need two Prometheus instances? One that is inside Kubernetes node and another one inside OMD? Why can't we just have the Kubernetes Prometheus instance feed directly to Grafana? Sorry, I'm just suffering from... Why not just create this inside? Okay. Which slide? Inside. Why not just have the inside Prometheus is the question? This one. Yeah, you could do this as well, but you have two problems, the same problems. When you're Kubernetes cluster doesn't work reliably anymore, you won't see any metrics, you won't have any monitoring in the worst case. So, of course, it's possible to deploy Grafana on the load manager in here. I think it's the most covered way to do that. But this problem is not gone even if you have two instances of Prometheus, right? One outside as well. Still, your problem is not gone. If Kubernetes is down, you still have the same problem. So, the question is why we knew another one if your same set of problems are still existent? Thanks. If this external runs separately, so if this cluster dies, you lose the metrics provided from this one, but you still would have those one that could scrape directly the cluster nodes or monitor crucial components from the Nagios. If you lose everything inside the Kubernetes, in the best case, it doesn't matter. Next question? Going once, going twice. Thank you very much.
 Hello, next up is Michael who is going to talk about monitoring Kubernetes using Kubernetes.  Hello everyone. As Kyle already said, I want to show you now one possible way to monitor Kubernetes with Prometheus and OMB Labs Edition.  You probably heard of already. You get to know what OMB Labs Edition is in a couple of minutes during the talk.  First, let me introduce me. I'm Michael. I'm doing monitoring now for about 12 years, mainly using Plano at Nagas.  As we learned today in the morning, this kind of old-fashioned Nagas, but it still works some way, sometimes.  But one focus is open-source monitoring. I do only open-source monitoring with the current tool set.  I'm a senior monitoring consultant at Munich-based console. Just to drop the logo here. They paid my expenses today.  And now for the real thing, to the background. I'm a Nagas guy. What should I do with Kubernetes monitoring?  Obviously, Nagas doesn't work in such an environment. But our DevOps team at console had the chance to implement a proof of concept of a Kubernetes environment  at a customer to be more precise. It is an open-shift environment, but from a monitoring point of view, it's quite comparable at some unknown Munich car manufacturer.  So it was a quite nice chance for us. And we have already monitoring instances there running. So it was obvious that we should try to monitor Kubernetes there,  but we did not have any clue about how to do that, especially with Nagas, the way to do.  So here is where Prometheus comes in. Prometheus seems to be a natural choice when doing Kubernetes monitoring.  It has an integrated service discovery for Kubernetes. And it has another example. It retains labels between Kubernetes and Prometheus.  For example, when you define a label or an annotation in a Kubernetes definition, you have a good chance to have it retained in your Prometheus metrics.  And another reason why using Prometheus is there are a lot of resources, tutorials, blog posts available on the net. For example, Brides Robust Perception blog or Fabian's Coroes blog.  And there are tons of examples on GitHub where you can browse through.  So I want to start with the actual implementation. We tried.  The first building block of this implementation is, as I already told, the Prometheus integrated service discovery of Kubernetes.  A very simplified configuration would look like this in your scrape targets definition within Prometheus.  Basically, in my demo environment, I used the Prometheus Kubernetes YAML example directly from Prometheus GitHub account. Just left some parts out which are not necessary for the demo.  So basically, this is like that. In this diagram, this is a Kubernetes cluster running with some cluster nodes.  There is a Prometheus instance deployed within, and this instance talks to the Kubernetes API and gets some metrics, for example, from the advisor or from the API itself.  For example, God of Metrics contains AP service data, container CPU statuses, Qubelet, etc.  This is the first part of this implementation.  The second part is the node exporter. We already heard about node exporter.  This is an exporter from Prometheus itself to export hardware and OS metrics, for example, provided by the kernel.  In our Kubernetes development here, this node exporter is deployed as a demon set. As a demon set, it runs pretty fast in every cluster node available.  If someone decides to add another cluster node, it will get automatically started there. With this annotation, Prometheus will automatically find this new node exporter.  When adding new cluster nodes, new cluster instances, no one would have to change something in the configuration. It would simply appear in Prometheus.  Here are the node exporter instances as demons that started on the nodes, and running Prometheus discovers these and scrapes its metrics.  This provides us with a lot of additional metrics, for example, CPU metrics, disk file system, and so on, from the separate cluster nodes.  One third component is CubeSlate metrics. It's a project I found on Kubernetes itself.  This one is focused on the health of the API, not the API, but the C advisor metrics that Prometheus already scrapes.  It exposes metrics, for example, deployment, health, node health, pot health, all of these internals of a Kubernetes or OpenChift deployment.  Here as well, I set the annotation script true so that it gets discovered automatically.  This is the diagram for that component. It's deployed as well and gets automatically discovered.  Here is again some of the metrics that are now exposed from this.  Now we have three types of ingestion, the Kubernetes services discovery itself, the node exporter for the node metrics of the cluster nodes, and the CubeSlate metrics.  In a little demo environment here, I can show you these metrics.  The environment is based on Minicube. It's a subproject of Kubernetes for running sample Kubernetes on your laptop.  The sample config you can get on my GitHub account, links and presentation slides.  By the way, you will find on the FOSTEM schedule event page for my talk here when you need it.  So try it out.  So I have here Minicube running.  Stages, for example. It is running.  I have already opened the dashboard where you see I have two deployments, CubeSlate metrics, Prometheus as a deployment,  and a demon set for the node exporters. In Minicube there is only one cluster node. Therefore, for the pods, I have only one running node exporter pod and one pod for the CubeSlate metrics and one pod for Prometheus itself.  So let's have a look at Prometheus.  This is the Prometheus instance running on Kubernetes on the Minicube demo environment right now.  As you see, it already ingests a lot of metrics of this running Minicube.  To be honest, I don't know what to do with this bunch of metrics, but the operations team of Kubernetes probably will know what to do with it.  I'm fine with providing these metrics for someone else to know what to do with it.  So the list is pretty, pretty large.  So you see here from the targets configuration, I have the scrape targets, Kubernetes AP servers, nodes, Kubernetes pods, service endpoints, and Prometheus itself.  As already said, this configuration is basically the same as available as an example on the Prometheus GitHub account.  So if trying that out or using my list of services.  Second.  You get some deployment files for your Kubernetes, 01207.  This is on my GitHub account, you get three example dashboards, the JSON files for Prefana, and later then the definition for the OMG.  For example, in the Prometheus config map is included the configuration for running Prometheus inside after the official semi-official examples.  So now we have a Powerpoint list.  Now we have a Prometheus running in just a metrics.  But we are at a point where I don't know what to do with it now.  Because we would need many other components to have a running monitoring system and a user monitoring system.  We would need persistent storage for the time series database, for example, for configuration files when needed.  We would have to separate instance of an alert manager with its configurations.  Obviously we would need Grafana, maybe we would need a push gate reset, typically in the instance.  This would look like that in a Kubernetes environment.  So deploying all that inside a Kubernetes, I thought that's not my kind of stuff.  As well, we already have that. We already have many of these components here in our classical OMD Labs edition monitoring.  The OMD Labs edition is basically monitoring in one package.  OMD stands for open source monitoring distribution, but distribution is kind of misleading.  It's a separate Linux distribution, for example.  It's a platform where we bundle stuff that has proven useful for us consultants to get a monitoring up and running in almost no time at a customer.  One big point is, of course, it's completely open source.  It's based upon the experiences around Nagios monitoring.  Nagios is in this package as well as Asinga or Asinga 2 or Neiman as a monitoring course.  All the forks are available.  It bundles, as I already said, best practices of many years of consulting experience.  So we simply wanted to do the same stuff over and over again at a customer.  You don't need no root access to run a monitoring site.  You only need this for installation of that monitoring package.  It's the muster-growsung, the sample solution at this Munich car manufacturer for monitoring projects there.  The OMD Labs edition, as the name implies, is developed by Console Labs.  Console Labs, you may have noticed it, is a platform from console employees that do open source stuff  in their work and their part time.  But we are already free for other contributions who want to work with us, who want to do something great.  And OMD is not only the only project here.  For example, Citrus, Asakoli for end-to-end testing is another project.  There's Fabia and stuff, for example, is already as well available in OMD Labs.  But it's not that OMD is only a single package for monitoring that would be okay for a fixed starting point.  It has already as well in one super nice feature, I think.  Sorry for that. You see what we here include.  It's a small part of what we ship with OMD.  We recognize Asinga, truck as an alternative user interface, influx DB, as well as Prometheus and Grafana is also built in there.  And also some monitoring plugins, proven useful, and also we shipping also Ansible, because we regularly need it to do some monitoring tasks.  So we decided to put it into one package to minimize configuration needed.  And here's the other feature, the killer feature, I think, besides.  The site from an OMD is a separate monitoring instance that runs off the same binaries on the file system.  Basically, you don't only create my site and have an environment where you can set, for example, your monitoring core to Nagios or to Asinga or to Neiman.  You can enable Prometheus or Grafana per instance.  You can also copy, for example, the prod instance to stage when you want to test larger changes in your monitoring.  Or if you want to do version upgrades of OMD, you can deploy several versions of OMD in parallel.  On your server and can selectively upgrade every site to a specific version.  So, for example, you could do OMD copy prod to stage, update this new site, test if all of your monitoring stuff still works.  And if it does, you could set this one or remove stage production.  It's also one command and disable or delete the old production environment.  So you're quite safe when doing upgrades, when doing tests.  And as already said, you have a possibility to run different configurations on one machine or different custom installations.  For example, customer one, customer two, and they are separated completely.  So you don't have to care about if customer one sees monitoring information of customer two.  More information about OMD Labs Edition you find on labs.console.de.  So, back to the implementation.  When setting up OMD, we already have Prometheus there.  So we could scrape the Prometheus Kubernetes metrics from the OMD.  I've decided not to do so. Why?  It's quite hard to access pods inside Kubernetes.  The SDN makes it quite hard to access some non-privileged port as I found out.  And it's also hard to access the API from outside Kubernetes.  This is especially true for OpenShift.  So you have to be a cluster admin and a cluster admin role to access the API and exposing that from outside with TLS certificates and tokens.  It's quite complicated to get this information outside.  So I decided to use Federation for that purpose.  I will come to that later.  Federation basically exposes the metrics from the inside Prometheus.  And with a Ragex matching, an outside Prometheus can scrape these metrics.  I thought this is a great idea to get something like that running.  We have this Kubernetes environment with the inner Prometheus and the outer Prometheus in the OMD environment,  and it's generating all Kubernetes metrics from the inside to the outside.  I found it worked.  So we can see here.  I start an OMD environment.  So my OMD is now running.  So someone will kind of have a flashback.  It's a little bit more modern interface for Nagios, but it should be very familiar.  So in this example, I have some monitored hosts with Nagios measurements, for example.  And then I can access the OMD Prometheus.  And here I have running the Kubernetes Federation and the metrics of the inner Prometheus.  Two scrape targets set up.  They are running straight up and they are delivering metrics.  So you should see the same huge list of metrics in the outside.  Prometheus.  There we are with Prometheus.  We also have Grafana here.  No.  That one.  Grafana.  This has access to the OMD Prometheus.  For example, I've added a node dashboard, which I got from Grafana.net,  which shows you the metrics and health state of the cluster node,  the node exporter gathered metrics, system load, memory, test usage.  Another dashboard here is the Kubernetes dashboard, which shows the running ports, the CPU usage.  You see Prometheus is quite high here, but I found out it still works.  Memory usage, NetRoute.io.  And I bundled as well an extended dashboard where more metrics are provided,  for example, system service, and two container CPU usage, for example.  These three dashboards are also available on the GitHub repository with the demo data for you to play with.  So, at first glance, this looks nice.  We have Prometheus.  We have data in there.  We have a Prometheus running inside Kubernetes where we don't have to care much about.  Simply deploy it.  It's simply running.  There's no storage there.  We have an outside Prometheus running within the OMD site, which garters its metrics via federation.  At first glance, it looks like everything is working.  But there are some issues.  Yeah, two weeks ago, at a blog post with Brian, who told me, no, this is not the purpose of federation.  One should do aggregation and minimize data when doing aggregation, federation, for example,  or putting only relevant metrics out of other instance of Prometheus,  pulling all that metrics could cause, for example, race conditions or something like that.  So, I decided to try it anyway.  It runs for a couple of months, to be honest, in our proof of concept, as well as in our internal testing environment.  And I didn't notice any problems.  So...  Yes.  Yes.  Let's see.  Another issue we're facing, especially this customer,  is that the inner Prometheus is now open to be accessed by everyone.  This is not allowed in production, in a proof of concept.  We can do this, but not in the production environment.  So, I still have to figure out how to secure this inner Prometheus and be still able to get metrics out of it.  So, this is an open point.  I have not looked into it up to now, but we know we cannot go live with this issue.  Another issue combining the traditional OMD and Prometheus is,  should Nogius alert, should Alert Manager alert when we mix them?  How do we get rid of an overhead of configuration?  So, when using Nogius, should we route through Alert Manager?  Or the other way around, should we gather metrics through Prometheus and create Nogius alerts out of it?  So, this is kind of an open task too.  As we learned, open long-term storage for Prometheus is in the making.  I think Nogius works on influx to be exporting.  I think I read something about it.  I don't know.  I thought it.  I don't know.  So, our customer wants long-term storage for at least some of its metrics.  We could try to do it with a federation of aggregated metrics.  Therefore, the customer needs to know what he wants to store for a long time.  But that could be a viable way using federation here for one long-term storage for Prometheus for the time being until another solution exists.  And the last issue with it is the coverage.  In our internal environment, our DevOps team had a severe outage last week.  And simply no ports, no deployments worked, nothing worked anymore.  And they did not see anything until the Kubernetes cluster was up and running again because all the internal Prometheus was down.  No metrics were delivered.  So, they decided they wanted to additionally have an external monitoring of crucial concepts, co-components.  For example, moving out the node exporter from inside Kubernetes to the machine itself or monitoring machine health traditionally using Nagios.  We have not yet decided what to do.  They also wanted to have important services like ADCD monitored as well as some main API queries that indicate the overall health state of a Kubernetes cluster.  This is the main point which is from the lessons learned here.  I think you need some kind of external monitoring.  The approach to have an inside Prometheus and gathering data there is nice for the customers, for example, of your Kubernetes environment,  where they need to know what their deployments do, if their service is running correctly, but from the point of view from the operation side from the Kubernetes cluster.  I think the approach of an inner Prometheus is not really sufficient.  You need an external monitoring as well.  Be it Prometheus, be it traditional Nagios, that doesn't matter here for us, or indeed, it consists of all of these components.  So, it doesn't matter really.  So, thanks for watching.  So, questions?  So, the system is running right now as a proof of concept still?  This is running as a proof of concept and in our internal testing environment, yes.  It's nothing yet deployed in production.  And your external Prometheus server is just one server right now?  You're running on one external Prometheus server?  Yes.  Our OMD environment runs normally at most of our customers on a virtual machine.  This is the fact here also.  The OMD runs fine on VMware or something like that.  Next question?  Why do we need two Prometheus instances?  One that is inside Kubernetes node and another one inside OMD?  Why can't we just have the Kubernetes Prometheus instance feed directly to Grafana?  Sorry, I'm just suffering from...  Why not just create this inside?  Okay.  Which slide?  Inside.  Why not just have the inside Prometheus is the question?  This one.  Yeah, you could do this as well, but you have two problems, the same problems.  When you're Kubernetes cluster doesn't work reliably anymore,  you won't see any metrics, you won't have any monitoring in the worst case.  So, of course, it's possible to deploy Grafana on the load manager in here.  I think it's the most covered way to do that.  But this problem is not gone even if you have two instances of Prometheus, right?  One outside as well.  Still, your problem is not gone.  If Kubernetes is down, you still have the same problem.  So, the question is why we knew another one if your same set of problems are still existent?  Thanks.  If this external runs separately, so if this cluster dies,  you lose the metrics provided from this one,  but you still would have those one that could scrape directly the cluster nodes  or monitor crucial components from the Nagios.  If you lose everything inside the Kubernetes, in the best case, it doesn't matter.  Next question?  Going once, going twice.  Thank you very much.
Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_0_131.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_132_1163.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_1164_1319.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_1320_4595.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_4596_5843.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_5844_6599.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_6600_7091.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_7092_7199.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_7200_8651.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_8652_9383.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_9384_9851.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_9852_11459.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_11460_11771.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_11772_13331.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_13332_13643.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_13644_13775.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_13776_14531.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_14532_14579.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_14580_14711.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_14712_14891.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_14892_15023.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_15024_15515.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_15516_15827.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_15828_16451.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_16452_16511.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_16512_16691.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_16692_16739.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_16740_17075.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_17076_17195.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_17196_17327.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_17328_17975.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_17976_18023.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_18024_18143.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_18144_19259.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_19260_32399.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_32400_32819.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_32820_32903.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_32904_35855.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_35856_38015.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_38016_52943.png Monitoring_Kubernetes_with_OMD_Labs_Edition_and_Prometheus_52944_53095.png