What do we want to monitor? All the databases!
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Gallego, Agustín
Your databases and monitoring are all set up and you've got your MySQL and MongoDB databases figured out - you're monitoring them and everything is fine. You're killing off those occasional monster queries and you have it all in check. But now you've been tasked to keep tabs on that new Cassandra cluster your company has - we'll show you how to incorporate monitoring it into the Percona Monitoring and Management tool and which features enable you to get the best out of any new and existing database you're incorporating. Database problems? Not on your watch. You will also find out how to include other databases into PMM as a handy open source tool you can use to monitor your databases. You're invited to ask questions during this presentation as well as in the Q&A section.
 Hello everyone, my name is Saustin Acheo and I work at Percona in the support team as a support engineer. We are going to be talking about how we can use Percona monitoring and management to monitor any database we want. So briefly, the agenda will be we are going to first talk about PMMs architecture just so we can understand certain base terminology when we say things like PMM admin, PMM clients, external exporters. So this will let us have all the same base knowledge and make sure that we are understanding what we are saying basically. So what is exposed. And then we are going to talk about the external exporters functionality in particular that is the one that is going to allow us to monitor any database that is potentially not supported by PMM already. And how to, after we have the metrics or the data, how we can make sure we are making the most out of it by visualizing it by the dashboards. And then we are briefly going to cover some other, let's say tangentially related topics like custom collectors that will also in some sense will let us, will give us this added functionality, this extra functionality that we need to monitor any database. So what PMM architecture, what is PMM like, let's say, so it follows a client server architecture in which the client is the part that has the exporters that will connect to the database that we want to monitor. And we make sure that the server has the, these raw data that is comprised of the metrics that we get out of the database for the server then to store and manage for end users then when we want to visualize it via the dashboards. So up to recently version 2.12 we used to use Prometheus for managing the metrics, but we are now changing for in favor of Victoria metrics. In short, it is because Victoria metrics supports not only the pool kind of architecture that Prometheus has, but also can push. So what we can do is from the client we can actively send metrics to the server before it used to be more passive, let's say, in which the server was the one in charge of constantly pulling data from the clients. This has a lot of benefits, but we are not going to go into too much depth here, but there is a blog post that covers this in more depth in percona.com. So you can search for Victoria metrics PMM and you will surely find it. And then on the PMM server side, the main components that we need to be aware of, at least for the purpose of this talk is our Grafana, which is the one that is, will be the one we will use for visualizing the metrics. And then Victoria metrics, Clickhouse and PostgreSQL, which are responsible for the metrics, the data related to query analytics, and then storing configuration respectively. So the important one here is of course, Victoria metrics, but it's also good that you know that Clickhouse and PostgreSQL also exist within the PMM server. So what does PMM offer out of the box? It has support for four main, let's say, kinds of databases because Ligno is always also there, but software. So Linux, MySQL, MongoDB and PostgreSQL would be the four, let's say, main branches. Of course, we then have Percona XTB cluster, ProxySQL, Percona server for MySQL, etc. So the point of this slide is if you want to monitor anything that is not here, you will need to do it via this external exporter functionality that we are going to discuss here. So lastly, regarding PMM, if you want to give it a test and you don't want to necessarily install it, we do have an online demo available that you can check. It's a nice way of getting the feeling of what PMM can offer in terms of the, let's say, native support for, as we were discussing before, for Linux, MySQL, etc. So if you're in doubt, you can just click on that link and it will take you to the home dashboard from which you will have access to all PMM functionality. Okay, great. So now that we know a bit about the versatile view of what PMM can offer, let's dive into how we can get the data, the metrics, the data we need from other databases that are not supported natively by PMM. So there are two main, let's say, scraping models for this. And for the purposes of this talk, we can just call them external and external serverless. But know that the serverless keyword here has nothing to do with what you are probably thinking about, like AWS serverless, for instance. This serverless means just that the PMM clients doesn't necessarily have to be executing in the same node as the databases or service we are going to get metrics from. So you can see clearly, the first example here is the PMM client is living in the same node that we want to take metrics from. And in the second one, we have a PMM client running in one node that is going to get metrics from the other node that is surrounding yellow here and named as external service. So what do we get with the external functionality? Okay, so as we were discussing before, the client on the server, in this case, the client, which we are calling PMM agent now. So the client will be running inside the node that has the database and the metrics exporters running. Additionally, in this case, we will get the side benefit, let's say, of having the operating system already monitored when we install the PMM client. So we will get code for free Linux metrics, basically. So we will use the PMM admin add external commands to do it, to do that. And then we will need some other flags that we are going to see now in the following slides. Then we have the external serverless, which as mentioned before, we will need a node for running the client that has access, of course, to the node that is running the database and the exporter. In this case, unless there is an explicit exporter for OS metrics, we won't have this added benefit because, of course, the PMM client is running in another node. For this, we will use the PMM admin add external serverless commands. And again, we are going to see the exact syntax in some of the slides. But you can always tap dash dash help. And in PMM, we have, it helps. It does a good job in what it should do. It can guide you very well just by tapping dash dash help within all the sub commands. You can already get the functionality that you need from the tool. So the basic steps for, let's say we are in the first case where we have the external exporter running within the same database server, the steps that we would need would be, first, of course, install PMM client. Then configure it and by configure, we mean that we make the PMM server note that we are starting a new PMM client. So the only thing that we do is make sure that PMM server knows that there is a new client that is going to do to export metrics. Additionally, as soon as we configure the PMM client, we will have this added benefit of having the operating system already being monitored. So it's going to start a PMM agent that will start scraping metrics from the OS. But then, of course, what we are here for mainly is for the database metrics and how do we do that? We need to explicitly add the external exporter. As mentioned before, we are going to use the PMM admin add external command and then we will need, in this case, we will need to at least add the ports where we have the exporter publishing the metrics. In this case, it was 7070. And then, optionally, we can add a service name and a group. We are going to see in some slides down the line how the service name and the group affects what the data we see from Grafana. But for now, just remember that we have a group that's named Cassandra Cluster and the service name, in this case, that Cassandra knows Dev1. So at this point, we have the exporter running, but we want to check if it's really getting metrics. How can we do that? The first step I would use is just PMM admin list and it will type, it will output all the exporters that are running within that client. And in this case, we see that the Cassandra Cluster external service type is running for the node Dev1. That's the first step. Then I would go to, in Grafana, we have what we call the advanced data exploration dashboard. In it, we have all the metrics from all the nodes that PMM service is gathering data from. So you can just, I'm not sure if it reads well, it may be a bit small, but the metric in this case was Cassandra Clients connected native clients. And then the node was Cassandra Dev1. So we indeed have data that is being collected. In this case, there were connections being made to the Cassandra node. Again, for the external serverless exporter, the steps are fairly the same or very similar. We have to have a PMM client or PMM agent, as we were saying before, installed and configured in a third node or second node that is not the database node. But that we need to make sure that that node can, of course, reach the database and exporting node. So after we have that installed and configured, we can use, as we mentioned before also, the PMM admin add external serverless commands. And in this case, we will, of course, need not only the port, but the address so we know how to reach that node. In the previous command, we didn't need one because we assumed it was localhost, because the external exporter was running within the same node that the PMM client was running. And then again, we have the external name, which was service name before, but it will serve for the same purpose. And then the group, we are going to have the same group for this Cassandra cluster node. As we can see, the tool will give us output on the service ID that it has. And we are going to see how we can use that information on some other slides. So how can we check if the external serverless exporter is running correctly? Again, PMM admin list would be the command to use as a first step. In this case, we can use it from any node because all the nodes will share this information. It would be like the second section of the output from PMM admin list. It will show all the external exporters that are running in this way. So the service ID, we can use as I was saying before. This one is going to be 14 CF something and 131. And as we can see here, we have the external exporter that is 14 CF something something on 131. So this is the way we can map and make sure that it's at least added to the PMM server environment. Now, again, the second step I would follow is the advanced data exploration dashboard. And checking in this case, we can see here it's Cassandra Dev03. And again, the connections to it. But this is an easy way we have to check because this advanced data exploration dashboard can show us every incoming metric that we have. Without prior knowledge, we just on the metric drop down box, we just start typing Cassandra and we will get all metrics that start with Cassandra. Another way we have to check is what we call the PMM inventory. And this is like the central repository where we have information on all services, agents and notes that are currently registered within the PMM server. So if like, like we can see here, we have the Cassandra Dev03. This was the serverless external exporter. And then we have 01 and 02 that these are the external exporters. So these have their own PMM agents running and this doesn't have its own PMM agent running. I'm not sure if it reads either, but in this case, the dash dash group can be useful so that we have some context as to this service name, which is the Cassandra Dev03 and Cassandra Node Dev01, that they are part of the same cluster. So this would be like logically grouping some external services for which otherwise we wouldn't have any information. And then the final thing I wanted to mention in this section was related to where we can get these exporters from. So Prometheus has a very nice and well-documented section on instrumentation and exporters. And in particular, I would use that as a first approach on this. Like if you need an exporter for, let's say, Cassandra, like we were using now, it has the JMX exporter and it is very well-documented and it takes you to links that have proven useful. So that would be the first approach. And then of course, GitHub is a great resource on this and Google is our friends here, so we can just type that search, those keywords and it will search only within the GitHub site. There will be a second approach and then of course, the third one is just Google without the site keyword and see what you get. But with these two pointers, I think that 99% of the cases should be resolved for getting exporters and instructions on how to run them. Typically, it's very, very easy and even now there are some exporters that running Docker, so it's really easy to, at least for having a quick instance to do some testing or proof of concept. And then the next section is okay, now we have the metrics and we need to make something out of them, right? So how can we do it? In BMM, we have the notion of dashboards, which would be like logically grouping series of graphs together. So what does that mean? If you check the top right corner on BMM, this is Grafana, so you will have the services tab which gets you the main four supported software or tools that we were discussing before. Linux, which is the notes overview, MySQL, MongoDB and Postgres. And then if you click on, let's say, notes overview, it will take you to all the dashboards that are relevant to the operating system. So this is the dashboard list that we have for OS metrics. And as you can see, they are really something very specific, right? Even if we are saying notes summary, which is broad and it will of course entice this CPU memory and all the subsystems, but it is detailed in itself because you know that this dashboard will have broad information about the whole server and then if you want to dig in more into, let's say, CPU information, you can use the CPU utilization details dashboard, which will have much more complete set of graphs. So dashboards are really important and are the core of the metrics visualization and at least how we view it in BMM, right? So we have three main options when it comes to presenting information to our users. The first one is, of course, create a new one from scratch. I wouldn't say this is for advanced users, really. It's very easy to add new dashboards to start using at least very basic promql to get some data quickly showing. And then the other two options that we have is we do have a GitHub page with some compatible dashboards, which are not currently included in BMM, but have proven to do a good job at whatever level of metrics showing it worked. So for instance, there was one proof of concept that we worked on with the process exporter. You will find it there. The dashboard for that proof of concept will be there and it generally is well documented so that you can either well documented or has a blog in prokona.com or slash blog that you can use. And then, of course, we have the graphana database that, again, as we were saying before with Prometheus exporters, this is the dashboard counterpart. And it has a great deal of dashboards that the community has worked on. So let me show you a quick example on how would adding a dashboard be in this particular case from the graphana database, but we can also import via adjacent file or text. So as you can see here, my keywords only were Cassandra and then the data source Prometheus and we already have a list of available dashboards. Let's say we're feeling lucky and we just clicked on the first one that appeared there and what we will see is the dashboard ID at the right section and we can just copy that and then easily import a dashboard just with that number. We're going to see that in the following slides, but before I move forward, let me also mention that typically you will have information on this page in the overview section, in particular for this one. I didn't want to extend too much here, but you will get information on the generally authors, including information on versions that are supported, how they tested it, etc. So this is important because different versions may be exporting different kinds of metrics with different names. So if you want full compatibility, make sure you are using the same versions as the authors did. This is what I was referring to with just the ID in PMM, if we click on this plus sign and we go to import, we will see this screen and we can just import Viagrafana.com simply with the dashboard ID that we were mentioning before. And then the second screen presents us with choosing a name and the ID that has to be unique, but Grafana will tell you if it's not in red. And then what I do want to stress here is that in here, the Prometheus data, you will have many, depending on the version used, you will have different ones. You will need to check which one applies for your version, maybe metrics, maybe Prometheus or maybe another kind of exporter if you manually edited it, but just be aware that you can try different ones and see which one works for you. And then finally, let me mention, so this is like the conceptual end of the talk, but then we also mentioned like this tangentially related topics that are custom collectors that also allow us to expand on what PMM already offers by default or out of the box, but it lets us get even more information from our systems. So there are two different custom data collectors that PMM offers and they are the query collectors and the text file collectors. As you can imagine, query collectors are just queries that we log into the databases we are monitoring and get data out of them. And there is a caveat here that only MySQL and Postgres are supported for now. And then the text file collectors allow us to expand much more on the metrics we have by generalizing the Prometheus ingestion just from a text file that conforms to certain specifications. We have support for all the kinds of resolution scraping times. We will see some outputs in some slides going forward. But as we mentioned, the custom queries are from MySQL and Postgres SQL and it will connect to monitor DB instance and get the query results from it. And then the interesting one in this case I think is the custom text file collector. So one possible way is to simply and it's very simple bash and cron and you will already be getting some interesting functionality at it. Of course it's not limited to this. You can use whatever you want that lets you get the data you want and then write to a file. That's all you need to do, just be able to write to a file. So custom queries collector, I'm going to skip this because we are reaching the end of the allowed time. But the text file collector is the one that I find interesting in this case. And the only thing that if you go to this path, you will see an example of the output that Prometheus or Victoria metrics expects to get. If you remove all these, the two comments here, you will have the file as should be. So the metrics that he's expecting is metric name, potentially some labels, the value, and optionally some timestamp. So if you can output a file with this format, we can have it input as metrics from PMM natively. So you don't have to do anything else. The only thing you have to do is just be able to write to this file. And then finally, I will leave some links here to a quote, unquote, real life example in which Peter demonstrated MySQL custom query collectors. And he even got some really nice dashboard that he used to check to have more insight into MySQL memory usage from the different subsystems within MySQL. So this is all. We will now accept some questions, I think. So thank you very much for your time.
 Hello everyone, my name is Saustin Acheo and I work at Percona in the support team as  a support engineer.  We are going to be talking about how we can use Percona monitoring and management to monitor  any database we want.  So briefly, the agenda will be we are going to first talk about PMMs architecture just  so we can understand certain base terminology when we say things like PMM admin, PMM clients,  external exporters.  So this will let us have all the same base knowledge and make sure that we are understanding  what we are saying basically.  So what is exposed.  And then we are going to talk about the external exporters functionality in particular that  is the one that is going to allow us to monitor any database that is potentially not supported  by PMM already.  And how to, after we have the metrics or the data, how we can make sure we are making the  most out of it by visualizing it by the dashboards.  And then we are briefly going to cover some other, let's say tangentially related topics  like custom collectors that will also in some sense will let us, will give us this added  functionality, this extra functionality that we need to monitor any database.  So what PMM architecture, what is PMM like, let's say, so it follows a client server architecture  in which the client is the part that has the exporters that will connect to the database  that we want to monitor.  And we make sure that the server has the, these raw data that is comprised of the metrics  that we get out of the database for the server then to store and manage for end users then  when we want to visualize it via the dashboards.  So up to recently version 2.12 we used to use Prometheus for managing the metrics, but  we are now changing for in favor of Victoria metrics.  In short, it is because Victoria metrics supports not only the pool kind of architecture that  Prometheus has, but also can push.  So what we can do is from the client we can actively send metrics to the server before  it used to be more passive, let's say, in which the server was the one in charge of constantly  pulling data from the clients.  This has a lot of benefits, but we are not going to go into too much depth here, but  there is a blog post that covers this in more depth in percona.com.  So you can search for Victoria metrics PMM and you will surely find it.  And then on the PMM server side, the main components that we need to be aware of, at  least for the purpose of this talk is our Grafana, which is the one that is, will be  the one we will use for visualizing the metrics.  And then Victoria metrics, Clickhouse and PostgreSQL, which are responsible for the  metrics, the data related to query analytics, and then storing configuration respectively.  So the important one here is of course, Victoria metrics, but it's also good that you know  that Clickhouse and PostgreSQL also exist within the PMM server.  So what does PMM offer out of the box?  It has support for four main, let's say, kinds of databases because Ligno is always  also there, but software.  So Linux, MySQL, MongoDB and PostgreSQL would be the four, let's say, main branches.  Of course, we then have Percona XTB cluster, ProxySQL, Percona server for MySQL, etc.  So the point of this slide is if you want to monitor anything that is not here, you  will need to do it via this external exporter functionality that we are going to discuss  here.  So lastly, regarding PMM, if you want to give it a test and you don't want to necessarily  install it, we do have an online demo available that you can check.  It's a nice way of getting the feeling of what PMM can offer in terms of the, let's  say, native support for, as we were discussing before, for Linux, MySQL, etc.  So if you're in doubt, you can just click on that link and it will take you to the home  dashboard from which you will have access to all PMM functionality.  Okay, great.  So now that we know a bit about the versatile view of what PMM can offer, let's dive into  how we can get the data, the metrics, the data we need from other databases that are  not supported natively by PMM.  So there are two main, let's say, scraping models for this.  And for the purposes of this talk, we can just call them external and external serverless.  But know that the serverless keyword here has nothing to do with what you are probably  thinking about, like AWS serverless, for instance.  This serverless means just that the PMM clients doesn't necessarily have to be executing in  the same node as the databases or service we are going to get metrics from.  So you can see clearly, the first example here is the PMM client is living in the same  node that we want to take metrics from.  And in the second one, we have a PMM client running in one node that is going to get metrics  from the other node that is surrounding yellow here and named as external service.  So what do we get with the external functionality?  Okay, so as we were discussing before, the client on the server, in this case, the client,  which we are calling PMM agent now.  So the client will be running inside the node that has the database and the metrics exporters  running.  Additionally, in this case, we will get the side benefit, let's say, of having the operating  system already monitored when we install the PMM client.  So we will get code for free Linux metrics, basically.  So we will use the PMM admin add external commands to do it, to do that.  And then we will need some other flags that we are going to see now in the following slides.  Then we have the external serverless, which as mentioned before, we will need a node for  running the client that has access, of course, to the node that is running the database and  the exporter.  In this case, unless there is an explicit exporter for OS metrics, we won't have this  added benefit because, of course, the PMM client is running in another node.  For this, we will use the PMM admin add external serverless commands.  And again, we are going to see the exact syntax in some of the slides.  But you can always tap dash dash help.  And in PMM, we have, it helps.  It does a good job in what it should do.  It can guide you very well just by tapping dash dash help within all the sub commands.  You can already get the functionality that you need from the tool.  So the basic steps for, let's say we are in the first case where we have the external exporter  running within the same database server, the steps that we would need would be, first,  of course, install PMM client.  Then configure it and by configure, we mean that we make the PMM server note that we are  starting a new PMM client.  So the only thing that we do is make sure that PMM server knows that there is a new client  that is going to do to export metrics.  Additionally, as soon as we configure the PMM client, we will have this added benefit  of having the operating system already being monitored.  So it's going to start a PMM agent that will start scraping metrics from the OS.  But then, of course, what we are here for mainly is for the database metrics and how  do we do that?  We need to explicitly add the external exporter.  As mentioned before, we are going to use the PMM admin add external command and then we  will need, in this case, we will need to at least add the ports where we have the exporter  publishing the metrics.  In this case, it was 7070.  And then, optionally, we can add a service name and a group.  We are going to see in some slides down the line how the service name and the group affects  what the data we see from Grafana.  But for now, just remember that we have a group that's named Cassandra Cluster and the  service name, in this case, that Cassandra knows Dev1.  So at this point, we have the exporter running, but we want to check if it's really getting  metrics.  How can we do that?  The first step I would use is just PMM admin list and it will type, it will output all the  exporters that are running within that client.  And in this case, we see that the Cassandra Cluster external service type is running for  the node Dev1.  That's the first step.  Then I would go to, in Grafana, we have what we call the advanced data exploration dashboard.  In it, we have all the metrics from all the nodes that PMM service is gathering data from.  So you can just, I'm not sure if it reads well, it may be a bit small, but the metric  in this case was Cassandra Clients connected native clients.  And then the node was Cassandra Dev1.  So we indeed have data that is being collected.  In this case, there were connections being made to the Cassandra node.  Again, for the external serverless exporter, the steps are fairly the same or very similar.  We have to have a PMM client or PMM agent, as we were saying before, installed and configured  in a third node or second node that is not the database node.  But that we need to make sure that that node can, of course, reach the database and exporting  node.  So after we have that installed and configured, we can use, as we mentioned before also, the  PMM admin add external serverless commands.  And in this case, we will, of course, need not only the port, but the address so we know  how to reach that node.  In the previous command, we didn't need one because we assumed it was localhost, because  the external exporter was running within the same node that the PMM client was running.  And then again, we have the external name, which was service name before, but it will  serve for the same purpose.  And then the group, we are going to have the same group for this Cassandra cluster node.  As we can see, the tool will give us output on the service ID that it has.  And we are going to see how we can use that information on some other slides.  So how can we check if the external serverless exporter is running correctly?  Again, PMM admin list would be the command to use as a first step.  In this case, we can use it from any node because all the nodes will share this information.  It would be like the second section of the output from PMM admin list.  It will show all the external exporters that are running in this way.  So the service ID, we can use as I was saying before.  This one is going to be 14 CF something and 131.  And as we can see here, we have the external exporter that is 14 CF something something  on 131.  So this is the way we can map and make sure that it's at least added to the PMM server  environment.  Now, again, the second step I would follow is the advanced data exploration dashboard.  And checking in this case, we can see here it's Cassandra Dev03.  And again, the connections to it.  But this is an easy way we have to check because this advanced data exploration dashboard can  show us every incoming metric that we have.  Without prior knowledge, we just on the metric drop down box, we just start typing Cassandra  and we will get all metrics that start with Cassandra.  Another way we have to check is what we call the PMM inventory.  And this is like the central repository where we have information on all services, agents  and notes that are currently registered within the PMM server.  So if like, like we can see here, we have the Cassandra Dev03.  This was the serverless external exporter.  And then we have 01 and 02 that these are the external exporters.  So these have their own PMM agents running and this doesn't have its own PMM agent running.  I'm not sure if it reads either, but in this case, the dash dash group can be useful so  that we have some context as to this service name, which is the Cassandra Dev03 and Cassandra  Node Dev01, that they are part of the same cluster.  So this would be like logically grouping some external services for which otherwise we wouldn't  have any information.  And then the final thing I wanted to mention in this section was related to where we can  get these exporters from.  So Prometheus has a very nice and well-documented section on instrumentation and exporters.  And in particular, I would use that as a first approach on this.  Like if you need an exporter for, let's say, Cassandra, like we were using now, it has the  JMX exporter and it is very well-documented and it takes you to links that have proven  useful.  So that would be the first approach.  And then of course, GitHub is a great resource on this and Google is our friends here, so  we can just type that search, those keywords and it will search only within the GitHub  site.  There will be a second approach and then of course, the third one is just Google without  the site keyword and see what you get.  But with these two pointers, I think that 99% of the cases should be resolved for getting  exporters and instructions on how to run them.  Typically, it's very, very easy and even now there are some exporters that running Docker,  so it's really easy to, at least for having a quick instance to do some testing or proof  of concept.  And then the next section is okay, now we have the metrics and we need to make something  out of them, right?  So how can we do it?  In BMM, we have the notion of dashboards, which would be like logically grouping series  of graphs together.  So what does that mean?  If you check the top right corner on BMM, this is Grafana, so you will have the services  tab which gets you the main four supported software or tools that we were discussing  before.  Linux, which is the notes overview, MySQL, MongoDB and Postgres.  And then if you click on, let's say, notes overview, it will take you to all the dashboards  that are relevant to the operating system.  So this is the dashboard list that we have for OS metrics.  And as you can see, they are really something very specific, right?  Even if we are saying notes summary, which is broad and it will of course entice this  CPU memory and all the subsystems, but it is detailed in itself because you know that  this dashboard will have broad information about the whole server and then if you want  to dig in more into, let's say, CPU information, you can use the CPU utilization details dashboard,  which will have much more complete set of graphs.  So dashboards are really important and are the core of the metrics visualization and  at least how we view it in BMM, right?  So we have three main options when it comes to presenting information to our users.  The first one is, of course, create a new one from scratch.  I wouldn't say this is for advanced users, really.  It's very easy to add new dashboards to start using at least very basic promql to get some  data quickly showing.  And then the other two options that we have is we do have a GitHub page with some compatible  dashboards, which are not currently included in BMM, but have proven to do a good job at  whatever level of metrics showing it worked.  So for instance, there was one proof of concept that we worked on with the process exporter.  You will find it there.  The dashboard for that proof of concept will be there and it generally is well documented  so that you can either well documented or has a blog in prokona.com or slash blog that  you can use.  And then, of course, we have the graphana database that, again, as we were saying before  with Prometheus exporters, this is the dashboard counterpart.  And it has a great deal of dashboards that the community has worked on.  So let me show you a quick example on how would adding a dashboard be in this particular  case from the graphana database, but we can also import via adjacent file or text.  So as you can see here, my keywords only were Cassandra and then the data source Prometheus  and we already have a list of available dashboards.  Let's say we're feeling lucky and we just clicked on the first one that appeared there  and what we will see is the dashboard ID at the right section and we can just copy that  and then easily import a dashboard just with that number.  We're going to see that in the following slides, but before I move forward, let me also mention  that typically you will have information on this page in the overview section, in particular  for this one.  I didn't want to extend too much here, but you will get information on the generally  authors, including information on versions that are supported, how they tested it, etc.  So this is important because different versions may be exporting different kinds of metrics  with different names.  So if you want full compatibility, make sure you are using the same versions as the authors  did.  This is what I was referring to with just the ID in PMM, if we click on this plus sign  and we go to import, we will see this screen and we can just import Viagrafana.com simply  with the dashboard ID that we were mentioning before.  And then the second screen presents us with choosing a name and the ID that has to be  unique, but Grafana will tell you if it's not in red.  And then what I do want to stress here is that in here, the Prometheus data, you will  have many, depending on the version used, you will have different ones.  You will need to check which one applies for your version, maybe metrics, maybe Prometheus  or maybe another kind of exporter if you manually edited it, but just be aware that  you can try different ones and see which one works for you.  And then finally, let me mention, so this is like the conceptual end of the talk, but  then we also mentioned like this tangentially related topics that are custom collectors  that also allow us to expand on what PMM already offers by default or out of the box, but it  lets us get even more information from our systems.  So there are two different custom data collectors that PMM offers and they are the query collectors  and the text file collectors.  As you can imagine, query collectors are just queries that we log into the databases we  are monitoring and get data out of them.  And there is a caveat here that only MySQL and Postgres are supported for now.  And then the text file collectors allow us to expand much more on the metrics we have  by generalizing the Prometheus ingestion just from a text file that conforms to certain  specifications.  We have support for all the kinds of resolution scraping times.  We will see some outputs in some slides going forward.  But as we mentioned, the custom queries are from MySQL and Postgres SQL and it will connect  to monitor DB instance and get the query results from it.  And then the interesting one in this case I think is the custom text file collector.  So one possible way is to simply and it's very simple bash and cron and you will already  be getting some interesting functionality at it.  Of course it's not limited to this.  You can use whatever you want that lets you get the data you want and then write to a  file.  That's all you need to do, just be able to write to a file.  So custom queries collector, I'm going to skip this because we are reaching the end of  the allowed time.  But the text file collector is the one that I find interesting in this case.  And the only thing that if you go to this path, you will see an example of the output  that Prometheus or Victoria metrics expects to get.  If you remove all these, the two comments here, you will have the file as should be.  So the metrics that he's expecting is metric name, potentially some labels, the value,  and optionally some timestamp.  So if you can output a file with this format, we can have it input as metrics from PMM natively.  So you don't have to do anything else.  The only thing you have to do is just be able to write to this file.  And then finally, I will leave some links here to a quote, unquote, real life example  in which Peter demonstrated MySQL custom query collectors.  And he even got some really nice dashboard that he used to check to have more insight  into MySQL memory usage from the different subsystems within MySQL.  So this is all.  We will now accept some questions, I think.  So thank you very much for your time.
nemopmm_0_131.png nemopmm_132_239.png nemopmm_240_599.png nemopmm_600_2807.png nemopmm_2808_2855.png nemopmm_2856_3083.png nemopmm_3084_6971.png nemopmm_6972_8279.png nemopmm_8280_9455.png nemopmm_9456_9719.png nemopmm_9720_9983.png nemopmm_9984_11651.png nemopmm_11652_14807.png nemopmm_14808_17879.png nemopmm_17880_18191.png nemopmm_18192_18755.png nemopmm_18756_19799.png nemopmm_19800_22667.png nemopmm_22668_23675.png nemopmm_23676_23843.png nemopmm_23844_24287.png nemopmm_24288_26435.png nemopmm_26436_27455.png nemopmm_27456_30083.png nemopmm_30084_30227.png nemopmm_30228_30287.png nemopmm_30288_30479.png nemopmm_30480_33275.png nemopmm_33276_36431.png nemopmm_36432_36875.png nemopmm_36876_38639.png nemopmm_38640_39179.png nemopmm_39180_40367.png nemopmm_40368_40499.png nemopmm_40500_40607.png nemopmm_40608_40955.png nemopmm_40956_42719.png nemopmm_42720_43811.png nemopmm_43812_44087.png nemopmm_44088_45443.png nemopmm_45444_45815.png nemopmm_45816_46187.png nemopmm_46188_46391.png nemopmm_46392_46463.png nemopmm_46464_46583.png