Scalable, Automated ML Model Monitoring with KFServing and Hopsworks
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Rúa Martínez, Javier de la
In this session, we will present an open-source stream processing architecture, based on Spark Structured Streaming, for automating model monitoring with some experiment results. We use Kafka to log model predictions, KFServing for model serving and a Kubernetes operator for the deployment and configuration of the different components. As for the analysis of inference data, we implemented an extendable monitoring framework on top of Spark Structured Streaming to detect outliers and data drift. In recent years, MLOps has emerged to bring DevOps processes to the machine learning (ML) development process, aiming at more automation in the execution of repetitive tasks and at smoother interoperability between tools. Among the different stages in the ML lifecycle, model monitoring involves the continuous supervision of the model performance over time, involving the combination of techniques in four categories: outlier detection, data drift detection, explainability and adversarial attacks. Nowadays, most of the available model monitoring tools follow a scheduled batch processing approach or analyse model performance using isolated subsets of the inference data. However, for the continuous monitoring of models, stream processing platforms show several advantages, including support for continuous data analytics, scalable processing of large amounts of data and first-class support for window-based aggregations useful for concept drift detection.
 Hello everyone, my name is Javier de Arroa Martinez. I'm an industrial PhD student at KT-8 and software engineer at Logical Clocks. In this session, I'm going to talk about the skin-level automated machine learning model monitoring with KF Serving and Hoppsworks. So first, I will talk about model serving and monitoring and some challenges. Then I will introduce briefly KF Serving model monitoring framework and Hoppsworks. Then I will describe the architecture for model monitoring in a streaming fashion. And finally, a brief demo. So machine learning projects are composed of multiple stages or steps. First, we want to collect some data and perform some future engineering on this data in order to extract meaningful features. Then we want to store these features in a central place, which is commonly the feature store. And then we will create training sets using these features. With these training sets, we will train our models and then we will deploy them into production. And after that, we have to continuously supervise the model behavior. So model serving refers to the packaging and deployment of models to make them as accessible for inference. It serves already existing challenges in traditional software applications such as auto scaling and resource allocation, but it also presents additional machine learning specific challenges. We have to deal with different types of model files and model servers regarding the frameworks used to train the models. And also we want to deploy more complex inference pipelines composed of sequence of models or data access graphs that might include as well transformers that can be used to preprocess the input data before making predictions. So in addition, we want to ensure that we can access external resources such as computed features to be combined with the request to use as input data. On the other hand, model monitoring refers to the continuous supervision of model performance. And this is important since once we train a model and put it into production, its performance might not be as expected in the real world. And this is because real input data contains due to a change of culture or location or malfunctioning measuring devices, for example. So the most common approach is for model monitoring are detecting outliers and data drift, analyzing bias in the dataset and featuring importance or ensuring data quality. So outliers are considered anomalous observations and they are most common metric and the most common metric to detect them, also not the only ones, are distance-based metrics such as, for example, analyzing if some values are CS3 standard deviations more or less from the mean, so farther from the mean. And as for data drift, as for drift detection, we can mainly distinguish between data drift and concept drift, while the first one focuses on the change of statistical properties in the input data, concept drift, focus on the interpretation itself of the concept that we pretend to represent. So the most common algorithm and metrics are error rate-based, such as AdWin, for example, which uses adaptive windows over time to compare statistics and also data distribution-based, such as distance metrics, including Wasserstein distance or Kulba-Kleber distance. So one of the main challenges in model monitoring is that inference data is an unbounded data stream, which is partitioned and handled by different replicas of the model-serving tool when it scales out. So therefore, somehow we need to centralize all the inference data in order to produce global statistics and ensure access to descriptive statistics of the train set so we can compare and identify when the data drifts. So for this reason, streaming platforms are suitable for model monitoring since they provide support for window-in techniques and facilitates as well the computation of aggregations, parallel over windows of different sizes. And depending on the window length, we can detect different types of drift. Therefore also we can use count-based windows when the window sample sizes are not big enough. So one popular tool for model-serving is KF-serving, which is part of Kubeflow and allows us to define inference pipelines composed of a predictor which runs a model-server and transformers to pre- or post-process the inference data. These are the requests and the predictions. So additionally, it provides a logger component which we can send requests and responses to other services for later analysis. So in this case, this analysis is run in a Spark structure streaming job using a model monitoring framework that facilitates the creation of a stream enquiries in Spark that groups data by timestamp, compute statistics, detect outliers, and compare also windows to detect drift. Last but not least, Hopsworks is a data intensive AI platform for the development and operation analysis of machine learning projects at scale. And with Hopsworks, we can easily run ML pipelines, deploy our models, launch Spark jobs, and leverage out-of-the-box features such as provenance that allows us to track down the training set and the features that were used for a specific model. And additionally, we can use the feature store to get the baseline statistics or the feature values in a specific point in time. So this is an overview of the architecture. On the left-hand side, we have Hopsworks, and on the right side, we have Kubernetes. We can use the Hopsworks API to create a serving for a model. And when Hopsworks will create an infinite service object in Kubernetes that we will handle by using the model serving webhook. This webhook is a mutating controller that mutates jobs to ensure default values, for example, or to inject other containers, which are called sidecars and other things. And in this case, we are injecting storage initializer container to download and prepare the model artifact and also inject an inference logger that produces the Kafka messages that are received from K serving. So once all of this is deployed, we can make requests and inference data will be stored in the inference topic. And then we will read or this inference data will be read by the monitoring job, which computes the statistics outlier and drift. So having said that, let's continue with the demo. So for this demo, I used a dataset available in Kaggle, which is related to department stores, and the main goal is to predict weekly sales in these stores. So there are three given notebooks here. The first one, in the first one, I used HSFS, which is a Python library to interact with the Hopsworks feature store. I used this library to create two different feature groups. The first one is sales feature group and the second one is exogenous feature group. In the first one, I add features that are related to the store and the sales, like the sales in the last month or the last year. And in exogenous, I add features that are not directly related to the stores, but they can have an impact somehow, like temperature or employment rate. So here, I read the data frames with Spark, then perform some transformations in the features, then use HSFS to create the feature group metadata in this case and save it. So when we create the features to metadata, we can specify if we want to compute the statistics, histograms or correlations, and also the primary keys for these feature groups. So in the same way, I create here the exogenous feature group as well specifying that I want the statistics to be computed. So after running that, if we go to the Hopsworks UI, to the features that we can see here, that both feature groups are available, if we click on one, we can check the schema of the features that are contained in the feature group, and we can also check the statistics per feature. There are some information regarding data quality, like completeness, uniqueness, distinctness, for example, which is quite handy. Then after that, in the second Jupyter notebook, I use HSFS to create a query that joins features from the two different feature groups. And with this query, I create a data frame of features that I will use later to create the training set. When creating the training set, I specify again that I want the statistics and the distributions to be computed. As well, we can specify which is the label. In this case, I selected five features, which weekly sales is the target to predict, and the other ones are sales last quarter store, temperature, unemployment, and CPA, which stands for Customer Popularity Index. So once we create the training set, we can go back to the UI, and we will see here that the training set is created. And in the same way as with the feature groups, we can see the schema of the training set and the statistics that were computed during creation. So after that, I created a very simple model to predict the weekly sales, and I export it into the Hobswerks registry using Hobs, which is a Python library to interact with Hobswerks API. After that, in the third notebook, I generate some synthetic data based on the baseline statistics of the training set, but the synthetic data, I transform it a little bit, I drift it a little bit, including outliers, and ensuring that data drift will be detected in the different features. So after that, using the Hobs Python library, I create a serving for the model that I already trained, and I specify that I want KF serving as a model serving tool. Then I start the serving and wait until it is running. If we go to the UI, we can see here in model serving that our model is currently running. Then after that, I created the Spark job, the monitoring job. For that, I used a genetic implementation of the model monitoring framework in Scala. This genetic implementation reads configuration from a file that can be certified, for example, as an argument or environment via a word, and it will adapt to the configuration of this file to compute the statistics or concept drift or outliers that you specify. So an example of this configuration file, this one, actually the one that I used in that job. It includes information such as the scheme of the request and the response, which can be retrieved from the feature store. Then we have a configuration about the windows, like the duration, the slide, and watermark delay for the windows that are computed in Spark. Then we can specify which statistics we want to compute. Also which outliers, these ones are based on descriptive statistics and more specifically the max mean and mean value and average. So if the value exceeds the max or is under the mean or three standard deviations far from the mean, it will be considered outlier. Also for drift, we can consider, I specified to compute drift detection using Vaststein, Kulbalk-Liebler, and Jens Stammer with the specific thresholds. Then we have the baselines, descriptive statistics, and the distributions of the features. This information is also obtained from the feature store we saw in the UI. And finally, we can add a configuration regarding from where to read the inference data, in this case from Kafka topic, which name is this one. And then the analysis, we can specify where do we want the statistics that are in the drift, in this case the statistics will be stored in the market files in this directory, the outliers and drift in these two different Kafka topics. So while the Spark monitoring job was running, I generated some traffic making predictions using the synthetic data that I created before. And after that, I can read the target files to see which statistics were computed for the inference data that I generated. So we can see here different windows, these are just examples. We have the window where these statistics were computed, the feature, and the different values for the statistics. Also we have the same information regarding the distribution of the different features, also correlation between features and covariance. And then regarding outliers and data drift, I consume the two different topics to get the outliers that were detected. So we have information about the feature that presented an outlier value, an anomalous value, which value was it, and which type of outlier approach this was considered in the scripted statistics and more specifically the maximum. So this value exceeded the maximum found in the baseline statistics. Also we have information about the request time and when the detection, when the outlier was detected, we can see this quite close to the request time. And then regarding data drift, we have, similarly, the windows where the drift was detected, the feature that was drifted, which metric we used and which value we obtained that was over the threshold we specified in the configuration file. And therefore drift was detected at this point in time, exactly. So, to conclude, we served our model with cave serving and attached a sidecar container to send inference logs to a Kafka topic, then used a streaming job to consume these logs and compute statistics outliers and drift detection. And it is important to consider the trade-off between data exploitation and latencies, since the longer the window, the bigger the sample size, but also the latencies will increase. So these have to be acceptable latencies. And then for further considerations, Spark 3 includes adaptive query execution that allows us to modify the execution plan at runtime, which can be handy to modify the window length if we detect that the sample size are not big enough or the other way around to make the window longer or shorter. And then also we can use continuous triggers. We can leverage continuous triggers to compute outliers much faster, since there are, for example, regarding the three standard deviations from the mean. This is not a windowed approach. So we can use continuous triggers to detect these outliers much faster. Thank you so much for your attention.
 Hello everyone, my name is Javier de Arroa Martinez.  I'm an industrial PhD student at KT-8 and software engineer at Logical Clocks.  In this session, I'm going to talk about the skin-level automated machine learning model  monitoring with KF Serving and Hoppsworks.  So first, I will talk about model serving and monitoring and some challenges.  Then I will introduce briefly KF Serving model monitoring framework and Hoppsworks.  Then I will describe the architecture for model monitoring in a streaming fashion.  And finally, a brief demo.  So machine learning projects are composed of multiple stages or steps.  First, we want to collect some data and perform some future engineering on this data in order  to extract meaningful features.  Then we want to store these features in a central place, which is commonly the feature  store.  And then we will create training sets using these features.  With these training sets, we will train our models and then we will deploy them into production.  And after that, we have to continuously supervise the model behavior.  So model serving refers to the packaging and deployment of models to make them as accessible  for inference.  It serves already existing challenges in traditional software applications such as  auto scaling and resource allocation, but it also presents additional machine learning  specific challenges.  We have to deal with different types of model files and model servers regarding the frameworks  used to train the models.  And also we want to deploy more complex inference pipelines composed of sequence of models or  data access graphs that might include as well transformers that can be used to preprocess  the input data before making predictions.  So in addition, we want to ensure that we can access external resources such as computed  features to be combined with the request to use as input data.  On the other hand, model monitoring refers to the continuous supervision of model performance.  And this is important since once we train a model and put it into production, its performance  might not be as expected in the real world.  And this is because real input data contains due to a change of culture or location or  malfunctioning measuring devices, for example.  So the most common approach is for model monitoring are detecting outliers and data  drift, analyzing bias in the dataset and featuring importance or ensuring data quality.  So outliers are considered anomalous observations and they are most common metric and the most  common metric to detect them, also not the only ones, are distance-based metrics such  as, for example, analyzing if some values are CS3 standard deviations more or less from  the mean, so farther from the mean.  And as for data drift, as for drift detection, we can mainly distinguish between data drift  and concept drift, while the first one focuses on the change of statistical properties in  the input data, concept drift, focus on the interpretation itself of the concept that  we pretend to represent.  So the most common algorithm and metrics are error rate-based, such as AdWin, for example,  which uses adaptive windows over time to compare statistics and also data distribution-based,  such as distance metrics, including Wasserstein distance or Kulba-Kleber distance.  So one of the main challenges in model monitoring is that inference data is an unbounded data  stream, which is partitioned and handled by different replicas of the model-serving tool  when it scales out.  So therefore, somehow we need to centralize all the inference data in order to produce  global statistics and ensure access to descriptive statistics of the train set so we can compare  and identify when the data drifts.  So for this reason, streaming platforms are suitable for model monitoring since they provide  support for window-in techniques and facilitates as well the computation of aggregations, parallel  over windows of different sizes.  And depending on the window length, we can detect different types of drift.  Therefore also we can use count-based windows when the window sample sizes are not big enough.  So one popular tool for model-serving is KF-serving, which is part of Kubeflow and allows us to  define inference pipelines composed of a predictor which runs a model-server and transformers  to pre- or post-process the inference data.  These are the requests and the predictions.  So additionally, it provides a logger component which we can send requests and responses to  other services for later analysis.  So in this case, this analysis is run in a Spark structure streaming job using a model  monitoring framework that facilitates the creation of a stream enquiries in Spark that  groups data by timestamp, compute statistics, detect outliers, and compare also windows to  detect drift.  Last but not least, Hopsworks is a data intensive AI platform for the development and operation  analysis of machine learning projects at scale.  And with Hopsworks, we can easily run ML pipelines, deploy our models, launch Spark jobs, and  leverage out-of-the-box features such as provenance that allows us to track down the training  set and the features that were used for a specific model.  And additionally, we can use the feature store to get the baseline statistics or the feature  values in a specific point in time.  So this is an overview of the architecture.  On the left-hand side, we have Hopsworks, and on the right side, we have Kubernetes.  We can use the Hopsworks API to create a serving for a model.  And when Hopsworks will create an infinite service object in Kubernetes that we will  handle by using the model serving webhook.  This webhook is a mutating controller that mutates jobs to ensure default values, for  example, or to inject other containers, which are called sidecars and other things.  And in this case, we are injecting storage initializer container to download and prepare  the model artifact and also inject an inference logger that produces the Kafka messages that  are received from K serving.  So once all of this is deployed, we can make requests and inference data will be stored  in the inference topic.  And then we will read or this inference data will be read by the monitoring job, which  computes the statistics outlier and drift.  So having said that, let's continue with the demo.  So for this demo, I used a dataset available in Kaggle, which is related to department  stores, and the main goal is to predict weekly sales in these stores.  So there are three given notebooks here.  The first one, in the first one, I used HSFS, which is a Python library to interact with  the Hopsworks feature store.  I used this library to create two different feature groups.  The first one is sales feature group and the second one is exogenous feature group.  In the first one, I add features that are related to the store and the sales, like the  sales in the last month or the last year.  And in exogenous, I add features that are not directly related to the stores, but they  can have an impact somehow, like temperature or employment rate.  So here, I read the data frames with Spark, then perform some transformations in the features,  then use HSFS to create the feature group metadata in this case and save it.  So when we create the features to metadata, we can specify if we want to compute the statistics,  histograms or correlations, and also the primary keys for these feature groups.  So in the same way, I create here the exogenous feature group as well specifying that I want  the statistics to be computed.  So after running that, if we go to the Hopsworks UI, to the features that we can see here,  that both feature groups are available, if we click on one, we can check the schema of  the features that are contained in the feature group, and we can also check the statistics  per feature.  There are some information regarding data quality, like completeness, uniqueness, distinctness,  for example, which is quite handy.  Then after that, in the second Jupyter notebook, I use HSFS to create a query that joins features  from the two different feature groups.  And with this query, I create a data frame of features that I will use later to create  the training set.  When creating the training set, I specify again that I want the statistics and the distributions  to be computed.  As well, we can specify which is the label.  In this case, I selected five features, which weekly sales is the target to predict, and  the other ones are sales last quarter store, temperature, unemployment, and CPA, which  stands for Customer Popularity Index.  So once we create the training set, we can go back to the UI, and we will see here that  the training set is created.  And in the same way as with the feature groups, we can see the schema of the training set  and the statistics that were computed during creation.  So after that, I created a very simple model to predict the weekly sales, and I export  it into the Hobswerks registry using Hobs, which is a Python library to interact with  Hobswerks API.  After that, in the third notebook, I generate some synthetic data based on the baseline  statistics of the training set, but the synthetic data, I transform it a little bit, I drift  it a little bit, including outliers, and ensuring that data drift will be detected in the different  features.  So after that, using the Hobs Python library, I create a serving for the model that I already  trained, and I specify that I want KF serving as a model serving tool.  Then I start the serving and wait until it is running.  If we go to the UI, we can see here in model serving that our model is currently running.  Then after that, I created the Spark job, the monitoring job.  For that, I used a genetic implementation of the model monitoring framework in Scala.  This genetic implementation reads configuration from a file that can be certified, for example,  as an argument or environment via a word, and it will adapt to the configuration of  this file to compute the statistics or concept drift or outliers that you specify.  So an example of this configuration file, this one, actually the one that I used in  that job.  It includes information such as the scheme of the request and the response, which can  be retrieved from the feature store.  Then we have a configuration about the windows, like the duration, the slide, and watermark  delay for the windows that are computed in Spark.  Then we can specify which statistics we want to compute.  Also which outliers, these ones are based on descriptive statistics and more specifically  the max mean and mean value and average.  So if the value exceeds the max or is under the mean or three standard deviations far from  the mean, it will be considered outlier.  Also for drift, we can consider, I specified to compute drift detection using Vaststein,  Kulbalk-Liebler, and Jens Stammer with the specific thresholds.  Then we have the baselines, descriptive statistics, and the distributions of the features.  This information is also obtained from the feature store we saw in the UI.  And finally, we can add a configuration regarding from where to read the inference data, in  this case from Kafka topic, which name is this one.  And then the analysis, we can specify where do we want the statistics that are in the  drift, in this case the statistics will be stored in the market files in this directory,  the outliers and drift in these two different Kafka topics.  So while the Spark monitoring job was running, I generated some traffic making predictions  using the synthetic data that I created before.  And after that, I can read the target files to see which statistics were computed for  the inference data that I generated.  So we can see here different windows, these are just examples.  We have the window where these statistics were computed, the feature, and the different  values for the statistics.  Also we have the same information regarding the distribution of the different features,  also correlation between features and covariance.  And then regarding outliers and data drift, I consume the two different topics to get  the outliers that were detected.  So we have information about the feature that presented an outlier value, an anomalous value,  which value was it, and which type of outlier approach this was considered in the scripted  statistics and more specifically the maximum.  So this value exceeded the maximum found in the baseline statistics.  Also we have information about the request time and when the detection, when the outlier  was detected, we can see this quite close to the request time.  And then regarding data drift, we have, similarly, the windows where the drift was detected,  the feature that was drifted, which metric we used and which value we obtained that was  over the threshold we specified in the configuration file.  And therefore drift was detected at this point in time, exactly.  So, to conclude, we served our model with cave serving and attached a sidecar container  to send inference logs to a Kafka topic, then used a streaming job to consume these logs  and compute statistics outliers and drift detection.  And it is important to consider the trade-off between data exploitation and latencies, since  the longer the window, the bigger the sample size, but also the latencies will increase.  So these have to be acceptable latencies.  And then for further considerations, Spark 3 includes adaptive query execution that allows  us to modify the execution plan at runtime, which can be handy to modify the window length  if we detect that the sample size are not big enough or the other way around to make  the window longer or shorter.  And then also we can use continuous triggers.  We can leverage continuous triggers to compute outliers much faster, since there are, for  example, regarding the three standard deviations from the mean.  This is not a windowed approach.  So we can use continuous triggers to detect these outliers much faster.  Thank you so much for your attention.
ml_model_monitoring_0_5855.png ml_model_monitoring_5856_8015.png ml_model_monitoring_8016_13967.png ml_model_monitoring_13968_15923.png ml_model_monitoring_15924_17675.png ml_model_monitoring_17676_20015.png ml_model_monitoring_20016_21551.png ml_model_monitoring_21552_22559.png ml_model_monitoring_22560_23423.png ml_model_monitoring_23424_24383.png ml_model_monitoring_24384_24515.png ml_model_monitoring_24516_24935.png ml_model_monitoring_24936_27023.png ml_model_monitoring_27024_29387.png ml_model_monitoring_29388_30191.png ml_model_monitoring_30192_30263.png ml_model_monitoring_30264_32999.png ml_model_monitoring_33000_33467.png