<p>I need to setup a shared cache in minikube in such a way that different services can use that cache to pull and update DVC models and data needed for training Machine Learning models. The structure of the project is to use 1 pod to periodically update the cache with new models and outputs. Then, multiple pods can read the cache to recreate the updated models and data. So I need to be able to update the local cache directory and pull from it using DVC commands, so that all the services have consistent view on the latest models and data created by a service.</p>
<p>More specifically, I have a docker image called <code>inference-service</code> that should only <code>dvc pull</code> or some how use the info in the shared dvc cache to get the latest model and data locally in <code>models</code> and <code>data</code> folders (see dockerfile) in minikube. I have another image called <code>test-service</code> that
runs the ML pipeline using <code>dvc repro</code> which creates the models and data that DVC needs (dvc.yaml) to track and store in the shared cache. So <code>test-service</code> should push created outputs from the ML pipeline into the shared cache so that <code>inference-service</code> can pull it and use it instead of running dvc repro by itself. <code>test-service</code> should only re-train and write the updated models and data into the shared cache while <code>inference-service</code> should only read and recreate the updated/latest models and data from the shared cache.</p>
<p><em><strong>Problem: the cache does get mounted on the minikube VM, but the inference service does not pull (using <code>dvc pull -f</code>) the data and models after the test service is done with <code>dvc repro</code> and results the following warnings and failures:</strong></em></p>
<p><em>relevant kubernetes pod log of inference-service</em></p>
<pre><code>WARNING: Output 'data/processed/train_preprocessed.pkl'(stage: 'preprocess') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
You can also use `dvc commit preprocess` to associate existing 'data/processed/train_preprocessed.pkl' with stage: 'preprocess'.
WARNING: Output 'data/processed/validation_preprocessed.pkl'(stage: 'preprocess') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
You can also use `dvc commit preprocess` to associate existing 'data/processed/validation_preprocessed.pkl' with stage: 'preprocess'.
WARNING: Output 'data/processed/test_preprocessed.pkl'(stage: 'preprocess') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
You can also use `dvc commit preprocess` to associate existing 'data/processed/test_preprocessed.pkl' with stage: 'preprocess'.
WARNING: Output 'data/interim/train_featurized.pkl'(stage: 'featurize') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
You can also use `dvc commit featurize` to associate existing 'data/interim/train_featurized.pkl' with stage: 'featurize'.
WARNING: Output 'data/interim/validation_featurized.pkl'(stage: 'featurize') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
You can also use `dvc commit featurize` to associate existing 'data/interim/validation_featurized.pkl' with stage: 'featurize'.
WARNING: Output 'data/interim/test_featurized.pkl'(stage: 'featurize') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
You can also use `dvc commit featurize` to associate existing 'data/interim/test_featurized.pkl' with stage: 'featurize'.
WARNING: Output 'models/mlb.pkl'(stage: 'featurize') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
WARNING: Output 'models/tfidf_vectorizer.pkl'(stage: 'featurize') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
WARNING: Output 'models/model.pkl'(stage: 'train') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
WARNING: Output 'reports/scores.json'(stage: 'evaluate') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
WARNING: No file hash info found for '/root/models/model.pkl'. It won't be created.
WARNING: No file hash info found for '/root/reports/scores.json'. It won't be created.
WARNING: No file hash info found for '/root/data/processed/train_preprocessed.pkl'. It won't be created.
WARNING: No file hash info found for '/root/data/processed/validation_preprocessed.pkl'. It won't be created.
WARNING: No file hash info found for '/root/data/processed/test_preprocessed.pkl'. It won't be created.
WARNING: No file hash info found for '/root/data/interim/train_featurized.pkl'. It won't be created.
WARNING: No file hash info found for '/root/data/interim/validation_featurized.pkl'. It won't be created.
WARNING: No file hash info found for '/root/data/interim/test_featurized.pkl'. It won't be created.
WARNING: No file hash info found for '/root/models/mlb.pkl'. It won't be created.
WARNING: No file hash info found for '/root/models/tfidf_vectorizer.pkl'. It won't be created.
10 files failed
ERROR: failed to pull data from the cloud - Checkout failed for following targets:
/root/models/model.pkl
/root/reports/scores.json
/root/data/processed/train_preprocessed.pkl
/root/data/processed/validation_preprocessed.pkl
/root/data/processed/test_preprocessed.pkl
/root/data/interim/train_featurized.pkl
/root/data/interim/validation_featurized.pkl
/root/data/interim/test_featurized.pkl
/root/models/mlb.pkl
/root/models/tfidf_vectorizer.pkl
Is your cache up to date?
</code></pre>
<p><em>relevant kubernetes pod log of test-service</em></p>
<pre><code>Stage 'preprocess' is cached - skipping run, checking out outputs
Generating lock file 'dvc.lock'
Updating lock file 'dvc.lock'
Stage 'featurize' is cached - skipping run, checking out outputs
Updating lock file 'dvc.lock'
Stage 'train' is cached - skipping run, checking out outputs
Updating lock file 'dvc.lock'
Stage 'evaluate' is cached - skipping run, checking out outputs
Updating lock file 'dvc.lock'
Use `dvc push` to send your updates to remote storage.
</code></pre>
<p><strong>Project Tree</strong></p>
<pre><code>├─ .dvc
│  ├─ .gitignore
│  ├─ config
│  └─ tmp
├─ deployment
│  ├─ docker-compose
│  │  ├─ docker-compose.yml
│  ├─ minikube-dep
│  │  ├─ inference-test-services_dep.yaml
│  ├─ startup_minikube_with_mount.sh.sh
├─ Dockerfile # for inference service
├─ dvc-cache # services should push and pull from this cache folder and see this as the DVC repo
├- dvc.yaml
├- params.yaml
├─ src
│  ├─ build_features.py
|  ├─ preprocess_data.py
|  ├─ serve_model.py
|  ├─ startup.sh  
|  ├─ requirements.txt
├─ test_dep
│  ├─ .dvc # same as .dvc in the root folder
|  |  ├─...
│  ├─ Dockerfile # for test service
│  ├─ dvc.yaml
|  ├─ params.yaml
│  └─ src
│     ├─ build_features.py # same as root src folder
|     ├─ preprocess_data.py # same as root src folder
|     ├─ serve_model.py # same as root src folder
|     ├─ startup_test.sh  
|     ├─ requirements.txt  # same as root src folder
</code></pre>
<p><strong>dvc.yaml</strong></p>
<pre><code>stages:
  preprocess:
    cmd: python ${preprocess.script}
    params:
      - preprocess
    deps:
      - ${preprocess.script}
      - ${preprocess.input_train}
      - ${preprocess.input_val}
      - ${preprocess.input_test}
    outs:
      - ${preprocess.output_train}
      - ${preprocess.output_val}
      - ${preprocess.output_test}
  featurize:
    cmd: python ${featurize.script}
    params:
      - preprocess
      - featurize
    deps:
      - ${featurize.script}
      - ${preprocess.output_train}
      - ${preprocess.output_val}
      - ${preprocess.output_test}
    outs:
      - ${featurize.output_train}
      - ${featurize.output_val}
      - ${featurize.output_test}
      - ${featurize.mlb_out}
      - ${featurize.tfidf_vectorizer_out}
  train:
    cmd: python ${train.script}
    params:
      - featurize
      - train
    deps:
      - ${train.script}
      - ${featurize.output_train}
    outs:
      - ${train.model_out}
  evaluate:
    cmd: python ${evaluate.script}
    params:
      - featurize
      - train
      - evaluate
    deps:
      - ${evaluate.script}
      - ${train.model_out}
      - ${featurize.output_val}
    metrics:
      - ${evaluate.scores_path}
</code></pre>
<p><strong>params.yaml</strong></p>
<pre><code>preprocess:
  script: src/preprocess/preprocess_data.py
  input_train: data/raw/train.tsv
  input_val: data/raw/validation.tsv
  input_test: data/raw/test.tsv
  output_train: data/processed/train_preprocessed.pkl
  output_val: data/processed/validation_preprocessed.pkl
  output_test: data/processed/test_preprocessed.pkl

featurize:
  script: src/features/build_features.py
  output_train: data/interim/train_featurized.pkl
  output_val: data/interim/validation_featurized.pkl
  output_test: data/interim/test_featurized.pkl
  mlb_out: models/mlb.pkl
  tfidf_vectorizer_out: models/tfidf_vectorizer.pkl

train:
  script: src/models/train_model.py
  model_out: models/model.pkl

evaluate:
  script: src/models/evaluate_model.py
  scores_path: reports/scores.json
  roc_json: reports/roc_plot.json
  prc_json: reports/prc_plot.json
</code></pre>

LinkCoder
<p>After running <code>dvc repro</code> in <code>test-service</code>, a new <code>dvc.lock</code> will be created, containing the file hashes relative to your pipeline (i.e. the hash for <code>models/model.pkl</code> etc).</p>
<p>If you're running a shared cache, <code>inference-service</code> should have access to the updated <code>dvc.lock</code>. If that is present, it will be sufficient to run <code>dvc checkout</code> to populate the workspace with the files corresponding to the hashes in the shared cache.</p>
