title: How to setup a DVC shared cache without git repository between different services in minikube?
body: <p>I need to setup a shared cache in minikube in such a way that different services can use that cache to pull and update DVC models and data needed for training Machine Learning models. The structure of the project is to use 1 pod to periodically update the cache with new models and outputs. Then, multiple pods can read the cache to recreate the updated models and data. So I need to be able to update the local cache directory and pull from it using DVC commands, so that all the services have consistent view on the latest models and data created by a service.</p>
  <p>More specifically, I have a docker image called <code>inference-service</code> that should only <code>dvc pull</code> or some how use the info in the shared dvc cache to get the latest model and data locally in <code>models</code> and <code>data</code> folders (see dockerfile) in minikube. I have another image called <code>test-service</code> that
  runs the ML pipeline using <code>dvc repro</code> which creates the models and data that DVC needs (dvc.yaml) to track and store in the shared cache. So <code>test-service</code> should push created outputs from the ML pipeline into the shared cache so that <code>inference-service</code> can pull it and use it instead of running dvc repro by itself. <code>test-service</code> should only re-train and write the updated models and data into the shared cache while <code>inference-service</code> should only read and recreate the updated/latest models and data from the shared cache.</p>
  <p><em><strong>Problem: the cache does get mounted on the minikube VM, but the inference service does not pull (using <code>dvc pull -f</code>) the data and models after the test service is done with <code>dvc repro</code> and results the following warnings and failures:</strong></em></p>
  <p><em>relevant kubernetes pod log of inference-service</em></p>
  <pre><code>WARNING: Output 'data/processed/train_preprocessed.pkl'(stage: 'preprocess') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
  You can also use `dvc commit preprocess` to associate existing 'data/processed/train_preprocessed.pkl' with stage: 'preprocess'.
  WARNING: Output 'data/processed/validation_preprocessed.pkl'(stage: 'preprocess') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
  You can also use `dvc commit preprocess` to associate existing 'data/processed/validation_preprocessed.pkl' with stage: 'preprocess'.
  WARNING: Output 'data/processed/test_preprocessed.pkl'(stage: 'preprocess') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
  You can also use `dvc commit preprocess` to associate existing 'data/processed/test_preprocessed.pkl' with stage: 'preprocess'.
  WARNING: Output 'data/interim/train_featurized.pkl'(stage: 'featurize') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
  You can also use `dvc commit featurize` to associate existing 'data/interim/train_featurized.pkl' with stage: 'featurize'.
  WARNING: Output 'data/interim/validation_featurized.pkl'(stage: 'featurize') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
  You can also use `dvc commit featurize` to associate existing 'data/interim/validation_featurized.pkl' with stage: 'featurize'.
  WARNING: Output 'data/interim/test_featurized.pkl'(stage: 'featurize') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
  You can also use `dvc commit featurize` to associate existing 'data/interim/test_featurized.pkl' with stage: 'featurize'.
  WARNING: Output 'models/mlb.pkl'(stage: 'featurize') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
  WARNING: Output 'models/tfidf_vectorizer.pkl'(stage: 'featurize') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
  WARNING: Output 'models/model.pkl'(stage: 'train') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
  WARNING: Output 'reports/scores.json'(stage: 'evaluate') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
  WARNING: No file hash info found for '/root/models/model.pkl'. It won't be created.
  WARNING: No file hash info found for '/root/reports/scores.json'. It won't be created.
  WARNING: No file hash info found for '/root/data/processed/train_preprocessed.pkl'. It won't be created.
  WARNING: No file hash info found for '/root/data/processed/validation_preprocessed.pkl'. It won't be created.
  WARNING: No file hash info found for '/root/data/processed/test_preprocessed.pkl'. It won't be created.
  WARNING: No file hash info found for '/root/data/interim/train_featurized.pkl'. It won't be created.
  WARNING: No file hash info found for '/root/data/interim/validation_featurized.pkl'. It won't be created.
  WARNING: No file hash info found for '/root/data/interim/test_featurized.pkl'. It won't be created.
  WARNING: No file hash info found for '/root/models/mlb.pkl'. It won't be created.
  WARNING: No file hash info found for '/root/models/tfidf_vectorizer.pkl'. It won't be created.
  10 files failed
  ERROR: failed to pull data from the cloud - Checkout failed for following targets:
  /root/models/model.pkl
  /root/reports/scores.json
  /root/data/processed/train_preprocessed.pkl
  /root/data/processed/validation_preprocessed.pkl
  /root/data/processed/test_preprocessed.pkl
  /root/data/interim/train_featurized.pkl
  /root/data/interim/validation_featurized.pkl
  /root/data/interim/test_featurized.pkl
  /root/models/mlb.pkl
  /root/models/tfidf_vectorizer.pkl
  Is your cache up to date?
  </code></pre>
  <p><em>relevant kubernetes pod log of test-service</em></p>
  <pre><code>Stage 'preprocess' is cached - skipping run, checking out outputs
  Generating lock file 'dvc.lock'
  Updating lock file 'dvc.lock'
  Stage 'featurize' is cached - skipping run, checking out outputs
  Updating lock file 'dvc.lock'
  Stage 'train' is cached - skipping run, checking out outputs
  Updating lock file 'dvc.lock'
  Stage 'evaluate' is cached - skipping run, checking out outputs
  Updating lock file 'dvc.lock'
  Use `dvc push` to send your updates to remote storage.
  </code></pre>
  <p><strong>Project Tree</strong></p>
  <pre><code>├─ .dvc
  │  ├─ .gitignore
  │  ├─ config
  │  └─ tmp
  ├─ deployment
  │  ├─ docker-compose
  │  │  ├─ docker-compose.yml
  │  ├─ minikube-dep
  │  │  ├─ inference-test-services_dep.yaml
  │  ├─ startup_minikube_with_mount.sh.sh
  ├─ Dockerfile # for inference service
  ├─ dvc-cache # services should push and pull from this cache folder and see this as the DVC repo
  ├- dvc.yaml
  ├- params.yaml
  ├─ src
  │  ├─ build_features.py
  |  ├─ preprocess_data.py
  |  ├─ serve_model.py
  |  ├─ startup.sh  
  |  ├─ requirements.txt
  ├─ test_dep
  │  ├─ .dvc # same as .dvc in the root folder
  |  |  ├─...
  │  ├─ Dockerfile # for test service
  │  ├─ dvc.yaml
  |  ├─ params.yaml
  │  └─ src
  │     ├─ build_features.py # same as root src folder
  |     ├─ preprocess_data.py # same as root src folder
  |     ├─ serve_model.py # same as root src folder
  |     ├─ startup_test.sh  
  |     ├─ requirements.txt  # same as root src folder
  </code></pre>
  <p><strong>dvc.yaml</strong></p>
  <pre><code>stages:
    preprocess:
      cmd: python ${preprocess.script}
      params:
        - preprocess
      deps:
        - ${preprocess.script}
        - ${preprocess.input_train}
        - ${preprocess.input_val}
        - ${preprocess.input_test}
      outs:
        - ${preprocess.output_train}
        - ${preprocess.output_val}
        - ${preprocess.output_test}
    featurize:
      cmd: python ${featurize.script}
      params:
        - preprocess
        - featurize
      deps:
        - ${featurize.script}
        - ${preprocess.output_train}
        - ${preprocess.output_val}
        - ${preprocess.output_test}
      outs:
        - ${featurize.output_train}
        - ${featurize.output_val}
        - ${featurize.output_test}
        - ${featurize.mlb_out}
        - ${featurize.tfidf_vectorizer_out}
    train:
      cmd: python ${train.script}
      params:
        - featurize
        - train
      deps:
        - ${train.script}
        - ${featurize.output_train}
      outs:
        - ${train.model_out}
    evaluate:
      cmd: python ${evaluate.script}
      params:
        - featurize
        - train
        - evaluate
      deps:
        - ${evaluate.script}
        - ${train.model_out}
        - ${featurize.output_val}
      metrics:
        - ${evaluate.scores_path}
  </code></pre>
  <p><strong>params.yaml</strong></p>
  <pre><code>preprocess:
    script: src/preprocess/preprocess_data.py
    input_train: data/raw/train.tsv
    input_val: data/raw/validation.tsv
    input_test: data/raw/test.tsv
    output_train: data/processed/train_preprocessed.pkl
    output_val: data/processed/validation_preprocessed.pkl
    output_test: data/processed/test_preprocessed.pkl
  
  featurize:
    script: src/features/build_features.py
    output_train: data/interim/train_featurized.pkl
    output_val: data/interim/validation_featurized.pkl
    output_test: data/interim/test_featurized.pkl
    mlb_out: models/mlb.pkl
    tfidf_vectorizer_out: models/tfidf_vectorizer.pkl
  
  train:
    script: src/models/train_model.py
    model_out: models/model.pkl
  
  evaluate:
    script: src/models/evaluate_model.py
    scores_path: reports/scores.json
    roc_json: reports/roc_plot.json
    prc_json: reports/prc_plot.json
  </code></pre>