<p>I have 1TB of images stored in GCS (data is splitted into 3 classes). I want to train custom Tensor Flow model on this data in Kubeflow.
Currently, I have pipeline components for training and persisting the model but I don't know how to correctly feed this data into the classifier.</p>

<p>It seems to me like downloading this data from GCS (gsutil cp / something other) every time I run (possibly with fail) the pipeline is not a proper way to do this. </p>

<p>How to use large volumes of data in Kubeflow pipelines without downloading them every time? How to express access to this data using Kubeflow DSL?</p>

<p>Additionally, if your data <em>is</em> in GCS, then TensorFlow supports the ability to access data in (and write to) GCS.
The <a href="https://www.tensorflow.org/guide/performance/datasets" rel="nofollow noreferrer">tf.data api</a> lets you set up a performant data input pipeline.</p>
