title: How to load images into a single tensorflow dataset in batches?
body: <p>I have ~1000 images each of which are ~20mb, I have a previous solution where I was able to create a tensorflow batch dataset of images (these images where smaller in size).</p>
  <p>input data for model : image</p>
  <p>output data for model : image</p>
  <p>I used this batch dataset to train the model and it worked fine. This new dataset does not fit in my memory so I created multiple tensorflow datasets like this:</p>
  <ol>
  <li>I loaded n images in 1 batch and saved them as tensorflow batch dataset after processing into a folder &quot;processed&quot;.</li>
  <li>I write multiple batches of n images into the same folder &quot;processed&quot;</li>
  <li>There is a single &quot;dataset_spec.pb&quot; file in this folder</li>
  </ol>
  <p>When I try to load the images for training, only 1 batch of n images is being loaded.
  How to load all the batches for training?
  Is there any other way to load the images while training?
  My dataset does not fall under the <a href="https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator" rel="nofollow noreferrer">ImageDataGenerator</a> flow or <code>flow_from_directory</code> regime.</p>
  <p>I could not find a viable solution to load image (input and target) in batch mode from folders, the solutions I found were usable when the target is a label.
  I am using tensorflow 2.7.0+nv22.2</p>