title: Keras 'flow_from_directory' is Extremely Slow
body: <p>I have an image classification project I'm working on that has 17,255 images and 49 categories.  This is a proof-of-concept exercise.  The actual end-product will involve somewhere between 100,000 and 500,000 images.  Given the large number of images and their size, I decided to look into the Keras 'flow_from_directory' functionality.</p>
  
  <p>When I initially ran the code below against the entire image set it ran for over an hour without completing.  To bound the problem I created a subset of images and directory categories.  For a hundred or so images the script completed in around 30 seconds.</p>
  
  <p>When I upped things to around 1,400 images the script took over 30 minutes to complete.  That would be 2,800 images per hour or over 6 hours for my dataset (feel free to check my math).  And that is just the data generation part, not including any actual training</p>
  
  <p>I'm running on a Google instance with 8 CPUs and 50 Gig of RAM.  The CPU and memory usage when the script was running was minimal so hardware isn't the problem.</p>
  
  <p>Machine specs:</p>
  
  <pre><code>instance-4 &gt; uname -a
  Linux instance-4 4.4.0-109-generic #132~14.04.1-Ubuntu SMP Tue Jan 9 21:46:42 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
  </code></pre>
  
  <p>Python specs:</p>
  
  <pre><code>&gt;&gt;&gt; print(keras.__version__)
  2.1.2
  &gt;&gt;&gt; import tensorflow as tf
  &gt;&gt;&gt; print(tf.__version__)
  1.4.1
  &gt;&gt;&gt; 
  instance-4 &gt; python -V
  Python 3.6.3 :: Anaconda, Inc.
  </code></pre>
  
  <p>The file store is Google Cloud.  The sample directory is:</p>
  
  <pre><code>instance-4 &gt; ls -1 ./data/val
  cat1
  cat2
  cat3
  cat4
  cat5
  cat6
  </code></pre>
  
  <p>Within each directory/category are symbolic links to the actual image files (also on Google Cloud).</p>
  
  <p>It occurred to me that the links could be the issue, but when I ran with a hundred or so image files the performance was about the same as with symbolic links (~ 30 sec)</p>
  
  <p><strong>So my question is</strong>: Am I doing something wrong or is the Keras 'flow_from_directory' just incapable of handling a large number of images (despite the advertising/documentation)?</p>
  
  <p>Sample code:</p>
  
  <pre><code>#!/usr/bin/env python
  
  import warnings
  
  #... Supress TensorFlow warnings
  with warnings.catch_warnings():
      warnings.simplefilter("ignore")
      import keras
  
  from keras.preprocessing.image import ImageDataGenerator
  
  from datetime import datetime
  import time
  
  test_datagen = ImageDataGenerator()
  validation_dir = './data/val'
  
  start_time = time.time()
  print( str(datetime.now()) )
  
  validation_generator = test_datagen.flow_from_directory(
          validation_dir,
          target_size=(100,100),
          batch_size=32,
          class_mode='categorical',
          follow_links=True
  )
  
  print(validation_generator)
  
  print("--- %s seconds ---" % (time.time() - start_time))
  </code></pre>