title: Deep learning Udacity course: Prob 2 assignment 1 (notMNIST)
body: <p>After reading <a href="https://discussions.udacity.com/t/assingment-1-1-notmnist-ipynb-well-convert-the-entire-dataset-into-a-3d-array-image-index-x-y-of-floating-point-values-normalized-to-have-approximately-zero-mean-and-standard-deviation-0-5-to-make-training-easier-down-the-road/45902" rel="noreferrer">this</a> and taking the courses, I am struggling to solve the second problem in assignment 1 (<a href="https://www.udacity.com/course/viewer#!/c-ud730/l-6370362152/m-6370314231" rel="noreferrer">notMnist</a>):</p>
  <blockquote>
  <p>Let's verify that the data still looks good. Displaying a sample of the labels and images from the ndarray. Hint: you can use matplotlib.pyplot.</p>
  </blockquote>
  <p>Here is what I tried:</p>
  <pre><code>import random
  rand_smpl = [ train_datasets[i] for i in sorted(random.sample(xrange(len(train_datasets)), 1)) ]
  print(rand_smpl)
  filename = rand_smpl[0]
  import pickle
  loaded_pickle = pickle.load( open( filename, &quot;r&quot; ) )
  image_size = 28  # Pixel width and height.
  import numpy as np
  dataset = np.ndarray(shape=(len(loaded_pickle), image_size, image_size),
                           dtype=np.float32)
  import matplotlib.pyplot as plt
  
  plt.plot(dataset[2])
  plt.ylabel('some numbers')
  plt.show()
  </code></pre>
  <p>but this is what I get:</p>
  <p><a href="https://i.stack.imgur.com/vY5Ql.png" rel="noreferrer"><img src="https://i.stack.imgur.com/vY5Ql.png" alt="enter image description here" /></a></p>
  <p>which doesn't make much sense. To be honest my code may too, since I am not really sure how to tackle that problem!</p>
  <hr />
  <p>The pickles are created like this:</p>
  <pre><code>image_size = 28  # Pixel width and height.
  pixel_depth = 255.0  # Number of levels per pixel.
  
  def load_letter(folder, min_num_images):
    &quot;&quot;&quot;Load the data for a single letter label.&quot;&quot;&quot;
    image_files = os.listdir(folder)
    dataset = np.ndarray(shape=(len(image_files), image_size, image_size),
                           dtype=np.float32)
    print(folder)
    num_images = 0
    for image in image_files:
      image_file = os.path.join(folder, image)
      try:
        image_data = (ndimage.imread(image_file).astype(float) - 
                      pixel_depth / 2) / pixel_depth
        if image_data.shape != (image_size, image_size):
          raise Exception('Unexpected image shape: %s' % str(image_data.shape))
        dataset[num_images, :, :] = image_data
        num_images = num_images + 1
      except IOError as e:
        print('Could not read:', image_file, ':', e, '- it\'s ok, skipping.')
      
    dataset = dataset[0:num_images, :, :]
    if num_images &lt; min_num_images:
      raise Exception('Many fewer images than expected: %d &lt; %d' %
                      (num_images, min_num_images))
      
    print('Full dataset tensor:', dataset.shape)
    print('Mean:', np.mean(dataset))
    print('Standard deviation:', np.std(dataset))
    return dataset
  </code></pre>
  <p>where that function is called like this:</p>
  <pre><code>  dataset = load_letter(folder, min_num_images_per_class)
    try:
      with open(set_filename, 'wb') as f:
        pickle.dump(dataset, f, pickle.HIGHEST_PROTOCOL)
  </code></pre>
  <p>The idea here is:</p>
  <blockquote>
  <p>Now let's load the data in a more manageable format. Since, depending on your computer setup you might not be able to fit it all in memory, we'll load each class into a separate dataset, store them on disk and curate them independently. Later we'll merge them into a single dataset of manageable size.</p>
  <p>We'll convert the entire dataset into a 3D array (image index, x, y) of floating point values, normalized to have approximately zero mean and standard deviation ~0.5 to make training easier down the road.</p>
  </blockquote>