title: How to prevent simple keras autoencoder from over compressing data?
body: <p>I am trying to use the <code>keras</code> frontend with <code>tensorflow</code> backend for a simple <code>autoencoder</code> as a multidimensional scaling technique to plot multidimensional data into 2 dimensions.  Many times when I run it (not sure how to set random seed for <code>keras</code> btw) one of the dimensions is collapsed to yield a 1 dimensional embedding (the plot should help explain).  Why is this happening? <strong>How can I make sure the dimensions are preserved and utilized by the autoencoder?</strong> I realize this is the most simple and basic form of an autoencoder that I have implemented but I would like to build on this to make better autoencoders for this task. </p>
  
  <pre><code>from sklearn.datasets import load_iris
  from sklearn import model_selection
  import tensorflow as tf
  import pandas as pd
  import numpy as np
  import matplotlib.pyplot as plt
  
  # Load data
  X = load_iris().data
  Y = pd.get_dummies(load_iris().target).as_matrix()
  X_tr, X_te, Y_tr, Y_te = model_selection.train_test_split(X,Y, test_size=0.3, stratify=Y.argmax(axis=1))
  dims = X_tr.shape[1]
  n_classes = Y_tr.shape[1]
  
  # Autoencoder
  encoding_dim = 2  
  
  # this is our input placeholder
  input_data = tf.keras.Input(shape=(4,))
  
  # "encoded" is the encoded representation of the input
  encoded = tf.keras.layers.Dense(encoding_dim, 
                                  activation='relu', 
                                 )(input_data)
  # "decoded" is the lossy reconstruction of the input
  decoded = tf.keras.layers.Dense(4, activation='sigmoid')(encoded)
  
  # this model maps an input to its reconstruction
  autoencoder = tf.keras.models.Model(input_data, decoded)
  
  # this model maps an input to its encoded representation
  encoder = tf.keras.models.Model(input_data, encoded)
  
  autoencoder.compile(optimizer='adam', loss='binary_crossentropy')
  network_training = autoencoder.fit(X_tr, X_tr,
                                      epochs=100,
                                      batch_size=5,
                                      shuffle=True,
                                      verbose=False,
                                      validation_data=(X_te, X_te))
  
  # Plot data
  embeddings = encoder.predict(X_te)
  plt.scatter(embeddings[:,0], embeddings[:,1], c=Y_te.argmax(axis=1), edgecolor="black", linewidth=1)
  </code></pre>
  
  <h3>Run algorithm once</h3>
  
  <p><a href="https://i.stack.imgur.com/JPT5W.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/JPT5W.png" alt="enter image description here"></a></p>
  
  <h3>Run algorithm again</h3>
  
  <p><a href="https://i.stack.imgur.com/wB3dt.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/wB3dt.png" alt="enter image description here"></a></p>