title: auto encoder with MFCC audio features
body: <p>I tried to train an auto encoder to learn feature representations from some segments of audio MFCC features(my dataset is TIMIT). So I used a fixed size window to cut the audio MFCC data(13 dimensional MFCC), My data is (None, window_size*13),here my window size is 20(frames). I trained a auto encoder with feed_forward type, use loss function as least squared error(the difference norm between prediction and label), the problem is my loss stops decrease at a high level. My network part code is here: </p>
  
  <pre><code>self.centers_placeholder = tf.placeholder(tf.float32,[None,self.train_centers[0].shape[1]],name = 'center_placeholder')
  
  
  
          layer = tf.layers.dense(self.centers_placeholder,2000,activation=tf.nn.relu)
          layer = tf.layers.batch_normalization(layer)
  
          layer = tf.layers.dense(layer,1500,activation=tf.nn.leaky_relu)
          layer = tf.layers.batch_normalization(layer)
  
  
          layer = tf.layers.dense(layer,500,activation=tf.nn.leaky_relu)
          layer = tf.layers.batch_normalization(layer)
  
          layer = tf.layers.dense(layer,100,activation=tf.nn.leaky_relu)
          layer = tf.layers.batch_normalization(layer)
  
          self.embedding= tf.layers.dense(layer,50,activation = tf.nn.leaky_relu)
  
          layer = tf.layers.dense(layer,100,activation=tf.nn.leaky_relu)
          layer = tf.layers.batch_normalization(layer)
  
          layer = tf.layers.dense(layer,500,activation=tf.nn.leaky_relu)
          layer = tf.layers.batch_normalization(layer)
  
          layer = tf.layers.dense(layer,1500,activation=tf.nn.leaky_relu)
          layer = tf.layers.batch_normalization(layer)
  
  
          layer = tf.layers.dense(layer,2000,activation=tf.nn.leaky_relu)
          layer = tf.layers.batch_normalization(layer)
  
          self.decoded = tf.layers.dense(layer,self.train_centers[0].shape[1],activation=None)
  
          #cost = tf.reduce_mean(tf.sqrt(tf.square(neighbors_placeholder-decoder4)))
          self.cost = tf.reduce_mean(tf.norm(self.centers_placeholder-self.decoded,ord=2,axis=1))
          #self.cost=tf.reduce_mean(tf.losses.mean_squared_error(labels=self.centers_placeholder,predictions=self.decoded))
          target_norm = tf.norm(self.centers_placeholder,ord=2,axis=1)
          difference_norm=tf.norm(self.centers_placeholder-self.decoded,ord=2,axis=1)
          self.metric = tf.reduce_mean(difference_norm/target_norm)
          print('model graph has built',flush=True)
  
          global_step=tf.Variable(0,trainable=False)
          starter_learning_rate=self.learning_rate
          lr=tf.train.exponential_decay(starter_learning_rate,global_step,100000,0.96,staircase=True)
          self.optimizer = tf.train.AdamOptimizer(lr).minimize(self.cost,global_step=global_step)
          self.saver = tf.train.Saver()
  </code></pre>
  
  <p>I tried different architecture, deep or shallow, but it always convergence at a high level. Here is one example of my losses<a href="https://i.stack.imgur.com/FmMvx.png" rel="nofollow noreferrer">enter image description here</a></p>
  
  <p>Can anyone help? Is there anyone has some experiments with MFCC in auto encoder?
  Thanks!</p>