title: Handling Binary Input / Output
body: <p>Which things should I consider if the input and output of my neural network are (or should be) binary values?</p>
  
  <p><strong>Example</strong></p>
  
  <p>I have a sequence of one-hot encoded vectors like this:</p>
  
  <pre><code>[0 1 0 0], [1 0 0 0], ... 
  </code></pre>
  
  <p>So, regarding this there are some thoughts or questions that arise:</p>
  
  <ul>
  <li><p>Is it reasonable to use it just like it is as an input for a neural network like a <code>LSTM</code>? Or should I transform it anyhow?</p></li>
  <li><p>The other thing is, LSTMs return continuous values between -1 and 1 (<code>tanh</code>), should I use another activation function instead? In the end I want discrete output as well, just like my input vectors. Should I round the values?</p></li>
  <li><p>And what I realized and is somehow weird is that my current network tends to set all it's (inner) outputs to nearly exact -1, 0 or 1... How can I (should I?) prevent the neural network to do this?</p></li>
  </ul>
  
  <p><em>EDIT:</em> My network architecture looks somehow like this, expecting a sequence of one-hot-encoded sequences, turning it to a vector (that also tends to have only nearly zero or one values) and the decoder should return the same as the input was (autoencoder). The encoder and decoder have some stacked LSTMs.</p>
  
  <p><a href="https://i.stack.imgur.com/pBpZ4m.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/pBpZ4m.png" alt="enter image description here"></a></p>
  
  <p>The input looks like this (one-hot-encoded, 120 time steps with 115 vector length).</p>
  
  <pre><code>array([[[1, 0, 0, ..., 0, 0, 0],
          [0, 1, 0, ..., 0, 0, 0],
          [0, 0, 1, ..., 0, 0, 0],
          ..., 
          [0, 0, 0, ..., 0, 0, 0],
          [0, 0, 0, ..., 0, 0, 0],
          [0, 0, 0, ..., 0, 0, 0]]])
  </code></pre>
  
  <p>I have 11.000 examples.</p>
  
  <p>This is my current coding:</p>
  
  <pre><code> inp = Input((120,115))
  
   out = LSTM(units = 200, return_sequences=True, activation='tanh')(inp)
   out = LSTM(units = 180, return_sequences=True)(out)
   out = LSTM(units = 140, return_sequences=True, activation='tanh')(out)
   out = LSTM(units = 120, return_sequences=False, activation='tanh')(out)
   encoder = Model(inp,out)   
  
   out_dec = RepeatVector(120)(out) # I also tried to use Reshapeinstead, not really a difference
  
   out1 = LSTM(200,return_sequences=True, activation='tanh')(out_dec)   
   out1 = LSTM(175,return_sequences=True, activation='tanh')(out1)   
   out1 = LSTM(150,return_sequences=True, activation='tanh')(out1)   
   out1 = LSTM(115,return_sequences=True, activation='sigmoid')(out1) # I also tried softmax instead of sigmoid, not really a difference
  
   decoder = Model(inp,out1)
  
  autoencoder = Model(encoder.inputs, decoder(encoder.inputs))
  
  autoencoder.compile(loss='binary_crossentropy',
                optimizer='RMSprop',
                metrics=['accuracy'])
  
  autoencoder.fit(padded_sequences[:9000], padded_sequences[:9000],
            batch_size=150,
            epochs=5,
            validation_data=(padded_sequences[9001:], padded_sequences[9001:]))
  </code></pre>
  
  <p>But after a few epochs of training, there is no improvement anymore.</p>
  
  <p>The output for the example in the beginning looks like this, not very much the same...</p>
  
  <pre><code>array([[[ 0.14739206,  0.49056929,  0.06915747, ...,  0.        ,
            0.        ,  0.        ],
          [ 0.03878205,  0.7227878 ,  0.03550367, ...,  0.        ,
            0.        ,  0.        ],
          [ 0.02073009,  0.74334699,  0.03663541, ...,  0.        ,
            0.        ,  0.        ],
          ..., 
          [ 0.        ,  0.08416401,  0.        , ...,  0.        ,
            0.        ,  0.        ],
          [ 0.        ,  0.08630376,  0.        , ...,  0.        ,
            0.        ,  0.        ],
          [ 0.        ,  0.08602102,  0.        , ...,  0.        ,
            0.        ,  0.        ]]], dtype=float32)
  </code></pre>
  
  <p>The embedding vector (produced by encoder.predict) looks like this (somehow weird as all values are nearly -1, 0, or 1).</p>
  
  <pre><code>array([[ -1.00000000e+00,  -0.00000000e+00,  -1.00000000e+00,
            1.00000000e+00,   1.00000000e+00,   9.99999523e-01,
            1.00000000e+00,   9.99999881e-01,   1.00000000e+00,
            9.99989152e-01,   9.99999821e-01,   9.99998808e-01,
            1.00000000e+00,  -0.00000000e+00,  -4.86032724e-01,
            9.99996543e-01,   1.00000000e+00,   0.00000000e+00,
            1.00000000e+00,   0.00000000e+00,   0.00000000e+00,
            1.00000000e+00,  -0.00000000e+00,   0.00000000e+00,
            0.00000000e+00,  -0.00000000e+00,   9.99999464e-01,
           -9.99999881e-01,  -0.00000000e+00,   4.75281268e-01,
            3.01986277e-01,   6.65608108e-01,  -9.99999881e-01,
            0.00000000e+00,  -0.00000000e+00,  -0.00000000e+00,
            0.00000000e+00,  -0.00000000e+00,  -3.65448680e-15,
           -9.99888301e-01,  -0.00000000e+00,  -1.00000000e+00,
           -1.00000000e+00,  -9.90761220e-01,  -9.96851087e-01,
           -0.00000000e+00,   0.00000000e+00,  -1.47916377e-02,
           -9.99999523e-01,  -2.90349454e-01,  -9.99999702e-01,
           -7.63339102e-02,  -1.00000000e+00,  -4.16638345e-01,
           -9.99999940e-01,  -1.00000000e+00,  -9.99996841e-01,
           ..............
  </code></pre>
  
  <p>My guess is that is has something to do with my binary input / output.</p>