<p>I have already solved my problem. After reading some papers and more trial and error, I figured out what my mistakes were.</p>

<p><strong>1) Dataset: I had a large dataset, but I didn't format it properly.</strong></p>

<ul>
<li>I checked the distribution of tweet labels (Neutral, Positive and Negative), realized there was a disparity in the distribution of said tweets and normalized it.</li>
<li>I cleaned it up even more by erasing url hashtags and unnecessary  punctuation.</li>
<li>I shuffled prior to vectorization.</li>
</ul>

<p><strong>2) Initialization:</strong></p>

<ul>
<li>I initialized the MultiRNNCell with zeros and I changed my custom final layer to tf.contrib.fully_connected. I also added the initialization of the bias and weight matrix. (By fixing this, I started to see better loss and accuracy plots in Tensorboard)</li>
</ul>

<p><strong>3) Dropout:</strong></p>

<ul>
<li>I read this paper, <a href="https://arxiv.org/pdf/1603.05118.pdf" rel="nofollow noreferrer">Recurrent Dropout without Memory Loss</a>, and I changed my dropouts accordingly; I started seeing improvements in the loss and accuracy.</li>
</ul>

<p><strong>4) Decaying the learning rate:</strong></p>

<ul>
<li>I added an exponential decaying rate after 10,000 steps to control over-fitting.</li>
</ul>

<p><strong><em>Final results:</em></strong></p>

<p>After applying all of these changes, I achieved a test accuracy of 84%, which is acceptable because my data set still sucks.</p>

<p>My final network config was:</p>

<ul>
<li>num_epochs = 20</li>
<li>tweet_size = 20</li>
<li>hidden_size = 400</li>
<li>vec_size = 300</li>
<li>batch_size = 512</li>
<li>number_of_layers= 2</li>
<li>number_of_classes= 3</li>
<li>start_learning_rate = 0.001</li>
</ul>
Tensorflow lstm for sentiment analysis not learning. UPDATED
<h1>UPDATED:</h1>
<p>i'm building a Neural Network for my final project and i need some help with it.</p>
<p>I'm trying to build a rnn to do sentiment analysis over Spanish text. I have about 200,000 labeled tweets and i vectorized them using a word2vec with a Spanish embedding</p>
<p><strong>Dataset &amp; Vectorization:</strong></p>
<ul>
<li>I erased duplicates and split the dataset into training and testing sets.</li>
<li>Padding, unknown and end of sentence tokens are applied when vectorizing.</li>
<li>I mapped the @mentions to known names in the word2vec model. Example: @iamthebest =&gt; &quot;John&quot;</li>
</ul>
<p><strong>My model:</strong></p>
<ul>
<li>My data tensor has shape = (batch_size, 20, 300).</li>
<li>I have 3 classes: neutral, positive and negative, so my target tensor has shape = (batch_size, 3)</li>
<li>I use BasicLstm cells and dynamic rnn to build the net.</li>
<li>I use Adam Optimizer, and softmax_cross entropy for the loss calculation</li>
<li>I use a dropout wrapper to decrease the overfitting.</li>
</ul>
<p><strong>Last run:</strong></p>
<ul>
<li>I have tried with different configurations and non of them seem to work.</li>
<li>Last setup:  2 Layers, 512 batch size, 15 epochs and 0.001 of lr.</li>
</ul>
<p><a href="https://i.stack.imgur.com/Y786W.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Y786W.jpg" alt="Accuracy" /></a></p>
<p><a href="https://i.stack.imgur.com/5OGrS.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/5OGrS.jpg" alt="Loss" /></a></p>
<p><strong>Weak points for me:</strong></p>
<p>im worried about the final layer and the handing of the final state in the dynamic_rnn</p>
<p><strong>Code:</strong></p>
<pre><code># set variables
num_epochs = 15
tweet_size = 20
hidden_size = 200
vec_size = 300
batch_size = 512
number_of_layers= 1
number_of_classes= 3
learning_rate = 0.001

TRAIN_DIR=&quot;/checkpoints&quot;

tf.reset_default_graph()

# Create a session 
session = tf.Session()

# Inputs placeholders
tweets = tf.placeholder(tf.float32, [None, tweet_size, vec_size], &quot;tweets&quot;)
labels = tf.placeholder(tf.float32, [None, number_of_classes], &quot;labels&quot;)

# Placeholder for dropout
keep_prob = tf.placeholder(tf.float32) 

# make the lstm cells, and wrap them in MultiRNNCell for multiple layers
def lstm_cell():
    cell = tf.contrib.rnn.BasicLSTMCell(hidden_size)
    return tf.contrib.rnn.DropoutWrapper(cell=cell, output_keep_prob=keep_prob)

multi_lstm_cells = tf.contrib.rnn.MultiRNNCell([lstm_cell() for _ in range(number_of_layers)], state_is_tuple=True)

# Creates a recurrent neural network
outputs, final_state = tf.nn.dynamic_rnn(multi_lstm_cells, tweets, dtype=tf.float32)

with tf.name_scope(&quot;final_layer&quot;):
    # weight and bias to shape the final layer
    W = tf.get_variable(&quot;weight_matrix&quot;, [hidden_size, number_of_classes], tf.float32, tf.random_normal_initializer(stddev=1.0 / math.sqrt(hidden_size)))
    b = tf.get_variable(&quot;bias&quot;, [number_of_classes], initializer=tf.constant_initializer(1.0))

    sentiments = tf.matmul(final_state[-1][-1], W) + b

prob = tf.nn.softmax(sentiments)
tf.summary.histogram('softmax', prob)

with tf.name_scope(&quot;loss&quot;):
    # define cross entropy loss function
    losses = tf.nn.softmax_cross_entropy_with_logits(logits=sentiments, labels=labels)
    loss = tf.reduce_mean(losses)
    tf.summary.scalar(&quot;loss&quot;, loss)

with tf.name_scope(&quot;accuracy&quot;):
    # round our actual probabilities to compute error
    accuracy = tf.to_float(tf.equal(tf.argmax(prob,1), tf.argmax(labels,1)))
    accuracy = tf.reduce_mean(tf.cast(accuracy, dtype=tf.float32))
    tf.summary.scalar(&quot;accuracy&quot;, accuracy)

# define our optimizer to minimize the loss
with tf.name_scope(&quot;train&quot;):
    optimizer = tf.train.AdamOptimizer(learning_rate).minimize(loss)

#tensorboard summaries
merged_summary = tf.summary.merge_all()
logdir = &quot;tensorboard/&quot; + datetime.datetime.now().strftime(&quot;%Y%m%d-%H%M%S&quot;) + &quot;/&quot;
writer = tf.summary.FileWriter(logdir, session.graph)

# initialize any variables
tf.global_variables_initializer().run(session=session)

# Create a saver for writing training checkpoints.
saver = tf.train.Saver()

# load our data and separate it into tweets and labels
train_tweets = np.load('data_es/train_vec_tweets.npy')
train_labels = np.load('data_es/train_vec_labels.npy')

test_tweets = np.load('data_es/test_vec_tweets.npy')
test_labels = np.load('data_es/test_vec_labels.npy')

**HERE I HAVE THE LOOP FOR TRAINING AND TESTING, I KNOW ITS FINE**
</code></pre>