<p>Your loss function value is not going into right direction. This means your model is not able to capture features it needs to focus on. There are 2 ways you can try</p>

<ol>
<li>Change your model. I'll suggest first pick up some widely used model for text classification. Then experiment with it towards better accuracy</li>
<li>Although your input representation makes sense, you can try with some specialized sentence-to-vectors models like <a href="https://arxiv.org/abs/1506.06726" rel="nofollow noreferrer">skip-thought</a> (one implementation <a href="https://github.com/ryankiros/skip-thoughts" rel="nofollow noreferrer">here</a>)</li>
</ol>
Convolutional Network for Text Classification
<p>I am trying to train a convolutional neural network with Keras at recognizing tags for Stack Exchange questions about cooking. </p>

<p>The i-th question element of my data-set is like this:</p>

<pre><code>id                                                         2
title                    How should I cook bacon in an oven?
content    &lt;p&gt;I've heard of people cooking bacon in an ov...
tags                                 oven cooking-time bacon
Name: 1, dtype: object
</code></pre>

<p>I have removed tags with BeautifulSoup and removed punctuation too.
Since questions' content are very big I have decided to focus on titles. 
I have used sklearn CountVectorizer to vectorize words in titles. However they were more than 8000 words (excluding stop words). So I decided apply a part of speech tagging and retrieve only Nouns and Gerunds.</p>

<pre><code>from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer(stop_words='english')
titles = dataframes['cooking']['title']
pos_titles = []
for i,title in enumerate(titles):
    pos = []
    pt_titl = nltk.pos_tag(word_tokenize(title))
    for pt in pt_titl:
        if pt[1]=='NN' or pt[1]=='NNS' or pt[1]=='VBG':# or pt[1]=='VBP' or pt[1]=='VBS':
            pos.append(pt[0])
    pos_titles.append(" ".join(pos))
</code></pre>

<p>This represents my input vector. I have vectorized tags too and extract dense matrixes for both input and tags. </p>

<pre><code>tags = [" ".join(x) for x in dataframes['cooking']['tags']]
Xd = X.todense()

Y = vectorizer.fit_transform(tags)
Yd = Y.todense()
</code></pre>

<p>Split data into train and validation set </p>

<pre><code>from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(Xd, Yd, test_size=0.33, random_state=42)
</code></pre>

<p>Now I am trying to train a Conv1D network</p>

<pre><code>from keras.models import Sequential
from keras.layers import Dense, Activation,Flatten
from keras.layers import Conv2D, MaxPooling2D,Conv1D, Embedding,GlobalMaxPooling1D,Dropout,MaxPooling1D

model = Sequential()

model.add(Embedding(Xd.shape[1],
                    128,
                    input_length=Xd.shape[1]))
model.add(Conv1D(32,5,activation='relu'))
model.add(MaxPooling1D(100,stride=50))
model.add(Conv1D(32,5,activation='relu'))
model.add(GlobalMaxPooling1D())
model.add(Dense(Yd.shape[1], activation ='softmax'))


model.compile(optimizer='rmsprop',
          loss='categorical_crossentropy',
          metrics=['accuracy'])
model.fit(X_train, y_train, batch_size=32,verbose=1)
</code></pre>

<p>But it gets stucked on a very low accuracy and it shows a barely increasing loss along the epochs</p>

<pre><code>Epoch 1/10
10320/10320 [==============================] - 401s - loss: 15.8098 - acc: 0.0604   
Epoch 2/10
10320/10320 [==============================] - 339s - loss: 15.5671 - acc: 0.0577   
Epoch 3/10
10320/10320 [==============================] - 314s - loss: 15.5509 - acc: 0.0578   
Epoch 4/10
10320/10320 [==============================] - 34953s - loss: 15.5493 - acc: 0.0578  
Epoch 5/10
10320/10320 [==============================] - 323s - loss: 15.5587 - acc: 0.0578   
Epoch 6/10
 6272/10320 [=================&gt;............] - ETA: 133s - loss: 15.6005 - acc: 0.0550
</code></pre>