<ol>
<li>you can do <a href="https://arxiv.org/pdf/1710.09282.pdf" rel="nofollow noreferrer">network compression</a>.</li>
<li>cut the sentence into pieces by <a href="http://www.aclweb.org/anthology/P16-1162" rel="nofollow noreferrer">byte-pair-encoding</a> or <a href="https://arxiv.org/abs/1804.10959" rel="nofollow noreferrer">unigram language model</a> and etc and then try <a href="https://arxiv.org/pdf/1503.00075.pdf" rel="nofollow noreferrer">TreeLSTM</a>.</li>
<li>you can try faster softmax like adaptive softmax](<a href="https://arxiv.org/pdf/1609.04309.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1609.04309.pdf</a>)</li>
<li>try <a href="https://www.tensorflow.org/api_docs/python/tf/compat/v1/keras/layers/CuDNNLSTM" rel="nofollow noreferrer">cudnnLSTM</a></li>
<li>try <a href="https://papers.nips.cc/paper/6613-dilated-recurrent-neural-networks.pdf" rel="nofollow noreferrer">dilated RNN</a></li>
<li>switch to CNN like dilated CNN, or BERT for parallelization and more efficient GPU support</li>
</ol>
how to speedup tensorflow RNN inference time
<p>We've trained a tf-seq2seq model for question answering. The main framework is from <a href="https://github.com/google/seq2seq" rel="nofollow noreferrer">google/seq2seq</a>. We use bidirectional RNN( GRU encoders/decoders 128units), adding soft attention mechanism.</p>
<p>We limit maximum length to 100 words. It mostly just generates 10~20 words.</p>
<p>For model inference, we try two cases:</p>
<ol>
<li>normal(greedy algorithm). Its inference time is about 40ms~100ms</li>
<li>beam search. We try to use beam width 5, and its inference time is about 400ms~1000ms.</li>
</ol>
<p>So, we want to try to use beam width 3, its time may decrease, but it may also influence the final effect.</p>
<p>So are there any suggestion to decrease inference time for our case? Thanks.</p>