<p>There are a few things coming to mind you may try:</p>

<h2>GRU instead of LSTM</h2>

<p>This cell type has one gate less and thus is more computationally efficient.</p>

<p>Furthermore they have a little less hyperparameters so possible search space is smaller.</p>

<p>Nothing conclusive but anecdotically they perform on-par with LSTMs at least for moderate length sequences.</p>

<h2>Use CUDA</h2>

<p>If possible use CUDA enabled device for training your network. Particularly, for <code>tensorflow</code> use <code>CuDNNGRU</code> version (for <code>tf2.x</code> see <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/GRU" rel="nofollow noreferrer">this</a>, there are options for <code>1.x</code> as well). </p>

<h2>Narrow your search space</h2>

<p>The less hyperparameters to search for, the faster it will go. <code>learning_rate</code> is usually very important to get right.</p>

<p>There is quite-known trick for approximating optimal learning rate originated by Leslie N. Smith in <a href="https://arxiv.org/abs/1506.01186" rel="nofollow noreferrer">Cyclical Learning Rates for Training Neural Networks</a> and popularised by <a href="https://docs.fast.ai/callbacks.lr_finder.html" rel="nofollow noreferrer">fastai</a> community.</p>

<p>Basic idea - you set learning rate really low (e.g. <code>1e-7</code>) and increment it exponentially every <code>n</code> batches until threshold is hit (highest <code>lr</code>, around <code>1.0</code> usually).</p>

<p>Monitoring <code>train</code> or <code>validation</code> loss you will see it will probably skyrocket at some learning rate. You take this point and set your approximately optimal learning <code>10</code> lower than that.</p>

<p>You can read about it <a href="https://www.pyimagesearch.com/2019/08/05/keras-learning-rate-finder/" rel="nofollow noreferrer">here</a>.</p>

<p>Other hyperparameters could be narrowed as well when you see some values are probably no-go early on, though bayesian optimisation takes care of that partially.</p>
How to reduce the computational cost
<p>I use GPyOpt to calculate the optimized parameters of an LSTM model based on Tensorflow. Unlike typical process, however, the validation of the parameters is implemented by minimizing variance, not bias. Hence, it is required to iterate the LSTM model running with a parameters set. In fact, my script does work well without any errors, but it takes up too much time to return the optimum parameters. I tried to find out any methods to improve the processing speed, but couldn't.  </p>

<p>I think the other modules, for instance GridSearch or RandomizedSearch are perhaps faster than one of GpyOp, but they use Bayesian Optimization to obtain the parameters.</p>

<p>Is there a way to reduce the computational cost of this operation?  </p>

<p>My PC: <code>MackBook, 2.3 GHz Intel Core i5</code></p>

<p>My script:</p>

<pre><code>import os
os.environ['KMP_DUPLICATE_LIB_OK']='True'
os.environ['PYTHONHASHSEED']='0'
os.environ['OMP_NUM_THREADS']='4'
os.environ['KMP_BLOCKTIME']='30'
os.environ['KMP_SETTINGS']='1'
os.environ['KMP_AFFINITY']='granularity=fine,verbose,compact,1,0'

import tensorflow as tf
import numpy as np
import GPyOpt

import random

NUM_PARALLEL_EXEC_UNITS=4
config=tf.ConfigProto(intra_op_parallelism_threads=NUM_PARALLEL_EXEC_UNITS,
                      inter_op_parallelism_threads=2,
                      allow_soft_placement=True,
                  device_count={'CPU':NUM_PARALLEL_EXEC_UNITS})

def parameter_opt(df_x,df_y):

    x_set_list=[]
    y_set_list=[]
    for i in range(10): #create 10 data sets for calculating variance
        t=random.randint(1,2300)
        x_set_=df_x[t:t+10,:,:] #x_set_.shape is (10,7,5)
        y_set_=df_y[t:t+10,:]
        x_set_list.append(x_set_)
        y_set_list.append(y_set_)

    pred_x=df_x[-10:,:,:]


    bounds=[{'name':'hidden_layer','type':'discrete','domain':(10,50,100,300,500)},
            {'name':'learn_rate','type':'discrete','domain':(0.1,0.01,0.001,0.0001)},
            {'name':'forget','type':'continuous','domain':(0.1,1.0)},
            {'name':'std','type':'continuous','domain':(0.01,1.0)},
            {'name':'epo','type':'discrete','domain':(50,100,200,400)},
            {'name':'cell_drop_','type':'discrete','domain':(0,1)},
            {'name':'output_keep','type':'continuous','domain':(0.1,1.0)}]

    def f(x):
        tf.reset_default_graph()

        def LSTMmse(hidden_layer,learn_rate,forget,std,epo,cell_drop_, output_keep):
            pred_array=np.empty((10,0))
            for x_set,y_set in zip(x_set_list,y_set_list):
                tf.reset_default_graph()
                .......
                #LSTM model
                #generate prediction_list consisting of 10 predictionvalue 
                .........
            variance_list=np.var(prediction_list,axis=1)
            variance_ave=sum(var_list)/len(var_list)
            return variance_ave

        for x_ in x:
            vari=LSTMmse(hidden_layer=int(x_[0]),
                        learn_rate=np.float32(x_[1]),
                        forget=np.float32(x_[2]),
                        std=np.float32(x_[3]),
                        epo=int(x_[4]),
                        cell_drop_=bool(x_[5]),
                        output_keep=np.float32(x_[6]))
         return vari

    myBopt=GPyOpt.methods.BayesianOptimization(f=f,domain=bounds,acquisition_type='MPI')

    myBopt.run_optimization(max_iter=10)
    opt=myBopt.x_opt
    opt=[int(opt[0]),opt[1],opt[2],opt[3],int(opt[4]),opt[5],opt[6]]

    key=['hidden_layer','learn_rate','forget','std','epo','cell_drop','output_keep']
    value=opt
    para_dict=dict(zip(key,value))
    return para_dict

parameter_opt=parameter_opt(df_x,df_y)
print(parameter_opt)  # spending too much time
</code></pre>