title: Tensorflow Check failed: status == CUDNN_STATUS_SUCCESS (7 vs. 0)Failed to set cuDNN stream
body: <p>When I run my tensorflow keras model it will sometimes stop after an epoch and throw the following error:</p>
  <pre><code>tensorflow/stream_executor/dnn.cc:613] CUDNN_STATUS_EXECUTION_FAILED
  in tensorflow/stream_executor/cuda/cuda_dnn.cc(1867): 'cudnnRNNForwardTraining( cudnn.handle(), rnn_desc.handle(), model_dims.max_seq_length, input_desc.handles(), input_data.opaque(), input_h_desc.handle(), input_h_data.opaque(), input_c_desc.handle(), input_c_data.opaque(), rnn_desc.params_handle(), params.opaque(), output_desc.handles(), output_data-&gt;opaque(), output_h_desc.handle(), output_h_data-&gt;opaque(), output_c_desc.handle(), output_c_data-&gt;opaque(), workspace.opaque(), workspace.size(), reserve_space.opaque(), reserve_space.size())'
  2020-06-27 17:19:45.741256: F tensorflow/stream_executor/cuda/cuda_dnn.cc:189] Check failed: status == CUDNN_STATUS_SUCCESS (7 vs. 0)Failed to set cuDNN stream.
  </code></pre>
  <p>Im using the GPU version of tensorflow and it has worked without problems up until about 3 days ago when it began giving me this error. I also have the following callbacks in my model so that it saves checkpoints and epochs locally. Please note I am using Autokeras thus why the nodes have the ak. library prefix. However it works just like keras, it just automatically tries to find the best set of hyperparameters when running the .fit function.</p>
  <pre><code>input_node = ak.Input()
  #output_node1 = ak.Normalization()(input_node)
  output_node1 = ak.RNNBlock(return_sequences=True,
                             layer_type='lstm')(input_node)
  output_node2 = ak.RNNBlock(return_sequences=True,
                             layer_type='lstm')(output_node1)
  output_node3 = ak.RNNBlock(return_sequences=True,
                             layer_type='lstm')(output_node2)
  output_node4 = ak.RNNBlock(layer_type='lstm')(output_node3)
  output_node5 = ak.DenseBlock()(output_node4)
  output_node = ak.Merge()(
      [output_node1, output_node2, output_node3, output_node4, output_node5])
  output_nodefinal = ak.ClassificationHead()(output_node)
  
  my_callbacks = [tf.keras.callbacks.EarlyStopping(patience=15, restore_best_weights=True, monitor=&quot;val_accuracy&quot;, mode=&quot;auto&quot;),
                  tf.keras.callbacks.ModelCheckpoint(filepath=&quot;D:\AutoKerasProject\TimeseriesCallbackModel\TimeseriesCallbackModel&quot;, save_weights_only=True, monitor='val_accuracy', mode='auto', save_best_only=True)]
  
  classifier = ak.AutoModel(
      inputs=input_node, outputs=output_nodefinal, max_trials=500, directory=&quot;D:\AutoKerasProject\TimeseriesModel&quot;, overwrite=True)
  </code></pre>
  <p>My PC and software specs are as follows</p>
  <pre><code>PC Specs:
  Windows 8.1
  GTX 1060 6GB
  Intel i7-4770
  
  Software Specs:
  Nvidia Geforce Experience Driver 451.81
  CuDNN 7.6.5 (windows 7 version)
  Cuda 10.1.243
  Conda Python 3.6.10
  Tensorflow 2.2.0
  </code></pre>