title: Reducing time taken to predict probabilities
body: <p>I am using Keras with Tensorflow backend. Once I have the model I am using model.predict_proba to predict probabilities. The time taken by this predict function seems to be independent of the size of the array, or rather less dependent on the size of the array that it is predicting. </p>
  
  <p>What can I do to reduce this time taken? </p>
  
  <p>First I run the loop 100 times and time the predictions as below </p>
  
  <pre><code>fname_model = folder+"/my_modelbuy_rocbased.h5"
  best_model = load_model(fname_model)
  best_model.compile(loss='categorical_crossentropy', optimizer='Adam', 
  metrics=['acc'])
  X_test_orig = pd.read_csv(folder+"/ret_out_20170402.csv",sep=",")
  x_test = X_test_orig.as_matrix()
  y1=np.array([x_test[1]])
  print y1.shape 
  t1=datetime.now().strftime("%H:%M:%S.%f")
  for i in range(0,100,1):
  prob1= best_model.predict_proba(y1)
  t2=datetime.now().strftime("%H:%M:%S.%f")   
  print "time taken ", t1,t2
  </code></pre>
  
  <p>**********************************************After Running***********************</p>
  
  <p>time taken  11:50:53.909226 11:50:54.372596</p>
  
  <hr>
  
  <p>So for 100 predictions, the time taken was 11:50:54.372596 minus 11:50:53.909226 which is about 463 milliseconds, which would imply 4.63 milliseconds per prediction </p>
  
  <p>However if i run the loop for just one prediction as shown below , </p>
  
  <pre><code>fname_model = folder+"/my_modelbuy_rocbased.h5"
  best_model = load_model(fname_model)
  best_model.compile(loss='categorical_crossentropy', optimizer='Adam', 
  metrics=['acc'])
  X_test_orig = pd.read_csv(folder+"/ret_out_20170402.csv",sep=",")
  x_test = X_test_orig.as_matrix()
  y1=np.array([x_test[1]])
  print y1.shape 
  t1=datetime.now().strftime("%H:%M:%S.%f")
  for i in range(0,1,1):
  prob1= best_model.predict_proba(y1)
  t2=datetime.now().strftime("%H:%M:%S.%f")   
  print "time taken ", t1,t2
  </code></pre>
  
  <p>**********************************************After Running***********************</p>
  
  <p>time taken  11:54:40.683225 11:54:41.144969</p>
  
  <hr>
  
  <p>So for 1 prediction, the time taken is 11:54:40.683225 minus 11:54:41.144969 which is also about 440 milliseconds. </p>
  
  <p>There seems to be a fixed amount of time the model will take regardless of the number of calls. How can I reduce this? </p>