title: Celery task for ML prediction hangs in execution
body: <p>I'm trying to create a web application that receives an input from POST request and provides some ML predictions based on that input.</p>
  
  <p>Since prediction model is quite heavy, I don't want that user waits for calculation to complete. Instead, I delegated heavy computation to Celery task and user can later inspect the result.</p>
  
  <p>I'm using simple Flask application with Celery, Redis and Flower.</p>
  
  <p>My <code>view.py</code>:</p>
  
  <pre><code>@ns.route('predict/')
  class Predict(Resource):
      ...
      def post(self):
          ...
          do_categorize(data)
          return jsonify(success=True)
  </code></pre>
  
  <p>My <code>tasks.py</code> file looks something like this:</p>
  
  <pre><code>from ai.categorizer import Categorizer
  categorizer = Categorizer(
      model_path='category_model.h5',
      tokenizer_path='tokenize.joblib',
      labels_path='labels.joblib'
  )
  
  
  @task()
  def do_categorize(data):
      result = categorizer.predict(data)
      print(result)
      # Write result to the DB
      ...
  </code></pre>
  
  <p>My <code>predict()</code> method inside <code>Categorizer</code> class:</p>
  
  <pre><code>def predict(self, value):
      K.set_session(self.sess)
      with self.sess.as_default():
          with self.graph.as_default():
              prediction = self.model.predict(np.asarray([value], dtype='int64'))
              return prediction
  </code></pre>
  
  <p>I'm running Celery like this:</p>
  
  <pre><code>celery worker -A app.celery --loglevel=DEBUG
  </code></pre>
  
  <p>The problem that I'm having for the last couple of days is that <code>categorizer.predict(data)</code> call hangs in the middle of the execution.</p>
  
  <p>I tried to run <code>categorizer.predict(data)</code> inside of post method and it works. But if I place it inside Celery tasks it stop working. There is no console log, if I try to debug it it just freezes on <code>.predict()</code>.</p>
  
  <p>My questions:</p>
  
  <ul>
  <li>How can I solve this issue?</li>
  <li>Is there any memory, CPU limit for the worker?</li>
  <li>Are Celery tasks the "right" way to do such heavy computations?</li>
  <li>How can I debug this problem? What am I doing wrong?</li>
  <li>Is it correct to initialize models at the top of the file?</li>
  </ul>