title: Celery pickling not playing nice with Cassandra driver, can't figure out the root cause
body: <p>I'm experiencing some behavior that I can't quite figure out.  I'm using Cassandra to store message objects, and I'm using Celery for async pulls and pushes to the database.  Everything is working fine, except for a single Celery task; the other tasks that use the same code/classes work.  Here's a rough breakdown of the code logic:</p>
  
  <pre><code>db_manager = DBManager()
  
  class User(object):
      def __init__(self, user_id):
          ... normal init stuff ...
          self.loader()
  
      @run_async
      def loader(self):
          ... loads from database if found, otherwise pulls from API ...
  
      # THIS WORKS
      @celery.task(name='user-to-db', filter=task_method)
      def to_db(self):
           # db_manager is a custom backend that handles relevant db reads, writes, etc.
           db_manager.add('users', self.user_payload)
  
       # THIS WORKS
       @celery.task(name='load-friends', filter=task_method)
       def load_friends(self):
            # Checks secondary redis index for friends of user
            friends = redis.srandmember('users:the-users-id:friends', self.id, 20)
            if not friends:
                 profiles = load_friends_from_api(user_id=self.id)
            else:
                 query = "SELECT * FROM keyspace.users WHERE id IN ({friends})".format(friends=friends)
            # Init a User object for every friend
            loaded_friends = [User(friend) for friend in profiles]
            # Returns a class container with all the instances of User(friend), accessible through a class property
            return FriendContainer(self.id, loaded_friends)
  
       # THIS DOES NOT WORK
       @celery.task(name='get-user-messages', filter=task_method)
       def get_user_messages(self):
            # THIS IS WHERE IT FAILS #
            messages = db_manager.get("SELECT message FROM keyspace.message_timelines WHERE user_id = {user_id}".format(user_id=self.id))
            # THAT LINE ABOVE #
  
            # Init a message class object for every message payload in database
            msgs = [Message(m, user=self) for m in messages]
            # Returns a message container class holding all the message objects, accessible through a class property
            return MessageContainer(msgs)
  </code></pre>
  
  <p>This last class method throws an error:</p>
  
  <pre><code>File "/usr/local/lib/python2.7/dist-packages/kombu/serialization.py", line 356, in pickle_dumps
  
  return dumper(obj, protocol=pickle_protocol)
  
  EncodeError: Can't pickle &lt;class 'cassandra.io.eventletreactor.message'&gt;: attribute lookup cassandra.io.eventletreactor.message failed
  </code></pre>
  
  <p><code>cassandra.io.eventletreactor.message</code> points to a user-defined type in Cassandra that I use as a container for message objects per user.  The line that throws this error is:</p>
  
  <pre><code>messages = db_manager.get("SELECT message FROM keyspace.message_timelines WHERE user_id = {user_id}".format(user_id=self.id))
  </code></pre>
  
  <p>This is the method from <code>DBManager()</code>:</p>
  
  <pre><code>class DBManager(object):
      ... stuff ...
  
      def get(self, query):
          # I do some stuff to prepare the query, namely substituting `WHERE this = that` for `WHERE this = ?` to create a Cassandra prepared statement.
          statement = cassandra.prepare(query_prepared)
          # I want these messages as a dict, not the default namedtuple
          cassandra.row_factory = dict_factory
          # User id is parsed out of query
          results = cassandra.execute(statement, (user_id,))
          rows = results.current_rows
          # rows is a list of dicts, no weird class references or anything in there
          return rows
  </code></pre>
  
  <p>I've read that Celery tasks out of class methods is/was kind of experimental, but I can't figure out why all the other methods qua tasks that use the same instance of <code>DBManager</code> are working.  </p>
  
  <p>The problem seems to be localized to some issue with the user-defined type <code>message</code> that's not playing nice within the Cassandra driver; however, <strong>if I run the <code>get</code> method from <code>DBManager</code> within the Celery task itself, it works.</strong>  That is, if I copy/paste the code that is throwing the error from <code>DBManager.get</code> into <code>User.get_user_messages</code>, it works fine.  If I try to call <code>DBManager.get</code> from within <code>User.get_user_messages</code>, it breaks.</p>
  
  <p>I just can't figure out where the problem is.  I can do all the following <strong>just fine</strong>: </p>
  
  <ol>
  <li>Run the <code>get_user_messages</code> method <strong>without</strong> Celery, and it works.</li>
  <li>Run the <code>get_user_messages</code> method <strong>WITH</strong> Celery if I run the <code>get</code> method code right in the Celery task method itself.</li>
  <li>I can run other methods registered as Celery tasks that point to other methods in <code>DBManager</code> that use the Cassandra driver, even ones that insert the same <code>message</code> user-defined type into the database.</li>
  <li>I've tried pickling <strong>ALL THE THINGS</strong> all the way down myself, and in various combinations, and can't reproduce the error.</li>
  </ol>
  
  <p>What I <strong>have not</strong> tried:</p>
  
  <ol>
  <li>Change serializer to <code>json</code> or <code>yaml</code>.  There are a few convenience items in the db payload that won't serialize with either of those two.</li>
  <li>Use <code>dill</code> instead of <code>pickle</code>.  It seems like this should work without having to switch serializers given that I can get various parts working separately.</li>
  </ol>
  
  <p>I could just say screw it and run the query directly through the Cassandra driver instead of my <code>DBManager</code> class, but I feel like this should be solvable and I'm just missing something really, really obvious, so obvious that I'm not seeing it.  Any suggestions on where to look would be greatly appreciated.</p>
  
  <p>In case of relevance: Cassandra 3.3, CQL 3.4, DataStax python driver 3.1</p>