title: How can I improve performance on DRF with high CPU time
body: <p>I have a REST api with <strong>DRF</strong> and start to see already a performance hit with 100 objects and 1 user requesting (me - testing).</p>
  
  <p>When requesting the more complex query, I get these results for <strong>CPU</strong>, always 5 - 10s:</p>
  
  <pre><code>Resource    Value
  &gt;User CPU time  5987.089 msec
  System CPU time 463.929 msec
  Total CPU time  6451.018 msec
  Elapsed time    6800.938 msec
  Context switches    9 voluntary, 773 involuntary
  </code></pre>
  
  <p>but the <strong>SQL</strong> query stays <strong>below 100 ms</strong></p>
  
  <p>The more simple queries show similar behaviour, with CPU times around 1s and query time around 20 ms</p>
  
  <p>So far, what I have tried out:</p>
  
  <ul>
  <li>I am doing select_related() and prefetch_related(), which did improve the query time but not CPU time</li>
  <li>I am using Imagekit to generate pictures, on a S3 instance. I removed the whole specification to test and this had minor impact</li>
  <li>I run a method field to fetch user-specific data. Removing this had only minor impact</li>
  </ul>
  
  <hr>
  
  <p>I have checked logs files on the backend and nothing specific shows up here...
  Backend is Nginx - supervisord - gunicorn - postgresql - django 1.8.1</p>
  
  <hr>
  
  <p>Here are the serializer and view:</p>
  
  <pre><code>class ParticipationOrganizationSerializer(ModelSerializer):
      organization = OrganizationSerializer(required=False, read_only=True, )
      bookmark = SerializerMethodField(
          required=False,
          read_only=True,
      )
      location_map = LocationMapSerializer(
          required=False,
          read_only=True,
      )
  
      class Meta:
          model = Participation
          fields = (
              'id',
              'slug',
              'organization',
              'location_map',
              'map_code',
              'partner',
              'looking_for',
              'complex_profile',
              'bookmark',
              'confirmed',
          )
          read_only_fields = (
              'id',
              'slug',
              'organization',
              'location_map',
              'map_code',
              'partner',
              'bookmark',
              'confirmed',
          )
  
      def get_bookmark(self, obj):
          request = self.context.get('request', None)
          if request is not None:
              if(request.user.is_authenticated()):
                  # print(obj.bookmarks.filter(author=request.user).count())
                  try:
                      bookmark = obj.bookmarks.get(author=request.user)
                      # bookmark = Bookmark.objects.get(
                      #     author=request.user,
                      #     participation=obj,
                      # )
                      return BookmarkSerializer(bookmark).data
                  except Bookmark.DoesNotExist:
                      # We have nothing yet
                      return None
                  except Bookmark.MultipleObjectsReturned:
                      # This should not happen, but in case it does, delete all
                      # the bookmarks for safety reasons.
                      Bookmark.objects.filter(
                          author=request.user,
                          participation=obj,
                      ).delete()
                      return None
          return None
  
  
  class ParticipationOrganizationViewSet(ReadOnlyModelViewSet):
      """
      A readonly ViewSet for viewing participations of a certain event.
      """
      serializer_class = ParticipationOrganizationSerializer
      queryset = Participation.objects.all().select_related(
          'location_map',
          'organization',
          'organization__logo_image',
      ).prefetch_related(
          'bookmarks',
      )
      lookup_field = 'slug'
  
  
      def get_queryset(self):
          event_slug = self.kwargs['event_slug']
  
          # Filter for the current event
          # Filter to show only the confirmed participations
          participations = Participation.objects.filter(
              event__slug=event_slug,
              confirmed=True
          ).select_related(
              'location_map',
              'organization',
              'organization__logo_image',
          ).prefetch_related(
              'bookmarks',
          )
  
          # Filter on partners? This is a parameter passed on in the url
          partners = self.request.query_params.get('partners', None)
          if(partners == "true"):
              participations = participations.filter(partner=True)
          return participations
  
  # http://stackoverflow.com/questions/22616973/django-rest-framework-use-different-serializers-in-the-same-modelviewset
      def get_serializer_class(self):
          if self.action == 'list':
              return ParticipationOrganizationListSerializer
          if self.action == 'retrieve':
              return ParticipationOrganizationSerializer
          return ParticipationOrganizationListSerializer
  </code></pre>
  
  <p>Any help is very much appreciated!</p>
  
  <hr>
  
  <p><strong>update</strong></p>
  
  <p>I dumped the data to my local machine and I am observing similar times. I guess this rules out the whole production setup (nginx, gunicorn)?</p>
  
  <hr>
  
  <p><strong>update 2</strong></p>
  
  <p>Here are the results of the profiler.</p>
  
  <p>Also I made some progress in improving the speeds by</p>
  
  <ol>
  <li><p>Simplifying my serializers</p></li>
  <li><p>Doing the tests with curl and having Debug Toolbar off</p></li>
  </ol>
  
  <pre>
      ncalls  tottime percall cumtime percall filename:lineno(function)
      0   0   0   profile:0(profiler)     
      1   0   0   3.441   3.441   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/views.py:442(dispatch)
      1   0   0   3.441   3.441   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/viewsets.py:69(view)
      1   0   0   3.441   3.441   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/django/views/decorators/csrf.py:57(wrapped_view)
      1   0   0   3.44    3.44    /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/mixins.py:39(list)
      1   0   0   3.438   3.438   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/serializers.py:605(to_representation)
      1   0   0   3.438   3.438   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/serializers.py:225(data)
      1   0   0   3.438   3.438   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/serializers.py:672(data)
      344/114 0.015   0   3.318   0.029   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/serializers.py:454(to_representation)
      805 0.01    0   2.936   0.004   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/fields.py:1368(to_representation)
      2767    0.013   0   2.567   0.001   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py:166(send)
      2070    0.002   0   2.52    0.001   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/registry.py:52(existence_required_receiver)
      2070    0.005   0   2.518   0.001   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/registry.py:55(_receive)
      2070    0.004   0   2.513   0.001   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/utils.py:147(call_strategy_method)
      2070    0.002   0   2.508   0.001   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/strategies.py:14(on_existence_required)
      2070    0.005   0   2.506   0.001   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py:86(generate)
      2070    0.002   0   2.501   0.001   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/backends.py:109(generate)
      2070    0.003   0   2.499   0.001   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/backends.py:94(generate_now)
      2070    0.01    0   2.496   0.001   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/backends.py:65(get_state)
      690 0.001   0   2.292   0.003   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py:148(__nonzero__)
      690 0.005   0   2.291   0.003   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py:124(__bool__)
      2070    0.007   0   2.276   0.001   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/backends.py:112(_exists)
      2070    0.01    0   2.269   0.001   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/storages/backends/s3boto.py:409(exists)
      4140    0.004   0   2.14    0.001   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/storages/backends/s3boto.py:282(entries)
      1633    0.003   0   2.135   0.001   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/storages/backends/s3boto.py:288()
      1633    0.001   0   2.129   0.001   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/boto/s3/bucketlistresultset.py:24(bucket_lister)
      2   0   0   2.128   1.064   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/boto/s3/bucket.py:390(_get_all)
      2   0   0   2.128   1.064   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/boto/s3/bucket.py:426(get_all_keys)
      1331    0.003   0   1.288   0.001   /usr/lib/python2.7/ssl.py:335(recv)
      1331    1.285   0.001   1.285   0.001   /usr/lib/python2.7/ssl.py:254(read)
      2   0   0   0.983   0.491   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/boto/connection.py:886(_mexe)
      2   0   0   0.983   0.491   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/boto/s3/connection.py:643(make_request)
      2   0   0   0.983   0.491   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/boto/connection.py:1062(make_request)
      2   0.004   0.002   0.896   0.448   /usr/lib/python2.7/httplib.py:585(_read_chunked)
      2   0   0   0.896   0.448   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/boto/connection.py:393(read)
      2   0   0   0.896   0.448   /usr/lib/python2.7/httplib.py:540(read)
      166 0.002   0   0.777   0.005   /usr/lib/python2.7/httplib.py:643(_safe_read)
      166 0.005   0   0.775   0.005   /usr/lib/python2.7/socket.py:336(read)
      2   0   0   0.568   0.284   /usr/lib/python2.7/httplib.py:793(send)
      2   0   0   0.568   0.284   /usr/lib/python2.7/httplib.py:998(_send_request)
      2   0   0   0.568   0.284   /usr/lib/python2.7/httplib.py:820(_send_output)
      2   0   0   0.568   0.284   /usr/lib/python2.7/httplib.py:977(request)
      2   0   0   0.568   0.284   /usr/lib/python2.7/httplib.py:962(endheaders)
      1   0   0   0.567   0.567   /usr/lib/python2.7/httplib.py:1174(connect)
      1380    0.001   0   0.547   0   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py:82(url)
      1380    0.007   0   0.546   0   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py:72(_storage_attr)
      105 0.009   0   0.528   0.005   /usr/lib/python2.7/socket.py:406(readline)
      2   0   0   0.413   0.207   /usr/lib/python2.7/httplib.py:408(begin)
      2   0   0   0.413   0.207   /usr/lib/python2.7/httplib.py:1015(getresponse)
      2   0   0   0.407   0.203   /usr/lib/python2.7/httplib.py:369(_read_status)
      2750    0.003   0   0.337   0   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/fields.py:399(get_attribute)
      1   0.223   0.223   0.335   0.335   /usr/lib/python2.7/socket.py:537(create_connection)
      2865    0.012   0   0.334   0   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/fields.py:65(get_attribute)
      1610    0.005   0   0.314   0   /home/my_app/.virtualenvs/my_app/src/django-s3-folder-storage/s3_folder_storage/s3.py:13(url)
      1610    0.012   0   0.309   0   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/storages/backends/s3boto.py:457(url)
      690 0.005   0   0.292   0   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/models/fields/utils.py:10(__get__)
      690 0.007   0   0.251   0   /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py:20(__init__)
      2   0   0   0.248   0.124   
      >>>> cutting here, low impact calls
  
  </pre>