title: Django avoid extra queries using prefetch_related not working
body: <p>I've worked previously with <code>select_related</code> and <code>prefetch_related</code> and it worked great.</p>
  
  <p>I'm working on a current project, and for some reason I can't figure out why my prefetch related query is not working right, hence I get a lot of redundant DB calls.</p>
  
  <p>My models:</p>
  
  <pre><code>class User(models.Model):
      user_extra_info = models.ManyToManyField(
          AppGeneralData,
          through='UserExtraInfo',
          null=True,
          blank=True
      )
  
  class AppGeneralData(models.Model):
      title = models.CharField(max_length=255)
      type = models.PositiveSmallIntegerField(
          choices=GENERAL_DATA_TYPE
      )
  
  class UserExtraInfo(models.Model):
      user = models.ForeignKey(settings.AUTH_USER_MODEL)
      info_item = models.ForeignKey(AppGeneralData)
  </code></pre>
  
  <p>The QuerySet:</p>
  
  <pre><code>User.objects.all().prefetch_related(
      Prefetch(
          'userextrainfo_set',
          queryset=UserExtraInfo.objects.select_related('info_item').all()
      )
  )
  </code></pre>
  
  <p><strong>The problem</strong>:</p>
  
  <p>When iterating over the queryset and calling a sub-query, it doesn't fetch it from the cached queryset:</p>
  
  <pre><code>for user in qs:
      user.userextrainfo_set.filter(
          info_item__type=general_data_type
      ).values_list(
          'info_item__title', flat=True
      ))
  </code></pre>
  
  <p>The sub-query just goes to the DB on every call and I don't understand what I am missing.</p>
  
  <p>Thanks.</p>