title: efficiency of django __in lookup for querysets
body: <p>I have a complex database model set up in Django, and I have to do a number of calculations based on filter data. I have a <code>Test</code> object, a <code>TestAttempt</code> object, and a <code>UserProfile</code> object (with a foreign key back to test and a foreign key back to a userprofile). There is a method that I run on a <code>TestAttempt</code> that calculates the test score (based on a number of user-supplied choices compared to the correct answers associated with each test). And then another method that I run on a <code>Test</code> that calculates the average test score based on each of its associated <code>TestAttempt</code>'s But sometimes I only want the average based on a supplied subset of the associated <code>TestAttempt</code>'s that are linked with a particular set of <code>UserProfiles</code>. So instead of calculating the average test score for a particular test this way:</p>
  
  <pre><code>[x.score() for x in self.test_attempts.all()]
  </code></pre>
  
  <p>and then averaging these values.
  I do a query like this:</p>
  
  <pre><code>[x.score() for x in self.test_attempts.filter(profile__id__in=user_id_list).all()]
  </code></pre>
  
  <p>where <code>user_id_list</code> is a particular subset of UserProfile id's for which I want to find the average test score in the form of a list. My question is this: if <code>user_id_list</code> is indeed the entire set of <code>UserProfile</code>'s (so the filter will return the same as <code>self.test_attempts.all()</code>) and most of the time this will be the case, does it pay to check for this case, and if so not execute the filter at all? or is the __in lookup efficient enough that even if <code>user_id_list</code> contains all users it'll be more efficient to run the filter. Also, do I need to worry about making the resulting test_attempts distinct()? or they can't possible turn up duplicates with the structure of my queryset? </p>
  
  <p>EDIT: For anyone who's interested in looking at the raw SQL query, it looks like this without the filter:</p>
  
  <pre><code>SELECT "mc_grades_testattempt"."id", "mc_grades_testattempt"."date", 
  "mc_grades_testattempt"."test_id", "mc_grades_testattempt"."student_id" FROM 
  "mc_grades_testattempt" WHERE "mc_grades_testattempt"."test_id" = 1
  </code></pre>
  
  <p>and this with the filter:</p>
  
  <pre><code>SELECT "mc_grades_testattempt"."id", "mc_grades_testattempt"."date", 
  "mc_grades_testattempt"."test_id", "mc_grades_testattempt"."student_id" FROM 
  "mc_grades_testattempt" INNER JOIN "mc_grades_userprofile" ON 
  ("mc_grades_testattempt"."student_id" = "mc_grades_userprofile"."id") WHERE 
  ("mc_grades_testattempt"."test_id" = 1  AND "mc_grades_userprofile"."user_id" IN (1, 2, 3))
  </code></pre>
  
  <p>note that the array (1,2,3) is just an example</p>