title: Redis performance in Django is very slow
body: <p>A Django webapp reads a redis database and displays the data in a Plotly plot.</p>
  <p>When the redis database has a small number of rows (less than 20k), queries are very fast (less than a second). As the number of rows grows over 50k, queries slow dramatically (8 to 15 seconds). Threads also stumble over each and don't complete in sync.</p>
  <p>This redis database uses a timestamp as a key. Each value is a string in json format. Like this:</p>
  <pre><code>key: 1621367811.769709000 value: {&quot;timestampx&quot;: &quot;1621367811.769709000&quot;, &quot;length&quot;: &quot;96&quot;, &quot;dscp&quot;: &quot;0&quot;, &quot;srcip&quot;: &quot;172.16.1.2&quot;, &quot;destip&quot;: &quot;172.17.4.2&quot;}
  </code></pre>
  <p>My code queries the database. It creates a dictionary where each key is one second. Each value is the total number of megabits transmitted in that second. Code snippet below:</p>
  <pre><code>r = redis.StrictRedis(**redis_config)
  keys = r.keys(pattern=&quot;*&quot;)
  keys.sort()
  start = datetime.now() 
  #keys = keys[0:1000] #For troubleshooting
  vals = r.mget(keys)
  end = datetime.now()
  print(&quot;There are &quot; + str(threading.active_count()) + &quot; active threads&quot;)
  print(&quot;The mget command took &quot; + str(end - start) )
  print(&quot;*******************************&quot;)
  
  &quot;&quot;&quot;
  Code to make a Pyplot scatter plot where the x axis is bps.keys() and the y axis is bps.values()
  &quot;&quot;&quot;
  </code></pre>
  <p>Initial output looks something like this:</p>
  <pre><code>There are 26966 keys in the database.
  There are 5 active threads
  The mget command took 0:00:00.198256
  *******************************
  There are 26852 keys in the database.
  There are 5 active threads
  The mget command took 0:00:00.199248
  *******************************
  </code></pre>
  <p>As the redis database gets more rows (keys), the output look like this:</p>
  <pre><code>There are 74988 keys in the database.
  There are 14 active threads
  The mget command took 0:00:10.768198
  *******************************
  There are 72906 keys in the database.
  There are 73386 keys in the database.
  There are 10 active threads
  The mget command took 0:00:15.715395
  *******************************
  </code></pre>
  <p>Note the entries don't come in order, as different threads complete their commands at different times.
  While I'm seeing this output in the console, the plot in the browser (which is set to refresh every 1 second) freezes for several seconds at a time. This is presumably because the r.mget() command holds up the update.</p>
  <p>If I remove the commented line &quot;keys = keys[0:1000]&quot;, the output remains normal. I expect that, as it limits the number of rows returned to 1,000.</p>
  <p>I'm new to Redis, so I'm not sure where the bottleneck is. Should I be using a method other than mget()? Should I make some modification to the Redis settings? Any help you can offer would be greatly appreciated.</p>