title: contourf does not show the color outside the levels range without colorbar
body: <p>I am trying to make some contourf figures with the option 'extend=both'.
  However, the colors outside the color range (both max and min) does not show in the figure.
  When calling the colorbar function after contourf, the outside colors appears, but I want to make figures without colorbars.</p>
  <p>Here is the sample code.</p>
  <pre><code>import numpy as np
  import matplotlib.pyplot as plt
  
  x=np.arange(10)
  y=np.arange(10)
  
  z=np.zeros((10,10))
  for n in range(10):
      for m in range(10):
          z[n,m]=x[n]*y[m]
  
  N=255.
  rgb=[(187/N,220/N,200/N),
  (120/N,185/N,171/N), 
  ( 52/N,150/N,142/N), 
  ( 26/N,105/N, 95/N)] 
  
  interval=[1,5,10,20,50]
  
  cf=plt.contourf(x,y,z,colors=rgb,levels=interval,extend='both')
  cf.cmap.set_under((255/N, 255/N,229/N))
  cf.cmap.set_over((0/N, 60/N,48/N))
  #plt.colorbar()
  
  plt.savefig('test1')
  plt.close()
  
  cf=plt.contourf(x,y,z,colors=rgb,levels=interval,extend='both')
  cf.cmap.set_under((255/N, 255/N,229/N))
  cf.cmap.set_over((0/N, 60/N,48/N))
  plt.colorbar()
  
  plt.savefig('test2')
  plt.close()
  
  </code></pre>
  <p><a href="https://i.stack.imgur.com/IgyMN.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/IgyMN.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.stack.imgur.com/oLQJL.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/oLQJL.png" alt="enter image description here" /></a></p>
  <p>test1.png does not show the outside color (&lt;1 and &gt;50). I want to get the figure like test2.png, but without colorbar.</p>
  <p>Does anyone know how to solve this problem?</p>
  <p>My environment is,
  Python=3.6.9
  Matplotlib=2.1.1</p>
  <p>Thank you.</p>