title: Deleting files in Ceph does not free up space
body: <p>I am using Ceph, uploading many files through radosgw.  After, I want to delete the files.  I am trying to do that in Python, like this:</p>
  
  <pre><code>bucket = conn.get_bucket(BUCKET)
  for key in bucket.list():
      bucket.delete_key(key)
  </code></pre>
  
  <p>Afterwards, I use <code>bucket.list()</code> to list files in the bucket, and this says that the bucket is now empty, as I intended.</p>
  
  <p>However, when I run <code>ceph df</code> on the mon, it shows that the OSDs still have high utilization (e.g. <code>%RAW USED 90.91</code>).  If I continue writing (thinking that the status data just hasn't caught up with the state yet), Ceph essentially locks up (100% utilization).</p>
  
  <p>What's going on?</p>
  
  <p>Note: I do have these standing out in <code>ceph status</code>:</p>
  
  <pre><code>   health HEALTH_WARN
              3 near full osd(s)
              too many PGs per OSD (2168 &gt; max 300)
              pool default.rgw.buckets.data has many more objects per pg than average (too few pgs?)
  </code></pre>
  
  <p>From what I gather online, this wouldn't cause my particular issue.  But I'm new to Ceph and could be wrong.</p>
  
  <p>I have one mon and 3 OSDs.  This is just for testing.</p>