title: Tracking down S3 Costs (S3FS)
body: <p>During a transition, our S3 costs jumped a lot due to ListBucket and HeadObject calls.  We are trying to figure out how to debug a sudden increase in our S3 costs. We made some changes that should NOT have affected it but the major change seems to be </p>
  
  <ul>
  <li>10-20X increase in HeadObject calls</li>
  <li>Sudden appearance of ListBucket calls</li>
  </ul>
  
  <p><a href="https://i.stack.imgur.com/V97SN.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/V97SN.png" alt="enter image description here"></a></p>
  
  <p>I have attached a chart showing the jump between the April 10, 2018 and April 14, 2018. The dates in between, we made the following changes</p>
  
  <ul>
  <li>Changed from (debian 8) S3FS v1.61 (super old from 2012, not even in Github) to v1.84 (latest)
  
  <ul>
  <li><a href="https://github.com/s3fs-fuse/s3fs-fuse" rel="nofollow noreferrer">https://github.com/s3fs-fuse/s3fs-fuse</a></li>
  </ul></li>
  <li>Moved from N. Virginia to N. California AZ (10% higher cost)</li>
  <li>The giant yellow bars are showing the moving of the files using Amazon CLI (April 11 to 13) </li>
  <li>In order to try to calm this down, we added to the mount command in /etc/fstab the following: 
  <code>noatime,stat_cache_expire=3600,enable_noobj_cache</code></li>
  <li>The bars that look uneven starting Apr 14 are now stable around $25/day </li>
  </ul>
  
  <p>Options that are already there were there since the start (no change)
  <code>_netdev,allow_other,use_cache=/tmp,umask=0000,use_path_request_style,ensure_diskfree=10240</code></p>
  
  <p>We have done the following to try to debug this</p>
  
  <ul>
  <li>Enabled S3 Logging</li>
  <li>Dumped the logs into Athena and then CSV export into MySQL</li>
  <li>These logs are just 1 days worth</li>
  <li>Screenshot "query 1" shows that there is 4.8m hits into a path ... basically, we think it is traversing the entire directory tree (with most like about 100k files) looking for a file if it exists
  Screenshot "query 2" shows the same thing (kind of) where it is also doing down a path </li>
  </ul>
  
  <p><a href="https://i.stack.imgur.com/GqWXX.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/GqWXX.png" alt="enter image description here"></a></p>
  
  <p><a href="https://i.stack.imgur.com/YvxUK.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/YvxUK.png" alt="enter image description here"></a></p>
  
  <p>Not really sure what else to do but our normal bill of about $5/day (including other services) is now about $25/day (5x increase) .. with the /etc/fstab changes, it is down to $13/day but still trying to get it to $5/day if we can get back to the zero ListBucket calls and 20% of the HeadObject calls.</p>
  
  <p>Any ideas on what to try greatly appreciated.</p>