Take the 2-minute tour ×

I'm trying to automate a process that collects data on one (or more) AWS instance(s), uploads the data to S3 hourly, to be retrieved by a decoupled process for parsing and further action. As a first step, I whipped up some crontab-initiated shell script (running in Ubuntu 12.04 LTS) that calls the boto utility s3multiput.

For the most part, this works fine, but very occasionally (maybe once a week) the file fails to appear in the s3 bucket, and I can't see any error or exception thrown to track down why.

I'm using the s3multiput utility included with boto 2.6.0. Python 2.7.3 is the default python on the instance. I have an IAM Role assigned to the instance to provide AWS credentials to boto.

I have a crontab calling a script that calls a wrapper that calls s3multiput. I included the -d 1 flag on the s3multiput call, and redirected all output on the crontab job with 2>&1 but the report for the hour that's missing data looks just like the report for the hour before and the hour after, each of which succeeded.

So, 99% of the time this works, but when it fails I don't know why and I'm having trouble figuring where to look. I only find out about the failure later when the parser job tries to pull the data from the bucket and it's not there. The data is safe and sound in the directory it should have uploaded from, so I can do it manually, but would rather not have to.

I'm happy to post the ~30-40 lines of related code if helpful, but wondered if anybody else had run into this and it sounded familiar.

Some grand day I'll come back to this part of the pipeline and rewrite it in python to obviate s3multiput, but we just don't have dev time for that yet.

How can I investigate what's going wrong here with the s3multiput upload?

share|improve this question

1 Answer 1

First, I would try updating boto; a commit to the development branch mentions logging when a multipart upload fails. Note that doing so will require using s3put instead, as s3multiput is being folded into s3put.

share|improve this answer
Hmm. Thanks for the tip - I've had a look at the github repo and some of the updates make a lot of sense, but we generally try to stay on production/stable releases and 2.6.0 is the latest available from PyPI. I haven't yet found mention of logging on fail, just logging when multipart isn't available due to missing dependencies. I'll keep looking, but can you point me to commit to investigate? Very much appreciated. –  Dave Nov 4 '12 at 20:49
github.com/boto/boto/commit/… hope that helps. As far as stable code, in a major project like that, I wouldn't worry too much about that; the company I'm freelancing at recently switched to the development version of sleekxmpp to get more features. I'd consider the development branches of these projects more stable than "stable" releases of my own. –  pydsigner Nov 6 '12 at 0:30

Your Answer


