title: Using AWS SQS for Job Queuing but minimizing "workers" uptime
body: <p>I am designing my first Amazon AWS project and I could use some help with the queue processing.</p>
  
  <p>This service accepts processing jobs, either via an ASP.net Web API service or a GUI web site (which just calls the API).  Each job has one or more files associated with it and some rules about the type of job.  I want to queue each job as it comes in, presumably using AWS SQS. The jobs will then be processed by a "worker" which is a python script with a .Net wrapper.  The python script is an existing batch processor that cannot be altered/customized for AWS, hence the wrapper in .Net that manages the AWS portions and passing in the correct params to python.</p>
  
  <p>The issue is that we will not have a huge number of jobs, but each job is somewhat compute intensive.  One of the reasons to go to AWS was to minimize infrastructure costs.  I plan on having the frontend web site (Web API + ASP.net MVC4 site) run on elastic beanstalk.  But I would prefer not to have a dedicated worker machine always online polling for jobs, since these workers need to be a bit "beefier" instance (for processing) and it would cost us a lot to mostly sit doing nothing.</p>
  
  <p>Is there a way to only run the web portion on beanstalk and then have the worker process only spin up if there are items in the queue?  I realize I could have a micro "controller" instance always online polling and then have it control the compute spinup, but even that seems like it shouldn't be needed.  Can EC2 instances be started based on a non-zero SQS queue size?  So basically web api adds job to queue, something watches the queue and sees it's non-zero, this triggers the EC2 worker to start, it spins up and polls the queue on startup.  It processes until the queue until empty, then something triggers it to shutdown.</p>