title: Scheduled, auto-scaled jobs in Google Cloud Platform
body: <p>I'm kinda new to cloud development, so go easy on me. I'm not even sure what I'm asking is <em>possible</em>.</p>
  
  <p>I'm trying to figure out how to schedule a job (every hour or two probably) to execute a worker process that will be packaged inside a custom Docker image. The worker process will check for any "scheduled work" it needs to perform (as requested by users of the application) by connecting to the application's data store, then execute any outstanding work. Once done, I want everything to tear down until the next schedule interval, so I'm not incurring any costs during the down time.</p>
  
  <p>One complication is that the scheduled work will be of different types. Ideally, I want to spin up one instance of my image <em>per type</em>, passing that type into the worker process, so it knows what type of scheduled work it is dealing with. Once it's done with its type of scheduled work, it can exit, even if other containers are still running their scheduled work.</p>
  
  <h1>Summary of requirements</h1>
  
  <ul>
  <li>Full control over the Docker image. I have custom tools and code I will need to put in the image.</li>
  <li>Time-based schedule to instigate execution of the code in the Docker image.</li>
  <li>Scheduled work execution may take quite some time (maybe up to 10 or 15 minutes) to complete.</li>
  <li>Custom, programmatically-controlled scaling and partitioning that creates <em>x</em> number of Docker containers, each with their own type of scheduled work. The determination of <em>x</em> would be based on data in the application's data store.</li>
  <li>During down-time, have absolutely no incurred costs.</li>
  </ul>
  
  <h1>Things I've tried/considered</h1>
  
  <h3>K8s</h3>
  
  <p>Seems to require that the cluster be running at all times. It's also unclear to me how I would have different nodes in a cluster responsible for different types of scheduled work.</p>
  
  <h3>AppEngine and Cloud Scheduler</h3>
  
  <p>I'd need to use the Flexible environment because of my bespoke Docker image. According to <a href="https://cloud.google.com/appengine/docs/the-appengine-environments" rel="nofollow noreferrer">this</a>, that means I have to leave at least one instance running at all times.</p>
  
  <p>Again, I'm not sure how I would scale it such that different instances deal with different types of scheduled work.</p>
  
  <h1>Questions</h1>
  
  <p>So, I have a few questions about all this:</p>
  
  <ol>
  <li>First and foremost, is this even possible with GCP? My research thus far suggests it is not.</li>
  <li>What would be the best approach to getting as close to my requirements as possible?</li>
  </ol>