<blockquote>
  <p>are any additional code changes needed to create a custom tier of any size?</p>
</blockquote>

<p>No; no changes are needed to the MNIST sample to get it to work with different number or type of worker. To use a <code>tf.estimator.Estimator</code> on CloudML engine, you must have your program invoke <code>learn_runner.run</code>, as <a href="https://github.com/GoogleCloudPlatform/cloudml-dist-mnist-example/blob/79f07aef969995f0e4445311b9771735fbd7173b/trainer/task.py#L118" rel="nofollow noreferrer">exemplified</a> in the samples. When you do so, the framework reads in the <a href="https://cloud.google.com/ml-engine/docs/concepts/trainer-considerations#use_tf_config" rel="nofollow noreferrer"><code>TF_CONFIG</code></a> environment variables and populates a <a href="https://www.tensorflow.org/api_docs/python/tf/estimator/RunConfig" rel="nofollow noreferrer"><code>RunConfig</code></a> object with the relevant information such as the <a href="https://www.tensorflow.org/api_docs/python/tf/train/ClusterSpec" rel="nofollow noreferrer"><code>ClusterSpec</code></a>. It will automatically do the right thing on Parameter Server nodes and it will use the provided Estimator to start training and evaluation.</p>

<p>Most of the magic happens because <code>tf.estimator.Estimator</code> automatically uses a device setter that distributes ops correctly. That device setter uses the cluster information from the <code>RunConfig</code> object whose constructor, by default, uses TF_CONFIG to do its magic (e.g. <a href="https://github.com/tensorflow/tensorflow/blob/593dc8e5d65f4db93e8f5fced772abb3531a9752/tensorflow/python/estimator/estimator.py#L790" rel="nofollow noreferrer">here</a>). You can see where the device setter is being used <a href="https://github.com/tensorflow/tensorflow/blob/593dc8e5d65f4db93e8f5fced772abb3531a9752/tensorflow/python/estimator/estimator.py#L627" rel="nofollow noreferrer">here</a>.</p>

<p>This all means that you can just change your <code>config.yaml</code> by adding/removing workers and/or changing their types and things should generally just work.</p>

<p>For sample code using a custom model_fn, see the <a href="https://github.com/GoogleCloudPlatform/cloudml-samples/blob/master/census/customestimator/trainer/model.py" rel="nofollow noreferrer">census/customestimator</a> example.</p>

<p>That said, please note that as you add workers, you are increasing your effective batch size (this is true regardless of whether or not you are using <code>tf.estimator</code>). That is, if your <code>batch_size</code> was 50 and you were using 10 workers, that means each worker is processing batches of size 50, for an effective batch size of 10*50=500. Then if you increase the number of workers to 20, your effective batch size becomes 20*50=1000. You may find that you may need to decrease your learning rate accordingly (linear seems to generally work well; <a href="https://arxiv.org/abs/1706.02677" rel="nofollow noreferrer">ref</a>).</p>

<blockquote>
  <p>I poked around some of the other ML Engine samples and found that
  reddit_tft uses distributed training, but they appear to have defined
  their own runconfig.cluster_spec within their trainer package:
  task.pyeven though they are also using the Estimator API. So, is there
  any additional configuration needed?</p>
</blockquote>

<p>No additional configuration needed. The reddit_tft sample does instantiate its own <code>RunConfig</code>, however, the constructor of <code>RunConfig</code> grabs any properties not explicitly set during instantiation by using <code>TF_CONFIG</code>. And it does so only as a convenience to figure out how many Parameter Servers and workers there are.</p>

<blockquote>
  <p>Does any of this change if the config.yaml specifies using GPUs?</p>
</blockquote>

<p>You should not need to change anything to use <code>tf.estimator.Estimator</code> with GPUs, other than possibly needing to manually assign ops to the GPU (but that's not specific to CloudML Engine); see <a href="https://www.tensorflow.org/tutorials/using_gpu" rel="nofollow noreferrer">this article</a> for more info. I will look into clarifying the documentation.</p>
Code changes needed for custom distributed ML Engine Experiment
<p>I completed this <a href="https://cloud.google.com/ml-engine/docs/tutorials/distributed-tensorflow-mnist-cloud-datalab" rel="nofollow noreferrer">tutorial</a> on distributed tensorflow experiments within an ML Engine experiment and I am looking to define my own custom tier instead of the <code>STANDARD_1</code> tier that they use in their <a href="https://github.com/GoogleCloudPlatform/cloudml-dist-mnist-example/blob/master/config/config.yaml" rel="nofollow noreferrer">config.yaml</a> file.  If using the <code>tf.estimator.Estimator</code> API, are any additional code changes needed to create a custom tier of any size?  For example, the article suggests: "If you distribute 10,000 batches among 10 worker nodes, each node works on roughly 1,000 batches." so this would suggest the config.yaml file below would be possible</p>

<pre><code>trainingInput:
  scaleTier: CUSTOM
  masterType: complex_model_m
  workerType: complex_model_m
  parameterServerType: complex_model_m
  workerCount: 10
  parameterServerCount: 4
</code></pre>

<p>Are any code changes needed to the mnist tutorial to be able to use this custom configuration? Would this distribute the X number of batches across the 10 workers as the tutorial suggests would be possible?  I poked around some of the other ML Engine samples and found that reddit_tft uses distributed training, but they appear to have defined their own <code>runconfig.cluster_spec</code> within their trainer package: <a href="https://github.com/GoogleCloudPlatform/cloudml-samples/blob/master/reddit_tft/trainer/task.py#L205" rel="nofollow noreferrer">task.py</a>even though they are also using the Estimator API. So, is there any additional configuration needed? My current understanding is that if using the Estimator API (even within your own defined model) that there should not need to be any additional changes.</p>

<p>Does any of this change if the config.yaml specifies using GPUs? This <a href="https://cloud.google.com/ml-engine/docs/how-tos/using-gpus" rel="nofollow noreferrer">article</a> suggests for the Estimator API "No code changes are necessary as long as your ClusterSpec is configured properly. If a cluster is a mixture of CPUs and GPUs, map the ps job name to the CPUs and the worker job name to the GPUs." However, since the config.yaml is specifically identifying the machine type for parameter servers and workers, I am expecting that within ML-Engine the ClusterSpec will be configured properly based on the config.yaml file. However, I am not able to find any ml-engine documentation that confirms no changes are needed to take advantage of GPUs.  </p>

<p>Last, within ML-Engine I am wondering if there are any ways to identify usage of different configurations? The line "If you distribute 10,000 batches among 10 worker nodes, each node works on roughly 1,000 batches." suggests that the use of additional workers would be roughly linear, but I don't have any intuition around how to determine if more parameter servers are needed? What would one be able to check (either within the cloud dashboards or tensorboard) to determine if they have a sufficient number of parameter servers?</p>