title: Need a clear simple approach to distributed learning in Tensorflow/Keras
body: <p>I have multiple machines, some with GPUs and some others not. I also have a keras model that works fine on a single machine but I want to train it in a distributed mode because I want to test it with a huge dataset and with a bigger number of layers.
  There is quite a lot of pages discussing the distribution strategy in tf.distribute, but at the same time there are a lot other pages showing how to do it with encapsulating keras model with an estimator, setting up the TF_CONFIG parameter and then call <code>tf.estimator.train_and_evaluate</code>.
  I used the second approach personally as it was more straightforward and am struggling to tune and debug it. It works anyway, but I am very confused what is the point in all of strategy-related stuff as I don't see any use them in the second approach, and the documentation is not helping to clear it.</p>
  
  <p>I also have some doubt if my file setting environment is correct: My understanding is that the PS server is going to hold the model parameters and the chief server is going to administer the whole training process, distributing data, and saving summaries and checkpoints. So I assume that:</p>
  
  <p>0- I need only one chief server, at least one PS server, possibly some workers, and one evaluator. The data and parameter sharing and communication between all of these servers is done by system and I am not engaged in it.</p>
  
  <p>1- The main python code for all machines should be exactly the same, except in TF_CONFIG definition that defines the task and index for that specific machine.</p>
  
  <p>2- I should have one shared copy of data in a folder available to all chief and workers.</p>
  
  <p>3- I should have one shared log directory accessible by all machines as defined in tf.estimator.RunConfig.</p>
  
  <p>4- Having this setting then a piece of code such as below will do the job (assuming the <code>model</code> has been defined elsewhere and the <code>read_datasets</code> function returns features and labels for running the model):</p>
  
  <pre><code>runConfig = tf.estimator.RunConfig(
          session_config=config,
          model_dir=log_dir,
          save_summary_steps=1,
          save_checkpoints_steps=train_steps
          )
  estimator = tf.keras.estimator.model_to_estimator(model, model_dir=log_dir, config=runConfig)
  train_spec = tf.estimator.TrainSpec(input_fn=lambda: read_datasets(...), max_steps=epochs*train_steps)
  eval_spec = tf.estimator.EvalSpec(input_fn=lambda: read_datasets(...), start_delay_secs=1, throttle_secs=1)
  tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
  </code></pre>
  
  <p>Although the above approach seems to work fine, I still have some difficulties understanding how the chief is partitioning the dataset among the workers and how to set the train_step and batch_size in this approach. Also I don't know how can I report accuracy and other metrics such as precision/recall/F1 in addition to loss when running the <code>tf.estimator.evaluate</code> without writing a custom model_fn for my encapsulated keras estimator.</p>