<p><strong>Does (the iterative process) take from these data randomly in each round? Or just take data from each client one batch at a time?</strong></p>
<p>The TFF tutorials all use <a href="https://www.tensorflow.org/federated/api_docs/python/tff/learning/build_federated_averaging_process" rel="noreferrer"><code>tff.learning.build_federated_averaging_process</code></a> which constructs a <a href="https://www.tensorflow.org/federated/api_docs/python/tff/templates/IterativeProcess" rel="noreferrer"><code>tff.templates.IterativeProcess</code></a> that implements the Federated Averaging algorithm (<a href="https://arxiv.org/abs/1602.05629" rel="noreferrer">McMahan et al. 2017</a>). In this algorithm each &quot;round&quot; (one invocation of <code>IterativePocess.next()</code>) processes as many batches of examples on each client as the <a href="https://www.tensorflow.org/api_docs/python/tf/data/Dataset" rel="noreferrer"><code>tf.data.Dataset</code></a> is setup to produce in one iteration. <a href="https://www.tensorflow.org/guide/data" rel="noreferrer">tf.data: Build TensorFlow input pipelines</a> is a great guide for <code>tf.data.Dataset</code>.</p>
<p>The order in which examples are processed is determined by how the <a href="https://www.tensorflow.org/api_docs/python/tf/data/Dataset" rel="noreferrer"><code>tf.data.Dataset</code></a>s that were passed into the <code>next()</code> method as arguments were constructed. For example, in the <a href="https://www.tensorflow.org/federated/tutorials/federated_learning_for_text_generation" rel="noreferrer">Federated Learning for Text Generation</a> tutorial's section titled <a href="https://www.tensorflow.org/federated/tutorials/federated_learning_for_text_generation#load_and_preprocess_the_federated_shakespeare_data" rel="noreferrer">Load and Preprocess the Federated Shakespeare Data</a>, each client dataset is setup with preprocessing pipeline:</p>
<pre class="lang-py prettyprint-override"><code>def preprocess(dataset):
  return (
      # Map ASCII chars to int64 indexes using the vocab
      dataset.map(to_ids)
      # Split into individual chars
      .unbatch()
      # Form example sequences of SEQ_LENGTH +1
      .batch(SEQ_LENGTH + 1, drop_remainder=True)
      # Shuffle and form minibatches
      .shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)
      # And finally split into (input, target) tuples,
      # each of length SEQ_LENGTH.
      .map(split_input_target))
</code></pre>
<p>The next function will iterate over these datasets in its entirety once each invocation of <code>next()</code>, in this case since there is no call to <a href="https://www.tensorflow.org/api_docs/python/tf/data/Dataset?hl=en#repeat" rel="noreferrer"><code>tf.data.Dataset.repeat()</code></a>, <code>next()</code> will have each client see all of its examples once.</p>
<p><strong>Assume that I have a list of tf.data.Datasets each representing a client data. How can I add some randomness to sampling from this list for the next iteration of federated learning?</strong></p>
<p>To add randomness to each client's dataset, one could use the <a href="https://www.tensorflow.org/api_docs/python/tf/data/Dataset?hl=en#shuffle" rel="noreferrer"><code>tf.data.Dataset.shuffle()</code></a> to first randomize the order of yielded examples, and then <a href="https://www.tensorflow.org/api_docs/python/tf/data/Dataset?hl=en#take" rel="noreferrer"><code>tf.data.Dataset.take()</code></a> to take only a sample of that new random ordering. This could be added to the <code>preprocess()</code> method above.</p>
<p>Alternatively, randomness in the <em>selection of clients</em> (e.g. randomly picking which clients participate each round) can be done using any Python library to sub-sample the list of datasets, e.g. Python's <a href="https://docs.python.org/3/library/random.html#random.sample" rel="noreferrer"><code>random.sample</code></a>.</p>
<p><strong>My datasets are not necessarily the same length. When one of them is completely iterated over, does this dataset waits for all other datasets to completely iterate over their data or not?</strong></p>
<p>Each dataset is only iterated over once on each invocation of <code>.next()</code>. This is in line with the synchronous communication &quot;rounds&quot; in <a href="https://arxiv.org/abs/1602.05629" rel="noreferrer">McMahan et al. 2017</a>. In some sense, yes, the datasets &quot;wait&quot; for each other.</p>
What exactly happens when we call IterativeProcess.next on federated training data?
<p>I went through the Federated Learning tutorial. I was wondering how .next function work when we call it on an iterative process.
Assuming that we have train data which is a list of lists. The outer list is a list of clients and the inner lists are batches of data for each client. Then, we create an iterative process, for example, a federated averaging process and we initialize the state.
What exactly happens when we call IterativeProcess.next on this training data. Does it take from these data randomly in each round? Or just take data from each client one batch at a time?</p>
<p>Assume that I have a list of tf.data.Datasets each representing a client data. How can I add some randomness to sampling from this list for the next iteration of federated learning?</p>
<p>My datasets are not necessarily the same length. When one of them is completely iterated over, does this dataset waits for all other datasets to completely iterate over their data or not?</p>