<p>Generally, TFF considers the feeding of data to be part of the "Python driver loop", which is a helpful distinction to make when writing TFF code.</p>

<p>In fact, when writing TFF, there are generally three levels at which one may be writing:</p>

<ol>
<li>TensorFlow defining local processing (IE, processing that will happen on the clients, or on the server, or in the aggregators, or at any other placement one may want, but only a <em>single</em> placement.</li>
<li>Native TFF defining the way data is communicated <em>across</em> placements. For example, writing <code>tff.federated_sum</code> inside of a <code>tff.federated_computation</code> decorator; writing this line declares "this data is moved from clients to server, and aggregated via the sum operator".</li>
<li>Python "driving" the TFF loop, e.g. running a single round. It is the job of this final level to do what a "real" federated learning runtime would do; one example here would be selecting the clients for a given round.</li>
</ol>

<p>If this breakdown is kept in mind, using a generator or some other lazy-evaluation-style construct to feed data in to a federated computation becomes relatively simple; it is just done at the Python level.</p>

<p>One way this could be done is via the <a href="https://www.tensorflow.org/federated/api_docs/python/tff/simulation/ClientData#create_tf_dataset_for_client" rel="nofollow noreferrer"><code>create_tf_dataset_for_client</code></a> method on the <code>ClientData</code> object; as you loop over rounds, your Python code can select from the list of <code>client_ids</code>, then you can instantiate a new list of <code>tf.data.Datasets</code>and pass them in as your new set of client data. An example of this relatively simple usage would be <a href="https://github.com/tensorflow/federated/blob/master/tensorflow_federated/python/research/simple_fedavg/emnist_fedavg.py#L161" rel="nofollow noreferrer">here</a>, and a more advanced usage (involving defining a custom <code>client_datasets_fn</code> which takes <code>client_id</code> as a parameter, and passing it to a separately-defined training loop would be <a href="https://github.com/tensorflow/federated/blob/master/tensorflow_federated/python/research/gans/experiments/emnist/train.py#L354" rel="nofollow noreferrer">here</a>, in the code associated to <a href="https://arxiv.org/abs/1911.06679" rel="nofollow noreferrer">this paper</a>.</p>

<p>One final note: instantiating a <code>tf.data.Dataset</code> does not actually load the dataset into memory; the dataset is only loaded in when it is iterated over. One helpful tip I have received from the lead author of <code>tf.data.Dataset</code> is to think of <code>tf.data.Dataset</code> more as a "dataset recipe" than a literal instantiation of the dataset itself. It has been suggested that perhaps a better name would have been <code>DataSource</code> for this construct; hopefully that may help the mental model on what is actually happening. Similarly, using the <code>tff.simulation.ClientData</code> object generally shouldn't really load anything into memory until it is iterated over in training on the clients; this should make some nuances around managing dataset memory simpler.</p>
Implement data generator in federated training
<p>(I have posted the question on <a href="https://github.com/tensorflow/federated/issues/793" rel="nofollow noreferrer">https://github.com/tensorflow/federated/issues/793</a> and maybe also here!)</p>

<p>I have customized my own data and model to federated interfaces and the training converged. But I am confused about an issue that in an images classification task, the whole dataset is extreme large and it can't be stored in a single <code>federated_train_data</code> nor be imported to memory for one time. So I need to load the dataset from the hard disk in batches to memory real-timely and use <code>Keras model.fit_generator</code> instead of <code>model.fit</code> during training, the approach people use to deal with large data.</p>

<p>I suppose in <code>iterative_process</code> shown in image classification tutorial, the model is fitted on a fixed set of data. Is there any way to adjust the code to let it fit to a data generator?I have looked into the source codes but still quite confused. Would be incredibly grateful for any hints.</p>