<p>The choice of how to split the datasets is really up to the evaluator and what they are trying to accomplish. The preprocessed datasets in TFF (from <code>tff.simulation.datasets</code>) are usually only split into two, but they can be rejoined and split again in whatever way is desired.</p>
<p>One thing to consider: there are (at least) two dimensions that may be interesting to split on for federated learning.</p>
<ol>
<li><em>examples</em>: Splitting a single client's dataset into train, test, and validation. This could possibly be seen as most analogous to the centralized training regime. Most TFF datasets use this.</li>
<li><em>users</em>: Splitting users into train, test, and heldout <em>users</em> might be particularly interesting in the federated regime. This might be able to answer how well a global model generalizes to <em>unseen</em> users, but might be heavily affected by the non-iid ness of the individual datasets and splits. This is used in a few TFF provided datasets.</li>
</ol>
<p>Furthermore, both of these could be time based (if there is a notion of time), for example splitting each clients dataset into &quot;previous day&quot; (train) and &quot;next day&quot; (test). Or, as is often the case in practice with cross-device FL, splitting by time of day (users available for training at night maybe different than mid-day), <a href="https://arxiv.org/abs/1904.10120" rel="nofollow noreferrer">Eichner 2019</a> performed some experiments using this setup.</p>
<p><strong>Note</strong>: the <a href="https://www.tensorflow.org/federated/api_docs/python/tff/simulation/datasets/stackoverflow/load_data" rel="nofollow noreferrer"><code>tff.simulation.datasets.stackoverflow.load_data</code></a> does have three splits named <code>train</code>, <code>held_out</code> and <code>test</code>. Please read the documentation carefully as it utilizes both types of splitting mentioned above.</p>
TFF: How split data of each client
<p>Why in the federated learning task, we don't split our dataset to train, test and validation, we make only train and test .</p>