<p>Running <code>eval_process</code> multiple rounds on the same <code>test_data</code> will not produce new information, and is expected to yield the same result every time. These results will be <em>stable</em> in the sense they don't change, but probably are not interesting.</p>
<p>Running <code>eval_process</code> on multiple rounds, using different <code>test_data</code> each round can be thought of as sampling a cohort of clients from the larger population to get an <em>estimate</em> of model quality. Computing many estimates from multiple samples can be used with statistical techniques, with more rounds leading to more <em>stable</em> an improved estimates of model quality.</p>
<p>Presumably this is the technique used in <a href="https://proceedings.mlsys.org/paper/2019/hash/bd686fd640be98efaae0091fa301e613-Abstract.html" rel="nofollow noreferrer">1</a> and <a href="https://arxiv.org/pdf/2102.08503.pdf" rel="nofollow noreferrer">2</a> which describe later aggregation services.</p>
Use tff.learning.build_federated_evaluation for multiple rounds
<p>I want to evaluate my federated learning model using <code>tff.learning.build_federated_evaluation</code>. Initially, got reasonable results. but can I run the evaluation process for multiple rounds (as in the training phase done <a href="https://stackoverflow.com/questions/71822452/epochs-vs-rounds-in-federated-learning">here</a>) to get more stable results?</p>
<p>The evaluation code is provided below.</p>
<pre><code>train, test = source.train_test_client_split(source, 2,seed=0)
test_client_ids = test.client_ids

test_data= [test.create_tf_dataset_from_all_clients().map(reshape_data)
.batch(batch_size=10)
 for c in test_client_ids]

eval_process=tff.learning.build_federated_evaluation(model_fn)

eval_process(state.model, test_data)

</code></pre>
<p>The evaluation output.</p>
<pre><code>OrderedDict([('eval',
              OrderedDict([('sparse_categorical_accuracy', 0.53447974),
                           ('loss', 1.0230521),
                           ('num_examples', 11514),
                           ('num_batches', 1152)]))])
</code></pre>