<p>In <a href="https://arxiv.org/abs/1602.05629" rel="nofollow noreferrer">McMahan et al., 2017</a>, the clients communicate the <em>model weights</em> after local training to the server, which are then averaged and re-broadcast for the next round. No server optimizer needed, the averaging step updates the global/server model.</p>
<p><a href="https://www.tensorflow.org/federated/api_docs/python/tff/learning/build_federated_averaging_process" rel="nofollow noreferrer"><code>tff.learning.build_federated_averaging_process</code></a> takes a slight different approach: the <em>delta</em> of the model weights the client received and the model weights after local training is sent back to the server. This <em>delta</em> can be though of as a pseudo-gradient, allowing the server to apply it to the global model using standard optimization techniques. <a href="https://arxiv.org/abs/2003.00295" rel="nofollow noreferrer">Reddi et al., 2020</a> delves into this formulation and how adaptive optimizers (Adagrad, Adam, Yogi) on the server can greatly improve convergence rates. Using SGD without momentum as the server optimizer, with a learning rate of <code>1.0</code>, exactly recovers the method described in McMahan et al., 2017.</p>
Learning rate setting when calling the function tff.learning.build_federated_averaging_process
<p>I'm carrying out a federated learning process and use the function  tff.learning.build_federated_averaging_process to create an iterative process of federated learning. As mentioned in the TFF tutorial,  this function has two arguments called client_optimizer_fn and server_optimizer_fn, which in my opinion, represent the optimizer for client and server, respectively. But in the FedAvg paper, it seems that only clients carry out the optimization while the server only do the averaging operation, so what exactly is the server_optimizer_fn doing and what does its learning rate mean?</p>