<p>Learning rates of the client and server optimizer can be very important in determining the final accuracy of the model.</p>

<p>Very high client learning rates can result in high <em>training</em> accuracy, since the <em>client local</em> training process is overfitting to the <em>client local</em> data (the two classes created in the synthetic split). However these overfit models can be averaged out to very little update to the <em>global</em> model. Lowering the client learning rate can be helpful here.</p>

<p>In <a href="https://arxiv.org/abs/2003.00295" rel="nofollow noreferrer">Reddi 2020</a> it was found that if using an adaptive optimizer (e.g. Adam/Yogi) on the server, tuning the epsilon parameter was necessary for best performance. Adding momentum to SGD also had significant improvements in convergence rates.</p>
Why I create a Non-IID data set like the FedAvg in paper of McMahan but test accuracy of this data set is just only 0.5?
<p>I create a Non-IID data set where I divide 60000 examples(10 classes and every class has 6000 examples) to 200 fragments, and every fragment  has 300 examples. There are 100 clients and I allocate 2 fragments randomly to every client. This is the situation of some clients. 
<a href="https://i.stack.imgur.com/P8Uhu.png" rel="nofollow noreferrer">the situation of some clients</a></p>

<p>I use this data set to train my TFF model. The accuracy of train set is about 0.99 but the accuracy of test set is only about 0.5. I try many times but no change. 
And I think maybe the model is over-fit so I add two dropout to test, but I get the same result. Then I change relu() funcion to leakyrelu(), and change the optimizer function from SGD to Adam, but accuracy also is about 0.5. I don't why. I know Non-IID will cause descend of accuracy and FedAvg can relieve it. TFF use FedAvg to aggregate client model that means I have use FedAvg to be my underlying structure, is it right? But why I get a so low accaracy?  </p>