<p>First, a bit of terminology:</p>

<ul>
<li><p>In <strong>ensembles</strong> (as I understand them) you have N models at test time and you combine their <strong>predictions</strong> (by voting, or even better combining probabilistic distributions and using as input for further decoding in case of autoregressive seq2seq decoders). You can have <em>independent ensembles</em> (training each model independently from scratch, with different random initialization) or <em>checkpoint ensembles</em> (taking N last checkpoints, or possibly N checkpoints with best validation score). See e.g. <a href="http://www.statmt.org/wmt17/pdf/WMT39.pdf" rel="nofollow noreferrer">Sennrich et al., 2017</a> for a comparison of these two types of ensembles.</p></li>
<li><p>In <strong>averaging</strong> you average the <strong>weights</strong> of N models, so at test time you have just one averaged model. This gives usually worse results than real ensembles, but it is much faster, so you can afford higher N. If the models are trained completely independently with different random initialization, averaging does not work at all. However, if the models share a reasonable number of initial training steps, averaging may work. A special case is <em>checkpoint averaging</em> where the last N checkpoints are averaged, but you can try even "forking" the training and using "semi-independent" models for averaging (in addition to checkpoint averaging). It may be very useful to use constant or cyclical learning rate, see <a href="https://arxiv.org/pdf/1803.05407.pdf" rel="nofollow noreferrer">Izmailov et al., 2018</a>.</p></li>
</ul>

<p>As for your question, how to do the averaging of Tensorflow checkpoints:
See <a href="https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/utils/avg_checkpoints.py" rel="nofollow noreferrer">avg_checkpoints.py</a> or <a href="https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/bin/t2t_avg_all.py" rel="nofollow noreferrer">t2t-avg-all</a>.</p>
Creating ensemble for sequence to sequence (seq2seq) tensorflow models?
<p>I have a trained a tensorflow <a href="https://www.tensorflow.org/tutorials/seq2seq" rel="nofollow noreferrer">seq2seq</a> model for 30 epochs, and saved a checkpoint for each epoch. What I want to do now is combining the best X of those checkpoints (based on results on a development set). Specifically, I'm looking for a way that lets me average different model weights and merge them into a new model that can be used for decoding. However, there does not seem to be a set way for this, and loading different models can be a bit tricky. But even if this succeeds, I can't find a good answer on how to combine the weights in a new model. </p>

<p>Any help would be greatly appreciated. </p>

<p>Related questions (that do not sufficiently answer in my opinion):</p>

<p><a href="https://stackoverflow.com/questions/38750635/building-multiple-models-in-the-same-graph">Building multiple models in the same graph</a></p>

<p><a href="https://stackoverflow.com/questions/36828600/how-to-load-several-identical-models-from-save-files-into-one-session-in-tensorf?noredirect=1&amp;lq=1">How to load several identical models from save files into one session in Tensorflow</a></p>

<p><a href="https://stackoverflow.com/questions/35464652/how-to-create-ensemble-in-tensorflow">How to create ensemble in tensorflow?</a></p>