<p>I think this paper answers your question:
<a href="https://arxiv.org/pdf/1507.04296.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1507.04296.pdf</a></p>

<p>This paper runs a central learner, with a central replay memory. Also, there are <code>n</code> workers which are replicates of the central learner, each has its own replay memory. Each worker fills its own replay memory, and in each train step can use its own replay memory (if it is large enough) or use the central replay memory. Before each action selection the weights of the network are synched with the server, and after each single step of training, the gradients are sent back to the server. </p>

<p>Also consider:
<a href="https://arxiv.org/pdf/1602.01783.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1602.01783.pdf</a></p>

<p>which proposes A3C and later A2C is proposed which is a simpler version of A3C. The point is that, asynchronous Q-learning algorithm did not get much of attention because of the A3C’s performance. Basically, it is not efficient to use distributed DQN algorithm since replay memory needs moving a lot of data back and forth from different workers to the servers. Indeed, A3C is proposed to solve this problem with the replay memory, which runs one instance of the model and env in each worker and only asynchronously updates the weights. </p>

<p>I hope this has answered your question. </p>

<p>Afshin</p>
Training DDQN concurrently
<p>My friend and I are training a DDQN for learning 2D soccer. I trained the model about 40.000 episodes but it tooks 6 days. Is there a way for training this model concurrently?</p>

<p>For example, I have 4 core and 4 thread and each thread trains the model 10.000 times concurrently. Therefore, time to training 40.000 episodes are reduced 6 days to 1,5 days like parallelism of for loop.</p>

<p>EDIT : If we train a model 10.000 episodes in 4 threads separately, would forming a new model consisting of the average of those trained models give the effect of training 40.000 episodes or would it be a model that was trained 10.000 episodes but a better one?</p>