How do I report results back to the main process from multiple processes? <p>I currently have 18 functions that perform different sets of validations on large XML files. I've created a custom <code>ValidationWarning</code> class that extends <code>UserWarning</code>, and these functions raise warnings for each validation failure. In the end, I need to produce an XLSX report containing all of the failures (identifying information for each failure is available in the <code>ValidationWarning</code> object).</p> <p>I'm planning to spawn a process for each function (I saw examples where tasks were placed in a <code>JoinableQueue</code>, but I don't understand why this is necessary, unless the task list will be modified later). The order of the validation failures is irrelevant, and I don't need to see the results until all of the validations have finished. Should I replace the <code>showwarning()</code> function for each validation function to write to a <code>list</code> returned by <code>Manager()</code>? I can export the results to XLSX after joining all of the processes. Is this better than writing to a <code>Queue</code>? <code>Queue</code> may be faster, but writing results isn't a bottleneck, and I don't understand how I'd solve the following problem (see <a href="http://docs.python.org/2/library/multiprocessing.html#programming-guidelines" rel="nofollow">http://docs.python.org/2/library/multiprocessing.html#programming-guidelines</a>):</p> <blockquote> <p>This means that whenever you use a queue you need to make sure that all items which have been put on the queue will eventually be removed before the process is joined. Otherwise you cannot be sure that processes which have put items on the queue will terminate.</p> </blockquote> <p>Is there a better approach that I'm ignoring? To my understanding, I can't use a <code>catch_warnings</code> context manager in the main process and expect it to catch warnings from each validation process, although I haven't tested it yet (edit: I tested it, and, as expected, it fails). Writing to XLSX directly instead of getting all of the results first also sounds difficult, because I'd need to create a blank XLSX file and update it in each process while managing access to the shared XLSX file.</p> <p>Thank you!</p>