<p>I have several heavy python functions performing various application critical tasks and management stuff running on my cluster.</p>
<p>Here's the thing: Noticing issues with Python threads getting stuck unpredictably.</p>
<p>I even have a service running 10+ threads, where a specific thread gets stuck while others are still actively running their jobs.</p>
<p>Most of these threads contain <code>while True</code> functions.</p>
<p>What is a good way to write reliable threads in Python, with a mechanism to self-recover if stuck</p>

Meet Shah
<p>The good way is to start learning about <a href="https://en.wikipedia.org/wiki/Deadlock" rel="nofollow noreferrer">Deadlocks</a> in multithreading. Threads getting stuck is a symptom of bad code causing the threads to wait for each other's resources.</p>
<p>Otherwise, make a &quot;retry x times&quot; and abort. And now you will need to make sure you don't lose your data by aborting. You should perform transactional operations and save the aborted jobs for later. This is very important for calls to databases and while dealing with inter-process messages. A common way to deal with this is having a (external) message queue, moving the messages of failed operations to dead letter queues (DLQ).</p>
