<p>I have an application which does this:</p>
<pre><code>subprocess.Popen([&quot;python3&quot;, &quot;-u&quot;, &quot;sub-program-1.py&quot;])
</code></pre>
<p>So Python program can start multiple long-lived processes on demand.</p>
<p>If I stop main Python program and start again, it knows that <code>sub-program-1.py</code> should be started, because there is a record about status in DB that tells it.</p>
<p>So simply it works fine when there is only one replica of Docker container, pod, virtual machine or whatever you call it.</p>
<hr />
<p>If I scale an app to 3 replicas, <code>subprocess</code> fails to achieve it.</p>
<ol>
<li><p>Each Docker container starts <code>sub-program-1.py</code>, while I want to start on one container</p>
</li>
<li><p>If one container fails, an app should be smart enough to failover <code>sub-program-1.py</code> to another container</p>
</li>
<li><p>An app should be smart enough to balance subprocesses across containers, for example: <code>sub-program-1.py</code> - <code>sub-program-9.py</code> ideally should be spread by putting 3 processes per container so in total there are 9 subprocesses running - I don't need this to be precise, most simplest solution is fine to balance it</p>
</li>
</ol>
<hr />
<p>I've tried to explore RQ (Redis Queue) and similar solutions, but they are heavily focused on tasks, ideally short-living. In my case, they are long-lived processes. E.g. <code>sub-program-1.py</code> can live for months and years.</p>
<hr />
<p>The scheme is this:</p>
<p>Main Python app -&gt; <code>sub-program-1.py</code>, <code>sub-program-2.py</code>, etc.</p>
<p>Any simple solution exists here without overhead?</p>
<p>Is writing statuses of each sub program to DB an option (also detecting when sub process fails to failover it to another container based on statuses in DB) or would you incorporate additional tool to solve <code>subprocess</code> scaling issue?</p>
<hr />
<p>Another option is to start <code>sub-program-1.py</code> on all containers and scale operations inside of it. <code>sub-program-1.py</code> basically calls some third-party APIs and does some operations based on user preference. So scaling those API calls based on each user preference is complicated, it has multiple threads in background when calling APIs simultaneously. In short, <code>sub-program-1.py</code> is tied to user1, <code>sub-program-2.py</code> is tied to user2, etc. So is it worth to make it complex by choosing this option?</p>
<hr />
<p>Update</p>
<p>If <code>subprocess</code> is used only in standalone apps and nobody tried to implement this mechanism at findable scale on Github, libraries, etc.</p>
<p>How would you solve this issue in Python?</p>
<p>I think about these entries in DB:</p>
<pre><code>ProcessName        ProcessHostname        LastHeartBeat            Enabled
process1            host-0                2022-07-10 15:00        true
process2            null                    null                    true
process3            host-1                2022-07-10 14:50        true
</code></pre>
<p>So to solve three points that I wrote above:</p>
<ol>
<li><p>Each container tries to pick up process that is not already picked up (where is <code>null</code> or old date of <code>LastHeartBeat</code>). When first container picked up a process, it writes date to
<code>LastHeartBeat</code> and then uses <code>subprocess</code> to start a process. Other containers cannot pick up if <code>LastHeartBeat</code> is constantly updated.</p>
</li>
<li><p>If process fails, it doesn't write <code>LastHeartBeat</code> so other container picks up the process as described in point 1. If failed container cannot reach DB, it stops the operations and restarts (if it's able to even do <code>exit</code>). If it cannot reach DB, it doesn't do anything. That is to not run same process twice.</p>
</li>
<li><p>To balance processes across containers, the container which is running less processes can pick up a new one. That info is on DB table to make a decision.</p>
</li>
</ol>
<p>Would you solve differently? Any best practices you recommend?</p>
<p>Thanks</p>

<p>TL;DR - This is a classical monolithic application scaling problem you can easily solve this by redesigning your application to a microservice architecture, since your application functionality is inherently decoupled between components. Once you've done that, it all really boils down to you deploying your application in a natively microservice-friendly fashion and all of your design requirements will be met.</p>
<blockquote>
<p>Edit: You're currently trying to &quot;scale-up&quot; your application in a micro-service system (multiple processes/containers in 1 pod), which defeats the whole purpose of using it. You will have to stick with 1 subprocess &lt;===&gt; 1 pod for the design to really work. Otherwise, you are only introducing immense complications and this is against many design principles of micro services. More details below, if you're interested.</p>
</blockquote>
<p>Let me first summarise all that you've said so we can coherently discuss the design.</p>
<h2>Application Requirements</h2>
<p>As I understand the requirements of your application from all the information you've provided, the following is true:</p>
<ol>
<li>You want your <code>processes</code> to be <code>long-lived</code>.</li>
<li>You have a parent <code>application</code> that spawns these <code>long-lived</code> processes.</li>
<li>These <code>processes</code> need to be started on-demand. (dynamically scaled - and scaled-out; see(7) below for 1 <code>process</code> per <code>container</code>)</li>
<li>If there is no load, your <code>application</code> should spawn just 1 <code>process</code> with <code>sub-process.py</code>.</li>
<li>If a <code>container</code> fails, you would like your <code>application</code> to be able to intelligently switch traffic to a healthy <code>container</code> that is also running your long-lived <code>process</code>.</li>
<li>The <code>application</code> should be able to shared load across all the <code>processes</code>/<code>containers</code> currently running.</li>
<li>A <code>process</code> is tied to user requests since it makes calls to 3rd party <code>API</code> systems in order to perform its function. So, it is favourable to have just one <code>process</code> inside a container for simplicity of design.</li>
</ol>
<h2>Limitations of the Current Design(s)</h2>
<h3>The Current Application Design</h3>
<p>Currently, you have the application setup in a way that:</p>
<ol>
<li>You have one <code>application</code> process that spawns multiple identical <code>sub-process.py</code> processes <em><strong>through the application process</strong></em>.</li>
<li>The <code>application</code> faces the user, and receives requests, and spawns <code>sub-process.py</code> processes as needed and scales well inside one compute unit (container, VM etc.)</li>
<li>These <code>processes</code> then perform their actions, and return the response to the <code>application</code> which return it to the user.</li>
</ol>
<p>Now let's discuss your current approach(es) that you've mentioned above and see what are the challenges that you've described.</p>
<h3>Scaling Design 1 - Simply Scaling Docker Containers</h3>
<p>This means simply creating more containers for your applications. And we know that it doesn't satisfy the requirements because scaling the application to multiple replicas starts all the <code>processes</code> and makes them <code>active</code>. This is not what you want, so there is no relationship between these replicas in different containers (since the sub-processes are tied to <code>application</code> running in each container, not the overall system). This is obviously because <code>application</code>'s in different containers are unaware of each-other (and more importantly, the sub-processes each are spawning).</p>
<p>So, this fails to meet our requirement (3), (4), (5).</p>
<h3>Scaling Design 2 - Use a DB as State Storage</h3>
<p>To try and meet (3), (4) and (5) we introduced a <code>database</code> that is central to our overall system and it can keep state data of different <code>processes</code> in our system and how certain <code>containers</code> can be &quot;bound&quot; to processes and manage them. However, this was also known to have certain limitations as you pointed out (plus my own thoughts):</p>
<ol>
<li>Such solutions are good for <code>short-lived</code> processes.</li>
<li>We have to introduce a database that is high speed and be able to maintain states at a very quick pace with a possibility of <code>race conditions</code>.</li>
<li>We will have to write a lot of house-keeping code on top of our containers for orchestration that will use this <code>database</code> and some known rules (that you defined as last 3 points) to achieve our goal. Especially an orchestration component that will know when to start containers on-demand. This is highly complicated.</li>
<li>Not only do we have to spawn new processes, we also want to be able to handle failures and automated traffic switching. This will require us to implement a &quot;networking&quot; component that will communicate with our orchestrator and detect failed containers and re-route incoming traffic to healthy ones and restarts the failed containers.</li>
<li>We will also require this networking service to be able to distribute incoming traffic load across all the containers currently in our system.</li>
</ol>
<p>This fails to meet our requirements (1) and (7) and most importantly THIS IS REINVENTING THE WHEEL!</p>
<p>LET'S TALK ABOUT KUBERNETES AND WHY IT IS <em>EXACTLY</em> WHAT YOU NEED.</p>
<h2>Proposed Solution</h2>
<p>Now let's see how this entire problem can be re-engineered with minimum effort and we can satisfy all of our requirements.</p>
<h3>The Proposed Application Design</h3>
<p>I propose that you can very simply detach your <code>application</code> from your <code>processes</code>. This is easy enough to do, since your application is accepting user requests and forwarding them to identical pool of workers which are performing their operation by making 3rd party API calls. Inherently, this maps perfectly on micro-services.</p>
<pre><code>user1 =====&gt;      |===&gt; worker1 =&gt; someAPIs
user2 =====&gt;  App |===&gt; worker2 =&gt; someAPIs
user2 =====&gt;      |===&gt; worker3 =&gt; someAPIs
    ...
</code></pre>
<p>We can intelligently leverage this. Note that not only are the elements decoupled, but all the workers are performing an identical set of functions (which can result in different output based on use inputs). Essentially you will replace</p>
<pre class="lang-py prettyprint-override"><code>subprocess.Popen([&quot;python3&quot;, &quot;-u&quot;, &quot;sub-program-1.py&quot;])
</code></pre>
<p>with an API call to a service that can provide a worker for you, on demand:</p>
<pre><code>output = some_api(my_worker_service, user_input)
</code></pre>
<p>This means, your design of the application has been preserved and you've simply placed your processes on different systems. So, the application now looks something like this:</p>
<pre><code>user1 =====&gt;                         |===&gt; worker1 =&gt; someAPIs
user2 =====&gt;  App ==&gt;worker_service  |===&gt; worker2 =&gt; someAPIs
user2 =====&gt;                         |===&gt; worker3 =&gt; someAPIs
    ...
</code></pre>
<p>With this essential component of application redesign in place, let's revisit our issues from previous designs and see if this helps us and how Kubernetes comes into the picture.</p>
<h3>The Proposed Scaling Solution - Enter Kubernetes!</h3>
<p>You were absolutely on the right path when you described usage of a database to maintain the <code>state</code> of our entire system and the orchestration logic being able to retrieve status of current <code>containers</code> in our system and make certain decisions. That's <em>exactly</em> how Kubernetes works!</p>
<p>Let's see how Kubernetes solves our problems now</p>
<ol>
<li><code>processes</code> in Kubernetes can be long lived. So, requirement (1) is met and limitation (1) of our database design is also mitigated.</li>
<li>We introduced a <code>service</code> that will manage all of the worker <code>processes</code> for us. So, requirement (2),satisfied. It will also be able to scale the <code>processes</code> on-demand, so requirement (3) is satisfied. It will also keep a minimum <code>process</code> count of <code>1</code> so we don't spawn unnecessary processes, so requirement (4) is satisfied. It will be intelligent enough to forward traffic only to <code>processes</code> at are <code>healthy</code>. So, requirement (5) is met. It will also load balance traffic across all the <code>processes</code> it governs, so requirement (6) is met. This service will also mitigate limitation (4) and (5) of our second design.</li>
<li>You will be allowed to size your <code>processes</code> as needed, to make sure that you only use the resources needed. So, requirement (7) is met.</li>
<li>It uses a central database called <code>etcd</code>, which stores the state of your entire cluster and keeps it updated at all times and accommodates for race conditions as well (multiple components updating the same information - it simply lets the first one to arrive win and fails the other one, forcing it to retry). We've solved problem (2) from our second design.</li>
<li>It comes with logic to orchestrate our <code>processes</code> out of the box so there is no need to write any code. This mitigates limitation (3) of our second design.</li>
</ol>
<p>So, not only were we able to meet all of our requirements, we were also able to implement the solution we were trying to achieve, without writing any additional code for the orchestration! (You will just have to restructure your program a little and introduce APIs).</p>
<h2>How to Implement This</h2>
<p>Just note that in the k8s literature the smallest unit of computation is referred to as <code>pod</code> which performs a single function. Architecturally, this is identical to your description of <code>sub-process</code>. So, whenever I talk about 'Pods' I simply refer to your <code>sub-processes</code>.</p>
<p>You will take (roughly) the following steps to implement the proposed design.</p>
<ol>
<li>Rewrite some part of your application to decouple <code>application</code> from <code>sub-process.py</code>, introducing an API between them.</li>
<li>Package <code>sub-process.py</code> into a container image.</li>
<li>Deploy a small Kubernetes cluster.</li>
<li>Create a <code>Deployment</code> using your <code>sub-process.py</code> image and set the min repica count to 1 and max to any number you want, say 10, for auto-scaling.</li>
<li>Expose this <code>Deployment</code> by creating a <code>Service</code>. This is the &quot;worker service&quot; I talked about, and your <code>application</code> will &quot;submit&quot; requests to this service. And it will not have to worry about anything other than simply making a request to an API endpoint, everything else is handled by k8s.</li>
<li>Configure your application to make API calls to this <code>Service</code>.</li>
<li>Test your application and watch it scale up and down!</li>
</ol>
<p>Now, the way this will function is as follows:</p>
<ol>
<li>Client makes a request to your <code>application</code>.</li>
<li><code>application</code> forwards it to the <code>Service</code> API endpoint.</li>
<li>The <code>Service</code> receives the API request and forwards it to one of the <code>Pods</code> that are running your <code>sub-process.py</code> functionality. If multiple requests are received the <code>Service</code> will balance the requests across all the pods that are available. If a pod fails, it will be take &quot;away&quot; from the service by K8s so requests don't fail.</li>
<li>The pod will perform your functionality and provide the output.</li>
<li>If all the pods in the <code>Service</code> are reaching saturation, the <code>Deployment</code> auto-scaling will trigger and create more pods for the <code>Service</code> and load sharing will resume again (scale-out). If the resource utilisation then reduces, the <code>Deployment</code> will remove certain pods that are not being used anymore and you will be back to 1 pod (scale-in).</li>
</ol>
<blockquote>
<p>If you want, you can put your <code>frontend</code> application into a <code>Deployment</code> and <code>Service</code> as well which will allow you to have an even friendlier cloud-native micro-service architecture. The user will interact with an API of your <code>front-end</code> which will invoke the <code>Service</code> that is managing your <code>sub-process.py</code> workers which will return results.</p>
</blockquote>
<p>I hope this helps you and you can appreciate how clearly the micro-service architecture fits into the design pattern you have, and how you can very simply adapt to it and scale your application as much as you want! Not only that, expressing your design this way will also allow you to redesign/manage/test different versions by simply managing a set of YAML manifests (text files) that you can use with Version Control as well!</p>
