<p>I'm working on a microservice feature from scratch and have to make some design choices.</p>
<p>My needs are :</p>
<ul>
<li>be able to periodically run a batch job (CronJob) that aggregates data</li>
<li>then send the data set to a service for processing</li>
<li>have resiliency over service failure (pod rescheduled, pod crashing, ..)</li>
<li>the data processing involve long database queries and even Big Queries, so parallelization is required</li>
</ul>
<p>I came up with an architecture in <strong>two parts</strong>, composed of a Kubernetes CronJob doing the initial data aggregation, and publishing messages into a queue. Then this queue is consumed by the service itself, inside a Go routine, so that the load is shared across all instances (pods) of the service, and the resiliency is ensured.</p>
<p>Now I am stuck in a decision dilemma between :</p>
<ul>
<li>decoupling the queue consumer from the service in separate pods</li>
<li>or keep the queue consumer in the same component (in the sense the executable) as the service</li>
</ul>
<p>While <strong>on one side</strong> I find it more logic in terms of architecture to decouple the consumer from the service (separate executables, separate pods), for example, so both can scale independently inside the cluster, <strong>on the other side</strong> the initial wish of introducing a queue in the design was to have resiliency over service failures and the ability to dispatch the load across instances that will consume messages, and that looks <strong>overkill</strong> to me.</p>
<p>I didn't choose the simple option of a CronJob doing API calls to the service because only the service instance receiving the call will do the data processing.</p>
<p>Is it a non-sense to have a service component that also runs a queue consumer, knowing that the service will have multiple instances of itself, in terms of design?</p>
<p>What kind of advantage or disadvantage do I have if I decouple both or keep them coupled?</p>

<p>Easy to maintain/manage if you <strong>HTTP</strong> service (API) running in a different pod and your consumer as different service pod</p>
<p>OOM killing not affecting each other  (if both running in different pods, considering memory leak scenario)</p>
<p>Things might get weird when you scale your system</p>
<p>Consider you have 15-20 queues in each you are adding different types of message for processing</p>
<p>If you will be running each consumer differently in Kubernetes, you might need to deploy 15-20 deployment.</p>
<p>Will you manage 15-20 different repo for each different consumer script? (mostly people follow this trend) If <strong>not</strong> you will be creating the <strong>Mono</strong> repo, now it's on the DevOps side if a single script gets changed in that mono repo only that build should get deployed.</p>
