<p>We are right now busy with a new project where we want to introduce SCDF, but running into one major issue and was wondering if you guys faced a similar issue and how did you solve it.</p>
<p>What we saw, for every stream we create in SCDF, the deployment(on Kubernetes) creates separate instances of the microservices per stream. So if microservice A is used in 3 different streams, at runtime we have 3 instances of microservice A. In our solution, we have a lot of reusable microservices but if SCDF instantiates these microservices per stream we are roughly running almost 400 instances (pods) in production, and if we scale on top of this, we are using an enormous amount of resources. We need to somehow find a way to share pods (instances) across streams.
Did you face this issue? If yes, what was your approach to this?</p>

Soham Dasgupta
<p>There are a couple of ways to reduce the number of pods.</p>
<ol>
<li><p>Use function composition. All of the prepackaged apps are now function based, meaning you can combine functions into a single source, sink or processor app. The SCDF stream definition requires at least a source and sink, but the out of the box functions are designed to be reused in custom apps which may apply the functions to implement intermediate steps as necessary. Bear in mind that composed functions processes data in memory, eliminating the messaging middleware used to stream data between separate pods. This could make your app more susceptible to data loss. There are always trade offs.</p>
</li>
<li><p>Use named destinations: You may share parts of a streaming pipeline using named destinations. This allows you to fan-in or fan-out.  In this example, 3 stream definitions enable 2 sources to feed a shared processor and sink.</p>
<p>source1 &gt; :my-named-destination<br/>
source2 &gt; :my-named-destination<br/>
:my-named-destination &gt; proccessor1 | sink1<br/></p>
</li>
<li><p>The commercial edition of SCDF supports stream definitions using custom components that implement multiple input/outputs. This gives you options similar to the above, where custom routing logic is implemented internally</p>
</li>
<li><p>You can deploy a custom task in place of a stream if appropriate for your use case. The task may incorporate out of the box functions and function composition as needed.</p>
</li>
</ol>
<p>An important consideration when combining components is increased coupling and dependencies among pipeline steps. Simple linear processing creates more pods but is much simpler to implement,deploy,manage, and reason about.</p>
