title: Distribute an incoming data stream into separate containers within the same network (Twitter API & AWS ECS)
body: <p>I am building a data pipeline on AWS which streams the data from a Twitter's v1.1 <a href="https://developer.twitter.com/en/docs/twitter-api/v1/tweets/filter-realtime/api-reference/post-statuses-filter" rel="nofollow noreferrer">POST statuses/filter</a>. A streamer app sits within an ECS (i.e. docker) container from which it sends the initial POST request. The app then sends the tweets to an AWS Kinesis Firehose stream (so it's possible to send the data to the same KF stream from different places/agents).</p>
  <p>I am using a VPC, hence it's possible to run multiple containers within the same network.</p>
  <p>The question is: <strong>Is it possible to distribute an incoming (Twitter) data stream into multiple containers within the same network (VPC)? And if yes, any hints how to do that?</strong></p>
  <p><strong>UPD.</strong> My pipeline is <code>Twitter API -&gt; [ECS container] streamer app -&gt; S3 -&gt; Lambda (predictions) -&gt; Elasticsearch</code>, and I'm talking about the streamer app part.</p>
  <p>The ultimate goal here is to be able to scale depending on the intensity of the stream. E.g., have one small (memory, CPU) container when the traffic from Twitter is low and spin up more of them when the stream is more intense.</p>