Event Store logo

sss https://eventstore.org Menu

Developer Blog

Projections 7: Multiple Streams

  |   Written by: Greg Young   |   Tutorials

Up until this point we have only used two event selection methods for our projections. We have used fromStream(‘stream’) which will select all of the events in a stream and we have used fromAll() which selects all events in the system. There is another quite useful selection that will move us from SEP (Simple Event Processing) to CEP (Complex Event Processing). This is the ability to select between multiple streams.

To select from multiple streams we use fromStreams(['stream1', 'stream2', 'stream3']) what this will do is bring together the events from multiple streams. This can also be called a Join operation. fromStreams(['stream1', 'stream2', 'stream3']) will take the three streams (stream1, stream2, stream3) and produce a single output stream containing events from all three that the projection will be run against.

This operation while seemingly simple is actually quite difficult. Generally the partition point of the system is streams. If you are running in a single node group (not partitioned, either the replicated group or single node) then this projection will have assurances that the events will come in perfect order (even when being processed in real time). But what happens if this is distributed?

You can imagine stream1 lives on one machine and stream2 lives on another machine. This could cause problems with ordering due to situations where the machines are partitioned from each other. Luckily projections allows you to solve this problem with some options that you can set. In particular you can add options

reorderEvents: true,
processingLag: 500 //time in ms

This will tell projections to introduce a delay to allow for the reordering of events from different partitions. This allows a much better handling for fromStreams in a distributed scenario. A buffer of processingLag milliseconds will be introduced to allow for the reordering of events before they are run through the projection.

Subscribe to the Event Store blog

Get the latest news and tutorials when they are released

You might also like

    Data ingestion with Logstash and EventStore

      |   Written by: Riccardo Di Nuzzo   |   Tutorials

    In this post our challenge is to load a CSV file and ingest it using the Event Store HTTP API to be ingested. To be precise, we want to convert this: ```csv Europe,Italy,Clothes,Online,M,12/17/2013,278155219,1/10/2014,1165,109.28,35.84,127311.20,41753.60,85557.60 ``` To this (which is an example of an HTTP POST to the Event Store HTTP API): ```shell [ { "eventId": "fbf4b1a1-b4a3-4dfe-a01f-ec52c34e16e4", "eventType": "InboundDataReceived", "data": { "message": "Europe,Italy,Clothes,Online,M,12/17/2013,278155219,1/10/2014,1165,109.28,35.84,127311.20,41753.60,85557.60" }, "metadata": { "host": "box-1", "path": "/usr/data/sales.csv" } } ] ``` In this example...

    Read article

    Playing with Projections

      |   Written by: Chris Ward   |   Tutorials

    Projections are common concept in event sourcing that allow you to create queries of your events and streams of events. Last year I attended Michel Grootjans's "[playing with projections](https://github.com/michelgrootjans/playing_with_projections)" workshop that taught attendees how to create projections in a variety of programming languages. I decided to convert the workshop exercises to use [Event Store's internal projections](/docs/projections/index.html) engine and show how to use our projections API. The data set is player interactions with a quiz. Visit...

    Read article

    Easier Development Builds on Unix-like Operating Systems

      |   Written by: James Nugent   |   Tutorials

    A common complaint we've heard about Event Store since it was originally open sourced is that it's complex to make development builds locally - and they were slow - and thus the project was hard to contribute to. As part of our recent work on Event Store v3.1.0 (more on this in another post) we decided to rectify this. Event Store is primarily written in C#, and builds on Mono on Linux and MacOS and...

    Read article