{"action":"created","comment":{"url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/pulls/comments/70731111","id":70731111,"diff_hunk":"@@ -0,0 +1,358 @@\n+---\n+layout: post\n+title: Announcing GenStage\n+author: José Valim\n+category: Announcements\n+excerpt: GenStage is a new Elixir behaviour for exchanging events with back-pressure between Elixir processes. In this blog post we will cover the background that led us to GenStage, some example use cases, and what we are exploring for future releases.\n+---\n+\n+Today we are glad to announce the official release of GenStage. GenStage is a new Elixir behaviour for exchanging events with back-pressure between Elixir processes. In the short-term, we expect GenStage to replace the use cases for GenEvent as well as providing a composable abstraction for consuming data from third party systems.\n+\n+In this blog post we will cover the background that led us to GenStage, some example use cases, and what we are exploring for future releases. If instead you are looking for a quick reference, [check the project source code](https://github.com/elixir-lang/gen_stage) and [access its documentation](https://hexdocs.pm/gen_stage/).\n+\n+## Background\n+\n+One of the original motivations for [creating and designing Elixir was to introduce better abstractions for working with collection](https://www.youtube.com/watch?v=Lqo9-pQuRKE). Not only that, we want to provide developers interested in manipulating collections with a path to take their code from eager, to lazy, to concurrent and then distributed.\n+\n+Let's discuss a simple but actual example: word counting. The idea of word counting is to receive one file and count how many times each word appeards in the document. Using the `Enum` module it could be implemented as follows:\n+\n+```elixir\n+File.read!(\"path/to/some/file\")\n+|> String.split(\"\\n\")\n+|> Enum.flat_map(fn line ->\n+    String.split(line, \" \")\n+   end)\n+|> Enum.reduce(%{}, fn word, acc ->\n+    Map.update(acc, word, 1, & &1 + 1)\n+   end)\n+|> Enum.to_list()\n+```\n+\n+While the solution above works fine and is efficient for small files, it is quite restrictive for large inputs as it loads the whole file into memory.\n+\n+Another issue with the solution above is that the `Enum.flat_map/2` step will build a huge list, with all the words in the file, before we effectively start counting them. Again, for a large document, this means more memory usage and a waste of processing time in building a list that will be traversed right after.\n+\n+Luckily, Elixir provides a solution to this problem (and has provided it for quite some time): streams. One of the advantage of streams is they are lazy, allowing us traverse collections item by item, in this case, line by line, instead of loading the whole data set into memory. Let's rewrite the example above to use streams:\n+\n+```elixir\n+File.stream!(\"path/to/some/file\")\n+|> Stream.flat_map(fn line ->\n+    String.split(line, \" \")\n+   end)\n+|> Enum.reduce(%{}, fn word, acc ->\n+    Map.update(acc, word, 1, & &1 + 1)\n+   end)\n+|> Enum.to_list()\n+```\n+\n+By using `File.stream!` and `Stream.flat_map`, we build a lazy computation that will emit a single line, break that line into words, and emit such words one by one without building huge lists in memory when enumerated. The functions in the [Stream module](http://elixir-lang.org/docs/stable/elixir/Stream.html) just express the computation we want to perform. The computation itself, like traversing the file or breaking into words in `flat_map`, only happen when we call a function in the `Enum` module. We have covered [the foundation for Enum and Streams](http://blog.plataformatec.com.br/2015/05/introducing-reducees/) in another article.\n+\n+The solution above allows us to work with large dataset without loading them all into memory. For large files, it is going to provide much better performance than the eager version. However, the solution above still does not leverage concurrency. For a machine with more than one core, which is the huge majority of machines we have available today, it is a suboptimal solution.\n+\n+That said, how could we leverage concurrency in the example above?\n+\n+During my ElixirConf 2015 keynote, [I discussed one of the most immediate solutions to this problem](http://confreaks.tv/videos/elixirconf2015-keynote) which was to convert parts of your pipeline to separate processes:\n+\n+```elixir\n+File.stream!(\"path/to/some/file\")\n+|> Stream.flat_map(fn line ->\n+    String.split(line, \" \")\n+   end)\n+|> Stream.async()  # NEW!\n+|> Enum.reduce(%{}, fn word, acc ->\n+    Map.update(acc, word, 1, & &1 + 1)\n+   end)\n+|> Enum.to_list()\n+```\n+\n+The idea is that `Stream.async` would run the previous computations in a separate process that would stream its messages to the process that called `Enum.reduce`. Unfortunately, the solution above is less than ideal.\n+\n+First of all, we want to avoid moving data between process as much as possible. Instead, we want to start multiple processes that are performing the same computation in parallel. Not only that, if we are requiring developers to place `Stream.async` manually, it may lead to inneficient and error prone solutions.","path":"_posts/2016-07-14-announcing-genstage.markdown","position":70,"original_position":70,"commit_id":"9aebeeac312be5dc8ff6dc7604b2c84f5b689cd0","original_commit_id":"9aebeeac312be5dc8ff6dc7604b2c84f5b689cd0","user":{"login":"benwilson512","id":247817,"avatar_url":"https://avatars.githubusercontent.com/u/247817?v=3","gravatar_id":"","url":"https://api.github.com/users/benwilson512","html_url":"https://github.com/benwilson512","followers_url":"https://api.github.com/users/benwilson512/followers","following_url":"https://api.github.com/users/benwilson512/following{/other_user}","gists_url":"https://api.github.com/users/benwilson512/gists{/gist_id}","starred_url":"https://api.github.com/users/benwilson512/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benwilson512/subscriptions","organizations_url":"https://api.github.com/users/benwilson512/orgs","repos_url":"https://api.github.com/users/benwilson512/repos","events_url":"https://api.github.com/users/benwilson512/events{/privacy}","received_events_url":"https://api.github.com/users/benwilson512/received_events","type":"User","site_admin":false},"body":"between process => between processes","created_at":"2016-07-14T00:18:39Z","updated_at":"2016-07-14T00:18:39Z","html_url":"https://github.com/elixir-lang/elixir-lang.github.com/pull/801#discussion_r70731111","pull_request_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/pulls/801","_links":{"self":{"href":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/pulls/comments/70731111"},"html":{"href":"https://github.com/elixir-lang/elixir-lang.github.com/pull/801#discussion_r70731111"},"pull_request":{"href":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/pulls/801"}}},"pull_request":{"url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/pulls/801","id":77273746,"html_url":"https://github.com/elixir-lang/elixir-lang.github.com/pull/801","diff_url":"https://github.com/elixir-lang/elixir-lang.github.com/pull/801.diff","patch_url":"https://github.com/elixir-lang/elixir-lang.github.com/pull/801.patch","issue_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/issues/801","number":801,"state":"open","locked":false,"title":"Announcing GenStage","user":{"login":"josevalim","id":9582,"avatar_url":"https://avatars.githubusercontent.com/u/9582?v=3","gravatar_id":"","url":"https://api.github.com/users/josevalim","html_url":"https://github.com/josevalim","followers_url":"https://api.github.com/users/josevalim/followers","following_url":"https://api.github.com/users/josevalim/following{/other_user}","gists_url":"https://api.github.com/users/josevalim/gists{/gist_id}","starred_url":"https://api.github.com/users/josevalim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/josevalim/subscriptions","organizations_url":"https://api.github.com/users/josevalim/orgs","repos_url":"https://api.github.com/users/josevalim/repos","events_url":"https://api.github.com/users/josevalim/events{/privacy}","received_events_url":"https://api.github.com/users/josevalim/received_events","type":"User","site_admin":false},"body":"@fishcakez @alco @lexmag @whatyouhide @ericmj, please review it if/when you have a while.","created_at":"2016-07-13T11:37:40Z","updated_at":"2016-07-14T00:18:39Z","closed_at":null,"merged_at":null,"merge_commit_sha":"722d053f1592515ae4d1fd87c4a1e2ea19371cf5","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/pulls/801/commits","review_comments_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/pulls/801/comments","review_comment_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/pulls/comments{/number}","comments_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/issues/801/comments","statuses_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/statuses/9aebeeac312be5dc8ff6dc7604b2c84f5b689cd0","head":{"label":"elixir-lang:jv-announcing-genstage","ref":"jv-announcing-genstage","sha":"9aebeeac312be5dc8ff6dc7604b2c84f5b689cd0","user":{"login":"elixir-lang","id":1481354,"avatar_url":"https://avatars.githubusercontent.com/u/1481354?v=3","gravatar_id":"","url":"https://api.github.com/users/elixir-lang","html_url":"https://github.com/elixir-lang","followers_url":"https://api.github.com/users/elixir-lang/followers","following_url":"https://api.github.com/users/elixir-lang/following{/other_user}","gists_url":"https://api.github.com/users/elixir-lang/gists{/gist_id}","starred_url":"https://api.github.com/users/elixir-lang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elixir-lang/subscriptions","organizations_url":"https://api.github.com/users/elixir-lang/orgs","repos_url":"https://api.github.com/users/elixir-lang/repos","events_url":"https://api.github.com/users/elixir-lang/events{/privacy}","received_events_url":"https://api.github.com/users/elixir-lang/received_events","type":"Organization","site_admin":false},"repo":{"id":3569633,"name":"elixir-lang.github.com","full_name":"elixir-lang/elixir-lang.github.com","owner":{"login":"elixir-lang","id":1481354,"avatar_url":"https://avatars.githubusercontent.com/u/1481354?v=3","gravatar_id":"","url":"https://api.github.com/users/elixir-lang","html_url":"https://github.com/elixir-lang","followers_url":"https://api.github.com/users/elixir-lang/followers","following_url":"https://api.github.com/users/elixir-lang/following{/other_user}","gists_url":"https://api.github.com/users/elixir-lang/gists{/gist_id}","starred_url":"https://api.github.com/users/elixir-lang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elixir-lang/subscriptions","organizations_url":"https://api.github.com/users/elixir-lang/orgs","repos_url":"https://api.github.com/users/elixir-lang/repos","events_url":"https://api.github.com/users/elixir-lang/events{/privacy}","received_events_url":"https://api.github.com/users/elixir-lang/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/elixir-lang/elixir-lang.github.com","description":"Website for Elixir","fork":false,"url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com","forks_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/forks","keys_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/keys{/key_id}","collaborators_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/teams","hooks_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/hooks","issue_events_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/issues/events{/number}","events_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/events","assignees_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/assignees{/user}","branches_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/branches{/branch}","tags_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/tags","blobs_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/git/refs{/sha}","trees_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/git/trees{/sha}","statuses_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/statuses/{sha}","languages_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/languages","stargazers_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/stargazers","contributors_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/contributors","subscribers_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/subscribers","subscription_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/subscription","commits_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/commits{/sha}","git_commits_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/git/commits{/sha}","comments_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/comments{/number}","issue_comment_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/issues/comments{/number}","contents_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/contents/{+path}","compare_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/compare/{base}...{head}","merges_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/merges","archive_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/downloads","issues_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/issues{/number}","pulls_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/pulls{/number}","milestones_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/milestones{/number}","notifications_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/labels{/name}","releases_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/releases{/id}","deployments_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/deployments","created_at":"2012-02-28T09:19:20Z","updated_at":"2016-07-06T04:22:17Z","pushed_at":"2016-07-13T15:49:46Z","git_url":"git://github.com/elixir-lang/elixir-lang.github.com.git","ssh_url":"git@github.com:elixir-lang/elixir-lang.github.com.git","clone_url":"https://github.com/elixir-lang/elixir-lang.github.com.git","svn_url":"https://github.com/elixir-lang/elixir-lang.github.com","homepage":"elixir-lang.org","size":9375,"stargazers_count":126,"watchers_count":126,"language":"CSS","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":380,"mirror_url":null,"open_issues_count":5,"forks":380,"open_issues":5,"watchers":126,"default_branch":"master"}},"base":{"label":"elixir-lang:master","ref":"master","sha":"7cdc64c23ab09f55e8c977a0fe75bd44fcc25f3a","user":{"login":"elixir-lang","id":1481354,"avatar_url":"https://avatars.githubusercontent.com/u/1481354?v=3","gravatar_id":"","url":"https://api.github.com/users/elixir-lang","html_url":"https://github.com/elixir-lang","followers_url":"https://api.github.com/users/elixir-lang/followers","following_url":"https://api.github.com/users/elixir-lang/following{/other_user}","gists_url":"https://api.github.com/users/elixir-lang/gists{/gist_id}","starred_url":"https://api.github.com/users/elixir-lang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elixir-lang/subscriptions","organizations_url":"https://api.github.com/users/elixir-lang/orgs","repos_url":"https://api.github.com/users/elixir-lang/repos","events_url":"https://api.github.com/users/elixir-lang/events{/privacy}","received_events_url":"https://api.github.com/users/elixir-lang/received_events","type":"Organization","site_admin":false},"repo":{"id":3569633,"name":"elixir-lang.github.com","full_name":"elixir-lang/elixir-lang.github.com","owner":{"login":"elixir-lang","id":1481354,"avatar_url":"https://avatars.githubusercontent.com/u/1481354?v=3","gravatar_id":"","url":"https://api.github.com/users/elixir-lang","html_url":"https://github.com/elixir-lang","followers_url":"https://api.github.com/users/elixir-lang/followers","following_url":"https://api.github.com/users/elixir-lang/following{/other_user}","gists_url":"https://api.github.com/users/elixir-lang/gists{/gist_id}","starred_url":"https://api.github.com/users/elixir-lang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elixir-lang/subscriptions","organizations_url":"https://api.github.com/users/elixir-lang/orgs","repos_url":"https://api.github.com/users/elixir-lang/repos","events_url":"https://api.github.com/users/elixir-lang/events{/privacy}","received_events_url":"https://api.github.com/users/elixir-lang/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/elixir-lang/elixir-lang.github.com","description":"Website for Elixir","fork":false,"url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com","forks_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/forks","keys_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/keys{/key_id}","collaborators_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/teams","hooks_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/hooks","issue_events_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/issues/events{/number}","events_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/events","assignees_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/assignees{/user}","branches_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/branches{/branch}","tags_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/tags","blobs_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/git/refs{/sha}","trees_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/git/trees{/sha}","statuses_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/statuses/{sha}","languages_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/languages","stargazers_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/stargazers","contributors_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/contributors","subscribers_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/subscribers","subscription_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/subscription","commits_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/commits{/sha}","git_commits_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/git/commits{/sha}","comments_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/comments{/number}","issue_comment_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/issues/comments{/number}","contents_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/contents/{+path}","compare_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/compare/{base}...{head}","merges_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/merges","archive_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/downloads","issues_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/issues{/number}","pulls_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/pulls{/number}","milestones_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/milestones{/number}","notifications_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/labels{/name}","releases_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/releases{/id}","deployments_url":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/deployments","created_at":"2012-02-28T09:19:20Z","updated_at":"2016-07-06T04:22:17Z","pushed_at":"2016-07-13T15:49:46Z","git_url":"git://github.com/elixir-lang/elixir-lang.github.com.git","ssh_url":"git@github.com:elixir-lang/elixir-lang.github.com.git","clone_url":"https://github.com/elixir-lang/elixir-lang.github.com.git","svn_url":"https://github.com/elixir-lang/elixir-lang.github.com","homepage":"elixir-lang.org","size":9375,"stargazers_count":126,"watchers_count":126,"language":"CSS","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":380,"mirror_url":null,"open_issues_count":5,"forks":380,"open_issues":5,"watchers":126,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/pulls/801"},"html":{"href":"https://github.com/elixir-lang/elixir-lang.github.com/pull/801"},"issue":{"href":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/issues/801"},"comments":{"href":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/issues/801/comments"},"review_comments":{"href":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/pulls/801/comments"},"review_comment":{"href":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/pulls/801/commits"},"statuses":{"href":"https://api.github.com/repos/elixir-lang/elixir-lang.github.com/statuses/9aebeeac312be5dc8ff6dc7604b2c84f5b689cd0"}}}}
{"actor":{"display_login":"benwilson512"}}