{"action":"created","issue":{"url":"https://api.github.com/repos/dotnet/corefx/issues/6517","repository_url":"https://api.github.com/repos/dotnet/corefx","labels_url":"https://api.github.com/repos/dotnet/corefx/issues/6517/labels{/name}","comments_url":"https://api.github.com/repos/dotnet/corefx/issues/6517/comments","events_url":"https://api.github.com/repos/dotnet/corefx/issues/6517/events","html_url":"https://github.com/dotnet/corefx/issues/6517","id":137293294,"number":6517,"title":"Feature request: Better low-level support for making use of streaming algorithms","user":{"login":"CurtHagenlocher","id":904803,"avatar_url":"https://avatars.githubusercontent.com/u/904803?v=3","gravatar_id":"","url":"https://api.github.com/users/CurtHagenlocher","html_url":"https://github.com/CurtHagenlocher","followers_url":"https://api.github.com/users/CurtHagenlocher/followers","following_url":"https://api.github.com/users/CurtHagenlocher/following{/other_user}","gists_url":"https://api.github.com/users/CurtHagenlocher/gists{/gist_id}","starred_url":"https://api.github.com/users/CurtHagenlocher/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CurtHagenlocher/subscriptions","organizations_url":"https://api.github.com/users/CurtHagenlocher/orgs","repos_url":"https://api.github.com/users/CurtHagenlocher/repos","events_url":"https://api.github.com/users/CurtHagenlocher/events{/privacy}","received_events_url":"https://api.github.com/users/CurtHagenlocher/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/dotnet/corefx/labels/Feature","name":"Feature","color":"5319e7"},{"url":"https://api.github.com/repos/dotnet/corefx/labels/System.IO","name":"System.IO","color":"d4c5f9"},{"url":"https://api.github.com/repos/dotnet/corefx/labels/System.Security","name":"System.Security","color":"d4c5f9"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":23,"created_at":"2016-02-29T15:55:58Z","updated_at":"2016-03-01T21:21:08Z","closed_at":null,"body":"There is a class of algorithms that works against a sequential stream of data. Typically, the algorithm is initialized with some starting state. It's then presented with a sequential stream of identically-typed elements. For each arbitrarily-chosed block of elements, the algorithm may either modify the starting state or produce an incremental block as part of an output stream, or both, depending on the algorithm in question.\r\n\r\nAlgorithms which follow this pattern include compression/decompression, encryption/decryption and hashing.\r\n\r\nIn the BCL, these algorithms are typically modeled as wrappers of System.IO.Stream. For instance, you write normally to a file by using a FileStream. But if you wrap this FileStream with a GZipStream, then writing to the wrapper will cause the file to be compressed. This is incredibly convenient when your use case exactly matches the functionality in the BCL. Where it falls apart, though, is for non-matching use cases.\r\n\r\nFor instance, let's say you want to compress a Stream while reading it. This simply isn't supported directly today. The only way to make use of GZipStream in this scenario is by creating two adapters -- one that reads from the original stream and writes to the GZipStream, and one that intercepts the writes out of the GZipStream and puts a pull interface back on top of it.\r\n\r\nIf the GZip functionality in the BCL were also exposed in a more primitive form -- for instance, something like the [Compressor interface in Hadoop](http://hadoop.apache.org/docs/current/api/org/apache/hadoop/io/compress/Compressor.html), it would be much easier to implement custom scenarios on top of the base functionality. Given a sufficiently general abstraction for the initialize/process next block/finish set of operations, it might even be possible to reduce the amount of per-algorithm code that exists simply to interface with Stream.\r\n\r\nUnfortunately, the CoreCLR footprint appears to be going backwards in this regard. As an example, the HashAlgorithm class previously had methods for TransformBlock and TransformFinalBlock, but these have removed. The only way to make use of HashAlgorithm in a streaming fashion is to go through the pain of exposing the data as a (blocking) Stream, which can be extremely inconvenient."},"comment":{"url":"https://api.github.com/repos/dotnet/corefx/issues/comments/190910674","html_url":"https://github.com/dotnet/corefx/issues/6517#issuecomment-190910674","issue_url":"https://api.github.com/repos/dotnet/corefx/issues/6517","id":190910674,"user":{"login":"psibernetic","id":621605,"avatar_url":"https://avatars.githubusercontent.com/u/621605?v=3","gravatar_id":"","url":"https://api.github.com/users/psibernetic","html_url":"https://github.com/psibernetic","followers_url":"https://api.github.com/users/psibernetic/followers","following_url":"https://api.github.com/users/psibernetic/following{/other_user}","gists_url":"https://api.github.com/users/psibernetic/gists{/gist_id}","starred_url":"https://api.github.com/users/psibernetic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/psibernetic/subscriptions","organizations_url":"https://api.github.com/users/psibernetic/orgs","repos_url":"https://api.github.com/users/psibernetic/repos","events_url":"https://api.github.com/users/psibernetic/events{/privacy}","received_events_url":"https://api.github.com/users/psibernetic/received_events","type":"User","site_admin":false},"created_at":"2016-03-01T21:21:08Z","updated_at":"2016-03-01T21:21:08Z","body":"Aren't you really asking for a low level form IEnumerable here, something that can be written to directly (stream) or consumed with separate positions in the stream? Or is this an argument for an auto-feeding StreamWriterReader that consumes stream A -> Stream B? @ayende @CurtHagenlocher."}}