Akka-Http + Twitter streaming API <p>I recently got an interest into trying out some of the streaming capabilities in the Scala world. This happened while I was reading up on the Iteratee API in Play 2.</p> <p>However, people seem to think the Iteratee API is close to deprecation, and have recommended me one of the following libs:</p> <ul> <li>scalaz-stream</li> <li>akka-streams, or more specifically, akka-http</li> </ul> <p>I didn't really feel like getting into the scalaz world, so I decided to check out akka-http.</p> <p>Unfortunately, documentation seems to be very sparse on the subject of akka-http at the moment, and I'm having a lot of trouble getting everything to work.</p> <p>As per usual, I chose everyone's favourite source of streaming data to play around with: Twitter.</p> <p>Googling the subject mostly leads to either</p> <ul> <li>Matthias Nehlsens's excellent work on his <a href="http://matthiasnehlsen.com/blog/2013/09/10/birdwatch-explained/" rel="nofollow">BirdWatch project</a>. Unfortunately, he still uses the Iteratees.</li> <li>People using akka-streams with a Twitter client, but I'm not very fond of using those, as I won't be really learning much from those.</li> </ul> <p>Performing a basic GET request with akka-http seems to be structured somewhat like this:</p> <pre><code>object AkkaHttpExample extends App { implicit val system = ActorSystem("akka-http-example") implicit val materializer = ActorMaterializer() import system.dispatcher val connectionFlow: Flow[HttpRequest, HttpResponse, Future[Http.OutgoingConnection]] = Http().outgoingConnection("www.stackoverflow.com", 80) val request: HttpRequest = HttpRequest( HttpMethods.GET, uri = "/" ) val future: Future[HttpResponse] = Source.single(request) .via(connectionFlow) .runWith(Sink.head) val result: HttpResponse = Await.result(future, 5 seconds) } </code></pre> <p>The code above works (though parsing the body of the results is pretty annoying for some reason).</p> <p>When I try to do the same but pointing to the <code>/1.1/statuses/sample.json</code> endpoint (which should emit an example stream), my Future just sits there and times out. While this may seem logical given the streaming nature of the data, this should instead return a 404, because at this point I'm not even doing proper OAuth.</p> <p>For reference, this is my code so far at this point:</p> <pre><code>object AkkaHttpExample extends App { implicit val system = ActorSystem("akka-http-example") implicit val materializer = ActorMaterializer() import system.dispatcher val connectionFlow: Flow[HttpRequest, HttpResponse, Future[Http.OutgoingConnection]] = Http().outgoingConnection("stream.twitter.com", 80) val request: HttpRequest = HttpRequest( HttpMethods.GET, uri = "/1.1/statuses/sample.json" ) val future: Future[HttpResponse] = Source.single(request) .via(connectionFlow) .runWith(Sink.head) val result: HttpResponse = Await.result(future, 50 seconds) } </code></pre> <p>Like I said, I figured that maybe the streaming nature of this might be causing the problem regardless, so I tried to change my code to handle the chunks one by one based on the only examples I could find, as shown here:</p> <pre><code>object AkkaHttpExample extends App { implicit val system = ActorSystem("akka-http-example") implicit val materializer = ActorMaterializer() import system.dispatcher val connectionFlow: Flow[HttpRequest, HttpResponse, Future[Http.OutgoingConnection]] = Http().outgoingConnection("stream.twitter.com", 80) val request: HttpRequest = HttpRequest( HttpMethods.GET, uri = "/1.1/statuses/sample.json" ) Source.single(request) .via(connectionFlow) .map(_.entity.dataBytes) .flatten(FlattenStrategy.concat) .map(_.decodeString("UTF-8")) .runForeach(println _) .onComplete(_ =&gt; system.terminate()) } </code></pre> <p>This causes the application to terminate immediately. Remove the <code>.onComplete</code> clause keeps the ActorSystem and the application running, but nothing seems to actually happen :'(</p> <p>Does anyone have any experience with this? The library has been one huge headache so far. Should I go back to Play + WS + Iteratees?</p>