title: real time log processing using apache spark streaming
body: <p>I want to create a system where I can read logs in real time, and use apache spark to process it. I am confused if I should use something like kafka or flume to pass the logs to spark stream or should I pass the logs using sockets. I have gone through a sample program in the spark streaming documentation- <a href="https://spark.apache.org/docs/latest/streaming-programming-guide.html" rel="noreferrer">Spark stream example</a>. But I will be grateful if someone can guide me a better way to pass logs to spark stream. Its kind of a new turf to me.</p>