title: How to parse the apache log files correctly in spark scala?
body: <p>I am try to read apache log file and trying to put that into dataframe after parsing.</p>
  
  <p>To start with I have taken a sample line of the log and did some exercise using regex to parse the same.However when I am trying to implement the same in the code it is saying "no match found" even though it works correctly in DEMO.Please help.</p>
  
  <p><a href="https://regex101.com/r/yS3lY3/2" rel="nofollow noreferrer">https://regex101.com/r/yS3lY3/2</a></p>
  
  <pre><code>package common
  
  object loganalysis extends App {
  
    import common._
    import org.apache.spark.sql.functions._
    import java.util.regex.Pattern
    import java.util.regex.Matcher
    import scala.util.{Try, Success, Failure}
  
    val spSession = SparkCommonUtils.spSession
    val spContext = SparkCommonUtils.spContext
  
    import spSession.implicits._
  
    val sqlContext = new org.apache.spark.sql.SQLContext(spContext)
  
    val regex = """(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})? (\S+) (\S+) (\[.+?\]) \"(.*?)\" (\d{3}) (\S+) \"(.*?)\" \"(.*?)\""""
  
    val p = Pattern.compile(regex)
  
    val logLine = s"""94.102.63.11 - - [21/Jul/2009:02:48:13 -0700] "GET / HTTP/1.1" 200 18209 "http://acme.com/foo.php" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)""""
  
    val matcher = p.matcher(logLine)
  
    println(matcher.group(1))
  
    }
  </code></pre>
  
  <p>At present it says no match found and errors out.I would like to see each of the parsed strings separately and finally I would like to generalise to apply it on the entire file so that it can handle the exceptions too when no match is found.</p>