{"action":"opened","issue":{"url":"https://api.github.com/repos/databricks/spark-xml/issues/84","repository_url":"https://api.github.com/repos/databricks/spark-xml","labels_url":"https://api.github.com/repos/databricks/spark-xml/issues/84/labels{/name}","comments_url":"https://api.github.com/repos/databricks/spark-xml/issues/84/comments","events_url":"https://api.github.com/repos/databricks/spark-xml/issues/84/events","html_url":"https://github.com/databricks/spark-xml/issues/84","id":133131832,"number":84,"title":"Slow performance for reading XML due to the XML library and not optimized logics.","user":{"login":"HyukjinKwon","id":6477701,"avatar_url":"https://avatars.githubusercontent.com/u/6477701?v=3","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-02-12T01:00:04Z","updated_at":"2016-02-12T01:00:04Z","closed_at":null,"body":"Sameer Farooqui said:\r\n\r\n> I'm reading the wikipedia file like this:\r\n> \r\n> ```scala\r\n> val wiki001DF = sqlContext.read\r\n>     .format(\"com.databricks.spark.xml\")\r\n>     .option(\"rowTag\", \"page\")\r\n>     .load(\"/mnt/wikipedia-readonly/en_wikipedia/enwiki-20160204-pages-articles-multistream.xml\")\r\n> ```\r\n> \r\n> When reading the 54 GB XML file, I noticed that on a Spark cluster with 15 Executors (4 cores on each), it took 40 minutes to read the file (about 839 tasks). Each task is running for about 2 minutes. I found that very slow/long.\r\n> \r\n> I then took some thread dumps on the Executors and noticed that a lot of time is being spent in the xerces XML parsing library:\r\n> \r\n> ```\r\n> com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.scanName(XMLEntityScanner.java:716)\r\n> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEntityReference(XMLDocumentFragmentScannerImpl.java:1844)\r\n> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:3067)\r\n> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)\r\n> com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerlmpl.java:117)\r\n> com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.next(XMLStreamReaderImpl.java:558)\r\n> com.sun.xml.internal.stream.XMLEventReaderImpl.peek(XMLEventReaderImpl.java:276)\r\n> com.databricks.spark.xml.parsers.StaxXmlParserUtils$.readDataFully(StaxXmlParserUtils.scala:29)\r\n> ```\r\n\r\nAlthough this issue sounds general, I think I should create this as a remainder. It looks there are two things to deal with.\r\n\r\n- Better XML library.\r\n- Optimization for parsing logic. \r\n"}}