{"action":"opened","issue":{"url":"https://api.github.com/repos/SparkTC/spark-bench/issues/156","repository_url":"https://api.github.com/repos/SparkTC/spark-bench","labels_url":"https://api.github.com/repos/SparkTC/spark-bench/issues/156/labels{/name}","comments_url":"https://api.github.com/repos/SparkTC/spark-bench/issues/156/comments","events_url":"https://api.github.com/repos/SparkTC/spark-bench/issues/156/events","html_url":"https://github.com/SparkTC/spark-bench/issues/156","id":297895166,"number":156,"title":"Spark can't read ORC files with numeric column names","user":{"login":"ecurtin","id":6586543,"avatar_url":"https://avatars2.githubusercontent.com/u/6586543?v=4","gravatar_id":"","url":"https://api.github.com/users/ecurtin","html_url":"https://github.com/ecurtin","followers_url":"https://api.github.com/users/ecurtin/followers","following_url":"https://api.github.com/users/ecurtin/following{/other_user}","gists_url":"https://api.github.com/users/ecurtin/gists{/gist_id}","starred_url":"https://api.github.com/users/ecurtin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ecurtin/subscriptions","organizations_url":"https://api.github.com/users/ecurtin/orgs","repos_url":"https://api.github.com/users/ecurtin/repos","events_url":"https://api.github.com/users/ecurtin/events{/privacy}","received_events_url":"https://api.github.com/users/ecurtin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-02-16T19:30:27Z","updated_at":"2018-02-16T19:30:27Z","closed_at":null,"author_association":"MEMBER","body":"<!-- \r\n    WELCOME to Spark-Bench! Thanks for being a part of the project!\r\n    Please use Markdown code and syntax highlighting for you config file \r\n    and your stack trace. It makes it much easier to read! \r\n    https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet#code\r\n-->\r\n \r\n### Spark-Bench version (version number, tag, or git commit hash)\r\na1975e52a2167dc3609e615753928ada29a96a8e\r\n\r\n### Details of your cluster setup (Spark version, Standalone/Yarn/Local/Etc)\r\nLocal[*] 2.2.1 built with Hadoop 2.7.5\r\n\r\n### Scala version on your cluster\r\n2.11.8\r\n\r\n### Your exact configuration file (with system details anonymized for security)\r\nN/A, custom scratch test script inside repo. See code below.\r\n\r\n### Relevant stacktrace\r\nThis is from a KMeansDataGen scratch test that I wrote to pinpoint the issue. The code:\r\n```\r\n    val spark = SparkSessionProvider.spark\r\n\r\n    val fileNAme = \"/tmp/stupid-garbage.orc\"\r\n\r\n    Utils.deleteRecursively(new File(fileNAme))\r\n\r\n    val data = KMeansDataGenerator.generateKMeansRDD(\r\n      spark.sparkContext,\r\n      10,\r\n      2,\r\n      2,\r\n      0.6,\r\n      1\r\n    )\r\n\r\n    val schemaString = data.first().indices.map(_.toString).mkString(\" \")\r\n    val fields = schemaString.split(\" \").map(fieldName => StructField(fieldName, DoubleType, nullable = false))\r\n    val schema = StructType(fields)\r\n\r\n    val rowRDD = data.map(arr => Row(arr:_*))\r\n    val df = spark.createDataFrame(rowRDD, schema)\r\n\r\n    df.write.orc(fileNAme)\r\n\r\n    val readingItIn = spark.read.format(\"orc\").load(fileNAme)\r\n\r\n    println(readingItIn.count())\r\n\r\n```\r\n\r\nAnd the stacktrace...\r\n\r\n```\r\nobjc[48187]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/bin/java (0x1061d14c0) and /Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/libinstrument.dylib (0x1081f14e0). One of the two will be used. Which one is undefined.\r\nUsing Spark's default log4j profile: org/apache/spark/log4j-defaults.properties\r\n18/02/16 14:04:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\r\n18/02/16 14:04:31 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id\r\n18/02/16 14:04:31 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id\r\n18/02/16 14:04:31 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id\r\n18/02/16 14:04:31 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap\r\n18/02/16 14:04:31 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition\r\n18/02/16 14:04:31 INFO FileOutputCommitter: File Output Committer Algorithm version is 2\r\n18/02/16 14:04:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 2\r\n18/02/16 14:04:32 INFO FileOutputCommitter: Saved output of task 'attempt_20180216140432_0000_m_000000_0' to file:/tmp/stupid-garbage.orc\r\n\r\n\r\nmismatched input '0' expecting '>'(line 1, pos 7)\r\n\r\n== SQL ==\r\nstruct<0:double,1:double>\r\n-------^^^\r\n\r\norg.apache.spark.sql.catalyst.parser.ParseException: \r\nmismatched input '0' expecting '>'(line 1, pos 7)\r\n\r\n== SQL ==\r\nstruct<0:double,1:double>\r\n-------^^^\r\n\r\n\tat org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:197)\r\n\tat org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:99)\r\n\tat org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parseDataType(ParseDriver.scala:37)\r\n\tat org.apache.spark.sql.hive.orc.OrcFileOperator$$anonfun$readSchema$2.apply(OrcFileOperator.scala:81)\r\n\tat org.apache.spark.sql.hive.orc.OrcFileOperator$$anonfun$readSchema$2.apply(OrcFileOperator.scala:77)\r\n\tat scala.Option.map(Option.scala:146)\r\n\tat org.apache.spark.sql.hive.orc.OrcFileOperator$.readSchema(OrcFileOperator.scala:77)\r\n\tat org.apache.spark.sql.hive.orc.OrcFileFormat.inferSchema(OrcFileFormat.scala:58)\r\n\tat org.apache.spark.sql.execution.datasources.DataSource$$anonfun$7.apply(DataSource.scala:184)\r\n\tat org.apache.spark.sql.execution.datasources.DataSource$$anonfun$7.apply(DataSource.scala:184)\r\n\tat scala.Option.orElse(Option.scala:289)\r\n\tat org.apache.spark.sql.execution.datasources.DataSource.org$apache$spark$sql$execution$datasources$DataSource$$getOrInferFileFormatSchema(DataSource.scala:183)\r\n\tat org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:387)\r\n\tat org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:152)\r\n\tat org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:135)\r\n\tat com.ibm.sparktc.sparkbench.datageneration.KMeansDataGenTest$$anonfun$1.apply$mcV$sp(KMeansDataGenTest.scala:142)\r\n\tat com.ibm.sparktc.sparkbench.datageneration.KMeansDataGenTest$$anonfun$1.apply(KMeansDataGenTest.scala:113)\r\n\tat com.ibm.sparktc.sparkbench.datageneration.KMeansDataGenTest$$anonfun$1.apply(KMeansDataGenTest.scala:113)\r\n\tat org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)\r\n\tat org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)\r\n\tat org.scalatest.Transformer.apply(Transformer.scala:22)\r\n\tat org.scalatest.Transformer.apply(Transformer.scala:20)\r\n\tat org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1682)\r\n\tat org.scalatest.TestSuite$class.withFixture(TestSuite.scala:196)\r\n\tat org.scalatest.FlatSpec.withFixture(FlatSpec.scala:1685)\r\n\tat org.scalatest.FlatSpecLike$class.invokeWithFixture$1(FlatSpecLike.scala:1679)\r\n\tat org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1692)\r\n\tat org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1692)\r\n\tat org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)\r\n\tat org.scalatest.FlatSpecLike$class.runTest(FlatSpecLike.scala:1692)\r\n\tat org.scalatest.FlatSpec.runTest(FlatSpec.scala:1685)\r\n\tat org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1750)\r\n\tat org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1750)\r\n\tat org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:396)\r\n\tat org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384)\r\n\tat scala.collection.immutable.List.foreach(List.scala:381)\r\n\tat org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)\r\n\tat org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:373)\r\n\tat org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:410)\r\n\tat org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384)\r\n\tat scala.collection.immutable.List.foreach(List.scala:381)\r\n\tat org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)\r\n\tat org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:379)\r\n\tat org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)\r\n\tat org.scalatest.FlatSpecLike$class.runTests(FlatSpecLike.scala:1750)\r\n\tat org.scalatest.FlatSpec.runTests(FlatSpec.scala:1685)\r\n\tat org.scalatest.Suite$class.run(Suite.scala:1147)\r\n\tat org.scalatest.FlatSpec.org$scalatest$FlatSpecLike$$super$run(FlatSpec.scala:1685)\r\n\tat org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1795)\r\n\tat org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1795)\r\n\tat org.scalatest.SuperEngine.runImpl(Engine.scala:521)\r\n\tat org.scalatest.FlatSpecLike$class.run(FlatSpecLike.scala:1795)\r\n\tat com.ibm.sparktc.sparkbench.datageneration.KMeansDataGenTest.org$scalatest$BeforeAndAfterAll$$super$run(KMeansDataGenTest.scala:32)\r\n\tat org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:213)\r\n\tat org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)\r\n\tat com.ibm.sparktc.sparkbench.datageneration.KMeansDataGenTest.run(KMeansDataGenTest.scala:32)\r\n\tat org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:45)\r\n\tat org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$1.apply(Runner.scala:1340)\r\n\tat org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$1.apply(Runner.scala:1334)\r\n\tat scala.collection.immutable.List.foreach(List.scala:381)\r\n\tat org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1334)\r\n\tat org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1011)\r\n\tat org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1010)\r\n\tat org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1500)\r\n\tat org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1010)\r\n\tat org.scalatest.tools.Runner$.run(Runner.scala:850)\r\n\tat org.scalatest.tools.Runner.run(Runner.scala)\r\n\tat org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:138)\r\n\tat org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:28)\r\n\r\n\r\nProcess finished with exit code 0\r\n```\r\n### Description of your problem and any other relevant info\r\nIf instead I use string names, everything runs fine.\r\n\r\n```\r\n    val spark = SparkSessionProvider.spark\r\n\r\n    val fileNAme = \"/tmp/stupid-garbage.orc\"\r\n\r\n    Utils.deleteRecursively(new File(fileNAme))\r\n\r\n    val data = KMeansDataGenerator.generateKMeansRDD(\r\n      spark.sparkContext,\r\n      10,\r\n      2,\r\n      2,\r\n      0.6,\r\n      1\r\n    )\r\n\r\n    val schema = StructType( Seq(\r\n      StructField(\"0\", DoubleType, nullable = false),\r\n      StructField(\"1\", DoubleType, nullable = false)\r\n    ))\r\n\r\n    val rowRDD = data.map(arr => Row(arr:_*))\r\n    val df = spark.createDataFrame(rowRDD, schema)\r\n\r\n    df.write.orc(fileNAme)\r\n\r\n    val readingItIn = spark.read.format(\"orc\").load(fileNAme)\r\n\r\n    println(readingItIn.count())\r\n```\r\n\r\nThis is a bug in Spark. Orctools can read this data just fine (note, this is from an earlier run where I used 10 columns instead of 2)\r\n\r\n```\r\n$ java -jar orc-tools-1.4.3-uber.jar data /tmp/stupid-garbage.orc/part-00000-b5c7bd91-96da-4ad4-be3e-344f9927f307.snappy.orc\r\nlog4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).\r\nlog4j:WARN Please initialize the log4j system properly.\r\nlog4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.\r\nProcessing data file /tmp/stupid-garbage.orc/part-00000-b5c7bd91-96da-4ad4-be3e-344f9927f307.snappy.orc [length: 2130]\r\n{\"0\":1.8270485047568874,\"1\":1.4710527183724604,\"2\":-1.5197866190254334,\"3\":-1.7711844582389404,\"4\":0.44956422091644715,\"5\":1.0953964730122485,\"6\":-1.307554251837963,\"7\":-2.2346294442848693,\"8\":-0.3055112209134002,\"9\":2.37794142782504}\r\n{\"0\":0.9347291692457076,\"1\":0.6301415325622199,\"2\":0.359320327320848,\"3\":-0.3041321118604228,\"4\":-0.5541592545207465,\"5\":1.7229785119202934,\"6\":-0.6360722151668361,\"7\":0.4460981019179567,\"8\":0.7512720252005798,\"9\":-0.40247952499145945}\r\n{\"0\":1.7465164231324293,\"1\":1.4889575936956514,\"2\":-0.0503504938463315,\"3\":-0.303724265255192,\"4\":0.9384281822779845,\"5\":1.6625736204947443,\"6\":0.3893484203521068,\"7\":-2.2359278852729876,\"8\":1.8225053639675293,\"9\":-0.5568787286192368}\r\n{\"0\":0.8745330692933655,\"1\":0.5681457474708536,\"2\":0.025755383462769732,\"3\":0.6581895996990914,\"4\":-2.1553239674214884,\"5\":0.8809879693299965,\"6\":-0.3837627235107132,\"7\":0.8362220036118913,\"8\":-9.817864998528103E-4,\"9\":0.8810145566600188}\r\n{\"0\":2.040964654340478,\"1\":-0.27542822800237676,\"2\":1.036641159426876,\"3\":-2.340304518778751,\"4\":-0.9096197049228927,\"5\":0.7003437776000738,\"6\":-0.2051150336764101,\"7\":-1.4684236963216115,\"8\":0.6594461906066581,\"9\":1.7659982932226246}\r\n{\"0\":2.231509625190634,\"1\":0.165553950162822,\"2\":0.4260949945490792,\"3\":-0.7693854454070543,\"4\":-0.6010184304571224,\"5\":-1.5322813419128352,\"6\":0.41645931177868184,\"7\":0.943646914808723,\"8\":1.529473323580286,\"9\":2.3824894963634837}\r\n{\"0\":0.7341645256607737,\"1\":-0.3868184653564506,\"2\":0.23028151334280977,\"3\":-1.9924592519362037,\"4\":-1.133785280324086,\"5\":1.0236465486339563,\"6\":-0.45250035600924654,\"7\":0.8016111499610622,\"8\":0.27133671356291955,\"9\":1.7948142028469825}\r\n{\"0\":0.8479812254247225,\"1\":-0.6755954007742706,\"2\":0.4091125897341032,\"3\":0.5271562961669225,\"4\":-0.3672388406340187,\"5\":0.4620552298752151,\"6\":-0.9780947611105275,\"7\":-0.35273824155931627,\"8\":-1.141817762685812,\"9\":1.1768733350637373}\r\n{\"0\":2.1483351669714503,\"1\":1.278789344023917,\"2\":1.8077303201534083,\"3\":-0.6180897559364122,\"4\":0.951903205934471,\"5\":1.1195476912817746,\"6\":-0.7280869672421849,\"7\":-3.0463262655482337,\"8\":-0.7507643629663004,\"9\":1.527159737224077}\r\n{\"0\":1.1873046360362465,\"1\":0.7992624919449651,\"2\":-0.02916161495356484,\"3\":-0.996049204864252,\"4\":-1.8623112886917839,\"5\":-0.02624561249499935,\"6\":-0.6966430197764303,\"7\":0.1556761234384225,\"8\":-1.6298216824343283,\"9\":-0.020073207052344944}\r\n```\r\n\r\nAnd the meta dump:\r\n\r\n```\r\n$ java -jar orc-tools-1.4.3-uber.jar meta /tmp/stupid-garbage.orc/part-00000-b5c7bd91-96da-4ad4-be3e-344f9927f307.snappy.orc\r\nlog4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).\r\nlog4j:WARN Please initialize the log4j system properly.\r\nlog4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.\r\nProcessing data file /tmp/stupid-garbage.orc/part-00000-b5c7bd91-96da-4ad4-be3e-344f9927f307.snappy.orc [length: 2130]\r\nStructure for /tmp/stupid-garbage.orc/part-00000-b5c7bd91-96da-4ad4-be3e-344f9927f307.snappy.orc\r\nFile Version: 0.12 with HIVE_8732\r\nRows: 10\r\nCompression: SNAPPY\r\nCompression size: 262144\r\nType: struct<0:double,1:double,2:double,3:double,4:double,5:double,6:double,7:double,8:double,9:double>\r\n\r\nStripe Statistics:\r\n  Stripe 1:\r\n    Column 0: count: 10 hasNull: false\r\n    Column 1: count: 10 hasNull: false min: 0.7341645256607737 max: 2.231509625190634 sum: 14.573087000052697\r\n    Column 2: count: 10 hasNull: false min: -0.6755954007742706 max: 1.4889575936956514 sum: 5.064061284099791\r\n    Column 3: count: 10 hasNull: false min: -1.5197866190254334 max: 1.8077303201534083 sum: 2.6956375601645646\r\n    Column 4: count: 10 hasNull: false min: -2.340304518778751 max: 0.6581895996990914 sum: -7.9099831164112135\r\n    Column 5: count: 10 hasNull: false min: -2.1553239674214884 max: 0.951903205934471 sum: -5.243561157843237\r\n    Column 6: count: 10 hasNull: false min: -1.5322813419128352 max: 1.7229785119202934 sum: 7.109002867740469\r\n    Column 7: count: 10 hasNull: false min: -1.307554251837963 max: 0.41645931177868184 sum: -4.582021596199523\r\n    Column 8: count: 10 hasNull: false min: -3.0463262655482337 max: 0.943646914808723 sum: -6.154791239248962\r\n    Column 9: count: 10 hasNull: false min: -1.6298216824343283 max: 1.8225053639675293 sum: 1.2051368014182793\r\n    Column 10: count: 10 hasNull: false min: -0.5568787286192368 max: 2.3824894963634837 sum: 10.926859588542921\r\n\r\nFile Statistics:\r\n  Column 0: count: 10 hasNull: false\r\n  Column 1: count: 10 hasNull: false min: 0.7341645256607737 max: 2.231509625190634 sum: 14.573087000052697\r\n  Column 2: count: 10 hasNull: false min: -0.6755954007742706 max: 1.4889575936956514 sum: 5.064061284099791\r\n  Column 3: count: 10 hasNull: false min: -1.5197866190254334 max: 1.8077303201534083 sum: 2.6956375601645646\r\n  Column 4: count: 10 hasNull: false min: -2.340304518778751 max: 0.6581895996990914 sum: -7.9099831164112135\r\n  Column 5: count: 10 hasNull: false min: -2.1553239674214884 max: 0.951903205934471 sum: -5.243561157843237\r\n  Column 6: count: 10 hasNull: false min: -1.5322813419128352 max: 1.7229785119202934 sum: 7.109002867740469\r\n  Column 7: count: 10 hasNull: false min: -1.307554251837963 max: 0.41645931177868184 sum: -4.582021596199523\r\n  Column 8: count: 10 hasNull: false min: -3.0463262655482337 max: 0.943646914808723 sum: -6.154791239248962\r\n  Column 9: count: 10 hasNull: false min: -1.6298216824343283 max: 1.8225053639675293 sum: 1.2051368014182793\r\n  Column 10: count: 10 hasNull: false min: -0.5568787286192368 max: 2.3824894963634837 sum: 10.926859588542921\r\n\r\nStripes:\r\n  Stripe: offset: 3 data: 830 rows: 10 tail: 130 index: 451\r\n    Stream: column 0 section ROW_INDEX start: 3 length 11\r\n    Stream: column 1 section ROW_INDEX start: 14 length 44\r\n    Stream: column 2 section ROW_INDEX start: 58 length 44\r\n    Stream: column 3 section ROW_INDEX start: 102 length 44\r\n    Stream: column 4 section ROW_INDEX start: 146 length 44\r\n    Stream: column 5 section ROW_INDEX start: 190 length 44\r\n    Stream: column 6 section ROW_INDEX start: 234 length 44\r\n    Stream: column 7 section ROW_INDEX start: 278 length 44\r\n    Stream: column 8 section ROW_INDEX start: 322 length 44\r\n    Stream: column 9 section ROW_INDEX start: 366 length 44\r\n    Stream: column 10 section ROW_INDEX start: 410 length 44\r\n    Stream: column 1 section DATA start: 454 length 83\r\n    Stream: column 2 section DATA start: 537 length 83\r\n    Stream: column 3 section DATA start: 620 length 83\r\n    Stream: column 4 section DATA start: 703 length 83\r\n    Stream: column 5 section DATA start: 786 length 83\r\n    Stream: column 6 section DATA start: 869 length 83\r\n    Stream: column 7 section DATA start: 952 length 83\r\n    Stream: column 8 section DATA start: 1035 length 83\r\n    Stream: column 9 section DATA start: 1118 length 83\r\n    Stream: column 10 section DATA start: 1201 length 83\r\n    Encoding column 0: DIRECT\r\n    Encoding column 1: DIRECT\r\n    Encoding column 2: DIRECT\r\n    Encoding column 3: DIRECT\r\n    Encoding column 4: DIRECT\r\n    Encoding column 5: DIRECT\r\n    Encoding column 6: DIRECT\r\n    Encoding column 7: DIRECT\r\n    Encoding column 8: DIRECT\r\n    Encoding column 9: DIRECT\r\n    Encoding column 10: DIRECT\r\n\r\nFile length: 2130 bytes\r\nPadding length: 0 bytes\r\nPadding ratio: 0%\r\n________________________________________________________________________________________________________________________\r\n\r\n```\r\n\r\nBecause this is a Spark issue, I can't fix the underlying cause. However, this will necessitate a rethink of the strategy for naming columns of RDD's to turn them into Dataframes, like with the KMeans Data Generator which is where I first noticed this issue."}}
{"actor":{"display_login":"ecurtin"}}