{"action":"created","issue":{"url":"https://api.github.com/repos/typelevel/frameless/issues/284","repository_url":"https://api.github.com/repos/typelevel/frameless","labels_url":"https://api.github.com/repos/typelevel/frameless/issues/284/labels{/name}","comments_url":"https://api.github.com/repos/typelevel/frameless/issues/284/comments","events_url":"https://api.github.com/repos/typelevel/frameless/issues/284/events","html_url":"https://github.com/typelevel/frameless/issues/284","id":315029872,"number":284,"title":"Assertion in UnsafeRow \"sizeInBytes should be a multiple of 8\"","user":{"login":"mfelsche","id":1152667,"avatar_url":"https://avatars0.githubusercontent.com/u/1152667?v=4","gravatar_id":"","url":"https://api.github.com/users/mfelsche","html_url":"https://github.com/mfelsche","followers_url":"https://api.github.com/users/mfelsche/followers","following_url":"https://api.github.com/users/mfelsche/following{/other_user}","gists_url":"https://api.github.com/users/mfelsche/gists{/gist_id}","starred_url":"https://api.github.com/users/mfelsche/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mfelsche/subscriptions","organizations_url":"https://api.github.com/users/mfelsche/orgs","repos_url":"https://api.github.com/users/mfelsche/repos","events_url":"https://api.github.com/users/mfelsche/events{/privacy}","received_events_url":"https://api.github.com/users/mfelsche/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2018-04-17T12:06:20Z","updated_at":"2018-04-17T21:03:27Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"We just tried to upgrade our pipeline to Spark 2.3 and thus frameless 0.6.0 and hit some nasty assertions in sparks UnsafeRow. We have assertions enabled in our codebase. We tried to reproduce the issue in the frameless repo, but found that `-ea` was not provided for the tests.\r\n\r\nHere is the PR introducing them in Spark 2.3: https://github.com/apache/spark/pull/18503/files\r\n\r\nAs it seems, during codegen, spark is getting a serialized row for a case class (using the RecordEncoder) which only contains not nullable fields, so the nullable bitset is completely `0` and the values are directly in the `values` section elements instead of pointing to the variable length section at the end. But the generated code seems to expect the struct, representing the serialized case class (or tuple) in the first field of the given row. So the `UnsafeRow.getStruct` method used and expects a field pointing to the variable length section at field ordinal `0` and thus interprets the 64 bits at this location (which are the actual long value) as a pointer with offset and size each 32 bits).\r\n\r\nI don't exactly know where the error is. If the `RecordEncoder`, which seems to be at play here is serializing/deserializing in a wrong way, or if the spark code generation makes different assumptions. I don't know the relation between the two well enough. Any clarification from your side would be much appreciated.\r\n\r\nWe used frameless 0.5.0 and spark 2.2 with the same datasets and operations on them and had correct results. And `sizeInBytes` never had a corrupt value as we see with 0.6.0 and spark 2.2.\r\n\r\n\r\nHere is a snippet of some generated code for our specific case class with the `If` construct below leading to the faulty access to the `UnsafeRow`:\r\n```java\r\n\r\n      public java.lang.Object generate(Object[] references) {\r\n        return new SpecificSafeProjection(references);\r\n      }\r\n\r\n      class SpecificSafeProjection extends org.apache.spark.sql.catalyst.expressions.codegen.BaseProjection {\r\n\r\n        private Object[] references;\r\n        private InternalRow mutableRow;\r\n        private boolean resultIsNull;\r\nprivate long argValue;\r\nprivate int argValue1;\r\nprivate long argValue2;\r\nprivate long argValue3;\r\nprivate boolean globalIsNull;\r\n\r\n        public SpecificSafeProjection(Object[] references) {\r\n          this.references = references;\r\n          mutableRow = (InternalRow) references[references.length - 1];\r\n          \r\n\r\n        }\r\n\r\n        public void initialize(int partitionIndex) {\r\n          \r\n        }\r\n\r\n        public java.lang.Object apply(java.lang.Object _i) {\r\n          InternalRow i = (InternalRow) _i;\r\n          \r\ncom.dcmn.db.BrandTrafficClassification value9 = If(i);\r\n            if (globalIsNull) {\r\n              mutableRow.setNullAt(0);\r\n            } else {\r\n              \r\n              mutableRow.update(0, value9);\r\n            }\r\n          \r\n          return mutableRow;\r\n        }\r\n\r\n        \r\nprivate com.dcmn.db.BrandTrafficClassification If(InternalRow i) {\r\n  InternalRow value2 = i.getStruct(0, 4);\r\nboolean isNull = false;\r\ncom.dcmn.db.BrandTrafficClassification value = null;\r\nif (!false && false) {\r\n  \r\nfinal com.dcmn.db.BrandTrafficClassification value3 = null;\r\n  isNull = true;\r\n  value = value3;\r\n} else {\r\n  \r\nresultIsNull = false;\r\n          if (!resultIsNull) {\r\n            \r\nboolean isNull5 = i.isNullAt(0);\r\nlong value5 = isNull5 ? -1L : (i.getLong(0));\r\n            resultIsNull = isNull5;\r\n            argValue = value5;\r\n          }\r\n        \r\n          if (!resultIsNull) {\r\n            \r\nint value6 = i.getInt(1);\r\n            \r\n            argValue1 = value6;\r\n          }\r\n        \r\n          if (!resultIsNull) {\r\n            \r\nlong value7 = i.getLong(2);\r\n            \r\n            argValue2 = value7;\r\n          }\r\n        \r\n          if (!resultIsNull) {\r\n            \r\nlong value8 = i.getLong(3);\r\n            \r\n            argValue3 = value8;\r\n          }\r\n        \r\n      \r\n      final com.dcmn.db.BrandTrafficClassification value4 = resultIsNull ? null : new com.dcmn.db.BrandTrafficClassification(argValue, argValue1, argValue2, argValue3);\r\n  isNull = resultIsNull;\r\n  value = value4;\r\n}\r\n  globalIsNull = isNull;\r\n  return value;\r\n}\r\n           \r\n      }\r\n    \r\n```\r\n\r\nHere is an exampe stacktrace leading to the assertion (with some generated code in between which is equal to the code listing above):\r\n\r\n```\r\n[info]   java.lang.AssertionError: sizeInBytes (13500) should be a multiple of 8\r\n[info]   at org.apache.spark.sql.catalyst.expressions.UnsafeRow.pointTo(UnsafeRow.java:166)\r\n[info]   at org.apache.spark.sql.catalyst.expressions.UnsafeRow.getStruct(UnsafeRow.java:462)\r\n[info]   at org.apache.spark.sql.catalyst.expressions.UnsafeRow.getStruct(UnsafeRow.java:62)\r\n[info]   at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificSafeProjection.If$(Unknown Source)\r\n[info]   at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificSafeProjection.apply(Unknown Source)\r\n[info]   at org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$collectFromPlan$1.apply(Dataset.scala:3275)\r\n[info]   at org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$collectFromPlan$1.apply(Dataset.scala:3272)\r\n[info]   at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\r\n[info]   at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\r\n[info]   at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)\r\n[info]   at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)\r\n[info]   at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)\r\n[info]   at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)\r\n[info]   at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collectFromPlan(Dataset.scala:3272)\r\n[info]   at org.apache.spark.sql.Dataset$$anonfun$collect$1.apply(Dataset.scala:2722)\r\n[info]   at org.apache.spark.sql.Dataset$$anonfun$collect$1.apply(Dataset.scala:2722)\r\n[info]   at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3253)\r\n[info]   at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)\r\n[info]   at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3252)\r\n[info]   at org.apache.spark.sql.Dataset.collect(Dataset.scala:2722)\r\n[info]   at frameless.TypedDataset$$anonfun$collect$1.apply(TypedDataset.scala:291)\r\n[info]   at frameless.TypedDataset$$anonfun$collect$1.apply(TypedDataset.scala:291)\r\n[info]   at frameless.Job$$anon$4.run(Job.scala:38)\r\n\r\n```\r\n\r\nI hope this information helps."},"comment":{"url":"https://api.github.com/repos/typelevel/frameless/issues/comments/382145204","html_url":"https://github.com/typelevel/frameless/issues/284#issuecomment-382145204","issue_url":"https://api.github.com/repos/typelevel/frameless/issues/284","id":382145204,"user":{"login":"imarios","id":6893839,"avatar_url":"https://avatars0.githubusercontent.com/u/6893839?v=4","gravatar_id":"","url":"https://api.github.com/users/imarios","html_url":"https://github.com/imarios","followers_url":"https://api.github.com/users/imarios/followers","following_url":"https://api.github.com/users/imarios/following{/other_user}","gists_url":"https://api.github.com/users/imarios/gists{/gist_id}","starred_url":"https://api.github.com/users/imarios/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/imarios/subscriptions","organizations_url":"https://api.github.com/users/imarios/orgs","repos_url":"https://api.github.com/users/imarios/repos","events_url":"https://api.github.com/users/imarios/events{/privacy}","received_events_url":"https://api.github.com/users/imarios/received_events","type":"User","site_admin":false},"created_at":"2018-04-17T21:03:27Z","updated_at":"2018-04-17T21:03:27Z","author_association":"COLLABORATOR","body":"@mfelsche makes sense why it shows up only with 2.3.0. @snadorp do you want to open a PR for adding this extra flag?"}}
{"actor":{"display_login":"imarios"}}