{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/88453610","pull_request_review_id":9026044,"id":88453610,"diff_hunk":"@@ -133,50 +153,79 @@ case class CreateMap(children: Seq[Expression]) extends Expression {\n     new ArrayBasedMapData(new GenericArrayData(keyArray), new GenericArrayData(valueArray))\n   }\n \n+  private def getAccessors(ctx: CodegenContext, dt: DataType, array: String,\n+      isPrimitive : Boolean, size: Int): (String, String, String) = {\n+    if (!isPrimitive) {\n+      val arrayClass = classOf[GenericArrayData].getName\n+      ctx.addMutableState(\"Object[]\", array, s\"this.$array = null;\")\n+      (s\"new $arrayClass($array)\",\n+        s\"$array = new Object[${size}];\", s\"this.$array = null;\")\n+    } else {\n+      val unsafeArrayClass = classOf[UnsafeArrayData].getName\n+      val javaDataType = ctx.javaType(dt)\n+      ctx.addMutableState(s\"${javaDataType}[]\", array,\n+        s\"this.$array = new ${javaDataType}[${size}];\")\n+      (s\"$unsafeArrayClass.fromPrimitiveArray($array)\", \"\", \"\")\n+    }\n+  }\n+\n   override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {\n-    val arrayClass = classOf[GenericArrayData].getName\n     val mapClass = classOf[ArrayBasedMapData].getName\n     val keyArray = ctx.freshName(\"keyArray\")\n     val valueArray = ctx.freshName(\"valueArray\")\n-    ctx.addMutableState(\"Object[]\", keyArray, s\"this.$keyArray = null;\")\n-    ctx.addMutableState(\"Object[]\", valueArray, s\"this.$valueArray = null;\")\n \n-    val keyData = s\"new $arrayClass($keyArray)\"\n-    val valueData = s\"new $arrayClass($valueArray)\"\n+    val MapType(keyDt, valueDt, _) = dataType\n+    val evalKeys = keys.map(e => e.genCode(ctx))\n+    val isPrimitiveArrayKey = ctx.isPrimitiveType(keyDt)\n+    val isNonNullKey = evalKeys.forall(_.isNull == \"false\")\n+    val evalValues = values.map(e => e.genCode(ctx))\n+    val isPrimitiveArrayValue =\n+      ctx.isPrimitiveType(valueDt) && evalValues.forall(_.isNull == \"false\")","path":"sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala","position":null,"original_position":99,"commit_id":"9745571dab2e3818112f312bfe5f8ed802e51f76","original_commit_id":"22e53565e52560369f43b705a10205ffc0271c96","user":{"login":"kiszk","id":1315079,"avatar_url":"https://avatars.githubusercontent.com/u/1315079?v=3","gravatar_id":"","url":"https://api.github.com/users/kiszk","html_url":"https://github.com/kiszk","followers_url":"https://api.github.com/users/kiszk/followers","following_url":"https://api.github.com/users/kiszk/following{/other_user}","gists_url":"https://api.github.com/users/kiszk/gists{/gist_id}","starred_url":"https://api.github.com/users/kiszk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kiszk/subscriptions","organizations_url":"https://api.github.com/users/kiszk/orgs","repos_url":"https://api.github.com/users/kiszk/repos","events_url":"https://api.github.com/users/kiszk/events{/privacy}","received_events_url":"https://api.github.com/users/kiszk/received_events","type":"User","site_admin":false},"body":"Done","created_at":"2016-11-17T13:45:03Z","updated_at":"2016-11-17T13:45:03Z","html_url":"https://github.com/apache/spark/pull/13909#discussion_r88453610","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/13909","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/88453610"},"html":{"href":"https://github.com/apache/spark/pull/13909#discussion_r88453610"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/13909"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/13909","id":75220939,"html_url":"https://github.com/apache/spark/pull/13909","diff_url":"https://github.com/apache/spark/pull/13909.diff","patch_url":"https://github.com/apache/spark/pull/13909.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/13909","number":13909,"state":"open","locked":false,"title":"[SPARK-16213][SQL] Reduce runtime overhead of a program that creates an primitive array in DataFrame","user":{"login":"kiszk","id":1315079,"avatar_url":"https://avatars.githubusercontent.com/u/1315079?v=3","gravatar_id":"","url":"https://api.github.com/users/kiszk","html_url":"https://github.com/kiszk","followers_url":"https://api.github.com/users/kiszk/followers","following_url":"https://api.github.com/users/kiszk/following{/other_user}","gists_url":"https://api.github.com/users/kiszk/gists{/gist_id}","starred_url":"https://api.github.com/users/kiszk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kiszk/subscriptions","organizations_url":"https://api.github.com/users/kiszk/orgs","repos_url":"https://api.github.com/users/kiszk/repos","events_url":"https://api.github.com/users/kiszk/events{/privacy}","received_events_url":"https://api.github.com/users/kiszk/received_events","type":"User","site_admin":false},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR reduces runtime overhead of a program the creates an primitive array in DataFrame by using the similar approach to #15044. Generated code performs boxing operation in an assignment from InternalRow to an `Object[]` temporary array (at Lines 051 and 061 in the generated code before without this PR). If we know that type of array elements is primitive, we apply the following optimizations:\r\n1. Eliminate a pair of `isNullAt()` and a null assignment\r\n2. Allocate an primitive array instead of `Object[]` (eliminate boxing operations)\r\n3. Call `UnsafeArrayData.fromPrimitiveArray()` to keep a primitive array in a row format instead of doing non-lightweight operations in constructor of `GenericArrayData` \r\nThe PR also performs the same things for `CreateMap`.\r\n\r\nHere are performance results of [DataFrame programs](https://github.com/kiszk/spark/blob/6bf54ec5e227689d69f6db991e9ecbc54e153d0a/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/PrimitiveArrayBenchmark.scala#L83-L112) by up to 16.6x over without this PR.\r\n\r\n```\r\nWithout SPARK-16043\r\nOpenJDK 64-Bit Server VM 1.8.0_91-b14 on Linux 4.4.11-200.fc22.x86_64\r\nIntel Xeon E3-12xx v2 (Ivy Bridge)\r\nRead a primitive array in DataFrame:     Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n------------------------------------------------------------------------------------------------\r\nInt                                           3805 / 4150          0.0      507308.9       1.0X\r\nDouble                                        3593 / 3852          0.0      479056.9       1.1X\r\n\r\nWith SPARK-16043\r\nRead a primitive array in DataFrame:     Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n------------------------------------------------------------------------------------------------\r\nInt                                            229 /  320          0.0       30524.9       1.0X\r\nDouble                                         202 /  223          0.0       26899.2       1.1X\r\n```\r\nNote : #15780 is enabled for these measurements\r\n\r\nAn motivating example\r\n\r\n``` java\r\nval df = sparkContext.parallelize(Seq(0.0d, 1.0d), 1).toDF\r\ndf.selectExpr(\"Array(value + 1.1d, value + 2.2d)\").show\r\n```\r\n\r\nGenerated code without this PR\r\n\r\n``` java\r\n/* 005 */ final class GeneratedIterator extends org.apache.spark.sql.execution.BufferedRowIterator {\r\n/* 006 */   private Object[] references;\r\n/* 007 */   private scala.collection.Iterator[] inputs;\r\n/* 008 */   private scala.collection.Iterator inputadapter_input;\r\n/* 009 */   private UnsafeRow serializefromobject_result;\r\n/* 010 */   private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder serializefromobject_holder;\r\n/* 011 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter serializefromobject_rowWriter;\r\n/* 012 */   private Object[] project_values;\r\n/* 013 */   private UnsafeRow project_result;\r\n/* 014 */   private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder project_holder;\r\n/* 015 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter project_rowWriter;\r\n/* 016 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter project_arrayWriter;\r\n/* 017 */\r\n/* 018 */   public GeneratedIterator(Object[] references) {\r\n/* 019 */     this.references = references;\r\n/* 020 */   }\r\n/* 021 */\r\n/* 022 */   public void init(int index, scala.collection.Iterator[] inputs) {\r\n/* 023 */     partitionIndex = index;\r\n/* 024 */     this.inputs = inputs;\r\n/* 025 */     inputadapter_input = inputs[0];\r\n/* 026 */     serializefromobject_result = new UnsafeRow(1);\r\n/* 027 */     this.serializefromobject_holder = new org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(serializefromobject_result, 0);\r\n/* 028 */     this.serializefromobject_rowWriter = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(serializefromobject_holder, 1);\r\n/* 029 */     this.project_values = null;\r\n/* 030 */     project_result = new UnsafeRow(1);\r\n/* 031 */     this.project_holder = new org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(project_result, 32);\r\n/* 032 */     this.project_rowWriter = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(project_holder, 1);\r\n/* 033 */     this.project_arrayWriter = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter();\r\n/* 034 */\r\n/* 035 */   }\r\n/* 036 */\r\n/* 037 */   protected void processNext() throws java.io.IOException {\r\n/* 038 */     while (inputadapter_input.hasNext()) {\r\n/* 039 */       InternalRow inputadapter_row = (InternalRow) inputadapter_input.next();\r\n/* 040 */       double inputadapter_value = inputadapter_row.getDouble(0);\r\n/* 041 */\r\n/* 042 */       final boolean project_isNull = false;\r\n/* 043 */       this.project_values = new Object[2];\r\n/* 044 */       boolean project_isNull1 = false;\r\n/* 045 */\r\n/* 046 */       double project_value1 = -1.0;\r\n/* 047 */       project_value1 = inputadapter_value + 1.1D;\r\n/* 048 */       if (false) {\r\n/* 049 */         project_values[0] = null;\r\n/* 050 */       } else {\r\n/* 051 */         project_values[0] = project_value1;\r\n/* 052 */       }\r\n/* 053 */\r\n/* 054 */       boolean project_isNull4 = false;\r\n/* 055 */\r\n/* 056 */       double project_value4 = -1.0;\r\n/* 057 */       project_value4 = inputadapter_value + 2.2D;\r\n/* 058 */       if (false) {\r\n/* 059 */         project_values[1] = null;\r\n/* 060 */       } else {\r\n/* 061 */         project_values[1] = project_value4;\r\n/* 062 */       }\r\n/* 063 */\r\n/* 064 */       final ArrayData project_value = new org.apache.spark.sql.catalyst.util.GenericArrayData(project_values);\r\n/* 065 */       this.project_values = null;\r\n/* 066 */       project_holder.reset();\r\n/* 067 */\r\n/* 068 */       project_rowWriter.zeroOutNullBytes();\r\n/* 069 */\r\n/* 070 */       if (project_isNull) {\r\n/* 071 */         project_rowWriter.setNullAt(0);\r\n/* 072 */       } else {\r\n/* 073 */         // Remember the current cursor so that we can calculate how many bytes are\r\n/* 074 */         // written later.\r\n/* 075 */         final int project_tmpCursor = project_holder.cursor;\r\n/* 076 */\r\n/* 077 */         if (project_value instanceof UnsafeArrayData) {\r\n/* 078 */           final int project_sizeInBytes = ((UnsafeArrayData) project_value).getSizeInBytes();\r\n/* 079 */           // grow the global buffer before writing data.\r\n/* 080 */           project_holder.grow(project_sizeInBytes);\r\n/* 081 */           ((UnsafeArrayData) project_value).writeToMemory(project_holder.buffer, project_holder.cursor);\r\n/* 082 */           project_holder.cursor += project_sizeInBytes;\r\n/* 083 */\r\n/* 084 */         } else {\r\n/* 085 */           final int project_numElements = project_value.numElements();\r\n/* 086 */           project_arrayWriter.initialize(project_holder, project_numElements, 8);\r\n/* 087 */\r\n/* 088 */           for (int project_index = 0; project_index < project_numElements; project_index++) {\r\n/* 089 */             if (project_value.isNullAt(project_index)) {\r\n/* 090 */               project_arrayWriter.setNullDouble(project_index);\r\n/* 091 */             } else {\r\n/* 092 */               final double project_element = project_value.getDouble(project_index);\r\n/* 093 */               project_arrayWriter.write(project_index, project_element);\r\n/* 094 */             }\r\n/* 095 */           }\r\n/* 096 */         }\r\n/* 097 */\r\n/* 098 */         project_rowWriter.setOffsetAndSize(0, project_tmpCursor, project_holder.cursor - project_tmpCursor);\r\n/* 099 */       }\r\n/* 100 */       project_result.setTotalSize(project_holder.totalSize());\r\n/* 101 */       append(project_result);\r\n/* 102 */       if (shouldStop()) return;\r\n/* 103 */     }\r\n/* 104 */   }\r\n/* 105 */ }\r\n```\r\n\r\nGenerated code with this PR\r\n\r\n``` java\r\n/* 005 */ final class GeneratedIterator extends org.apache.spark.sql.execution.BufferedRowIterator {\r\n/* 006 */   private Object[] references;\r\n/* 007 */   private scala.collection.Iterator[] inputs;\r\n/* 008 */   private scala.collection.Iterator inputadapter_input;\r\n/* 009 */   private UnsafeRow serializefromobject_result;\r\n/* 010 */   private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder serializefromobject_holder;\r\n/* 011 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter serializefromobject_rowWriter;\r\n/* 012 */   private Object[] project_values;\r\n/* 013 */   private double[] project_values;\r\n/* 014 */   private UnsafeRow project_result;\r\n/* 015 */   private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder project_holder;\r\n/* 016 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter project_rowWriter;\r\n/* 017 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter project_arrayWriter;\r\n/* 018 */\r\n/* 019 */   public GeneratedIterator(Object[] references) {\r\n/* 020 */     this.references = references;\r\n/* 021 */   }\r\n/* 022 */\r\n/* 023 */   public void init(int index, scala.collection.Iterator[] inputs) {\r\n/* 024 */     partitionIndex = index;\r\n/* 025 */     this.inputs = inputs;\r\n/* 026 */     inputadapter_input = inputs[0];\r\n/* 027 */     serializefromobject_result = new UnsafeRow(1);\r\n/* 028 */     this.serializefromobject_holder = new org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(serializefromobject_result, 0);\r\n/* 029 */     this.serializefromobject_rowWriter = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(serializefromobject_holder, 1);\r\n/* 030 */     this.project_values = null;\r\n/* 031 */     this.project_values = new double[2];\r\n/* 032 */     project_result = new UnsafeRow(1);\r\n/* 033 */     this.project_holder = new org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(project_result, 32);\r\n/* 034 */     this.project_rowWriter = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(project_holder, 1);\r\n/* 035 */     this.project_arrayWriter = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter();\r\n/* 036 */\r\n/* 037 */   }\r\n/* 038 */\r\n/* 039 */   protected void processNext() throws java.io.IOException {\r\n/* 040 */     while (inputadapter_input.hasNext()) {\r\n/* 041 */       InternalRow inputadapter_row = (InternalRow) inputadapter_input.next();\r\n/* 042 */       double inputadapter_value = inputadapter_row.getDouble(0);\r\n/* 043 */\r\n/* 044 */       boolean project_isNull1 = false;\r\n/* 045 */\r\n/* 046 */       double project_value1 = -1.0;\r\n/* 047 */       project_value1 = inputadapter_value + 1.1D;\r\n/* 048 */       project_values[0] = project_value1;\r\n/* 049 */\r\n/* 050 */       boolean project_isNull4 = false;\r\n/* 051 */\r\n/* 052 */       double project_value4 = -1.0;\r\n/* 053 */       project_value4 = inputadapter_value + 2.2D;\r\n/* 054 */       project_values[1] = project_value4;\r\n/* 055 */       final ArrayData project_value = org.apache.spark.sql.catalyst.expressions.UnsafeArrayData.fromPrimitiveArray(project_values);\r\n/* 056 */       project_holder.reset();\r\n/* 057 */\r\n/* 058 */       // Remember the current cursor so that we can calculate how many bytes are\r\n/* 059 */       // written later.\r\n/* 060 */       final int project_tmpCursor = project_holder.cursor;\r\n/* 061 */\r\n/* 062 */       if (project_value instanceof UnsafeArrayData) {\r\n/* 063 */         final int project_sizeInBytes = ((UnsafeArrayData) project_value).getSizeInBytes();\r\n/* 064 */         // grow the global buffer before writing data.\r\n/* 065 */         project_holder.grow(project_sizeInBytes);\r\n/* 066 */         ((UnsafeArrayData) project_value).writeToMemory(project_holder.buffer, project_holder.cursor);\r\n/* 067 */         project_holder.cursor += project_sizeInBytes;\r\n/* 068 */\r\n/* 069 */       } else {\r\n/* 070 */         final int project_numElements = project_value.numElements();\r\n/* 071 */         project_arrayWriter.initialize(project_holder, project_numElements, 8);\r\n/* 072 */\r\n/* 073 */         for (int project_index = 0; project_index < project_numElements; project_index++) {\r\n/* 074 */           if (project_value.isNullAt(project_index)) {\r\n/* 075 */             project_arrayWriter.setNullDouble(project_index);\r\n/* 076 */           } else {\r\n/* 077 */             final double project_element = project_value.getDouble(project_index);\r\n/* 078 */             project_arrayWriter.write(project_index, project_element);\r\n/* 079 */           }\r\n/* 080 */         }\r\n/* 081 */       }\r\n/* 082 */\r\n/* 083 */       project_rowWriter.setOffsetAndSize(0, project_tmpCursor, project_holder.cursor - project_tmpCursor);\r\n/* 084 */       project_result.setTotalSize(project_holder.totalSize());\r\n/* 085 */       append(project_result);\r\n/* 086 */       if (shouldStop()) return;\r\n/* 087 */     }\r\n/* 088 */   }\r\n/* 089 */ }\r\n```\r\n## How was this patch tested?\r\n\r\nAdded unit tests into `DataFrameComplexTypeSuite`\r\n","created_at":"2016-06-26T04:07:29Z","updated_at":"2016-11-17T13:45:03Z","closed_at":null,"merged_at":null,"merge_commit_sha":"b085162d25f1251e102f3ae8d06fcecbb5ae046b","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/13909/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/13909/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/13909/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/9745571dab2e3818112f312bfe5f8ed802e51f76","head":{"label":"kiszk:SPARK-16213","ref":"SPARK-16213","sha":"9745571dab2e3818112f312bfe5f8ed802e51f76","user":{"login":"kiszk","id":1315079,"avatar_url":"https://avatars.githubusercontent.com/u/1315079?v=3","gravatar_id":"","url":"https://api.github.com/users/kiszk","html_url":"https://github.com/kiszk","followers_url":"https://api.github.com/users/kiszk/followers","following_url":"https://api.github.com/users/kiszk/following{/other_user}","gists_url":"https://api.github.com/users/kiszk/gists{/gist_id}","starred_url":"https://api.github.com/users/kiszk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kiszk/subscriptions","organizations_url":"https://api.github.com/users/kiszk/orgs","repos_url":"https://api.github.com/users/kiszk/repos","events_url":"https://api.github.com/users/kiszk/events{/privacy}","received_events_url":"https://api.github.com/users/kiszk/received_events","type":"User","site_admin":false},"repo":{"id":45008789,"name":"spark","full_name":"kiszk/spark","owner":{"login":"kiszk","id":1315079,"avatar_url":"https://avatars.githubusercontent.com/u/1315079?v=3","gravatar_id":"","url":"https://api.github.com/users/kiszk","html_url":"https://github.com/kiszk","followers_url":"https://api.github.com/users/kiszk/followers","following_url":"https://api.github.com/users/kiszk/following{/other_user}","gists_url":"https://api.github.com/users/kiszk/gists{/gist_id}","starred_url":"https://api.github.com/users/kiszk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kiszk/subscriptions","organizations_url":"https://api.github.com/users/kiszk/orgs","repos_url":"https://api.github.com/users/kiszk/repos","events_url":"https://api.github.com/users/kiszk/events{/privacy}","received_events_url":"https://api.github.com/users/kiszk/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/kiszk/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/kiszk/spark","forks_url":"https://api.github.com/repos/kiszk/spark/forks","keys_url":"https://api.github.com/repos/kiszk/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kiszk/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kiszk/spark/teams","hooks_url":"https://api.github.com/repos/kiszk/spark/hooks","issue_events_url":"https://api.github.com/repos/kiszk/spark/issues/events{/number}","events_url":"https://api.github.com/repos/kiszk/spark/events","assignees_url":"https://api.github.com/repos/kiszk/spark/assignees{/user}","branches_url":"https://api.github.com/repos/kiszk/spark/branches{/branch}","tags_url":"https://api.github.com/repos/kiszk/spark/tags","blobs_url":"https://api.github.com/repos/kiszk/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kiszk/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kiszk/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/kiszk/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kiszk/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/kiszk/spark/languages","stargazers_url":"https://api.github.com/repos/kiszk/spark/stargazers","contributors_url":"https://api.github.com/repos/kiszk/spark/contributors","subscribers_url":"https://api.github.com/repos/kiszk/spark/subscribers","subscription_url":"https://api.github.com/repos/kiszk/spark/subscription","commits_url":"https://api.github.com/repos/kiszk/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/kiszk/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/kiszk/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/kiszk/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/kiszk/spark/contents/{+path}","compare_url":"https://api.github.com/repos/kiszk/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kiszk/spark/merges","archive_url":"https://api.github.com/repos/kiszk/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kiszk/spark/downloads","issues_url":"https://api.github.com/repos/kiszk/spark/issues{/number}","pulls_url":"https://api.github.com/repos/kiszk/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/kiszk/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/kiszk/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kiszk/spark/labels{/name}","releases_url":"https://api.github.com/repos/kiszk/spark/releases{/id}","deployments_url":"https://api.github.com/repos/kiszk/spark/deployments","created_at":"2015-10-27T01:25:07Z","updated_at":"2016-06-12T04:05:26Z","pushed_at":"2016-11-17T13:44:18Z","git_url":"git://github.com/kiszk/spark.git","ssh_url":"git@github.com:kiszk/spark.git","clone_url":"https://github.com/kiszk/spark.git","svn_url":"https://github.com/kiszk/spark","homepage":null,"size":244944,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"1386fd28daf798bf152606f4da30a36223d75d18","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","deployments_url":"https://api.github.com/repos/apache/spark/deployments","created_at":"2014-02-25T08:00:08Z","updated_at":"2016-11-17T13:05:57Z","pushed_at":"2016-11-17T13:44:20Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":610591,"stargazers_count":10760,"watchers_count":10760,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":10043,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":444,"forks":10043,"open_issues":444,"watchers":10760,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/13909"},"html":{"href":"https://github.com/apache/spark/pull/13909"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/13909"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/13909/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/13909/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/13909/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/9745571dab2e3818112f312bfe5f8ed802e51f76"}}}}
{"actor":{"display_login":"kiszk"}}