{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/12161","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/12161/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/12161/comments","events_url":"https://api.github.com/repos/apache/spark/issues/12161/events","html_url":"https://github.com/apache/spark/pull/12161","id":145849403,"number":12161,"title":"[SPARK-14394][SQL] Generate AggregateHashMap class during TungstenAggregate codegen","user":{"login":"sameeragarwal","id":166454,"avatar_url":"https://avatars.githubusercontent.com/u/166454?v=3","gravatar_id":"","url":"https://api.github.com/users/sameeragarwal","html_url":"https://github.com/sameeragarwal","followers_url":"https://api.github.com/users/sameeragarwal/followers","following_url":"https://api.github.com/users/sameeragarwal/following{/other_user}","gists_url":"https://api.github.com/users/sameeragarwal/gists{/gist_id}","starred_url":"https://api.github.com/users/sameeragarwal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sameeragarwal/subscriptions","organizations_url":"https://api.github.com/users/sameeragarwal/orgs","repos_url":"https://api.github.com/users/sameeragarwal/repos","events_url":"https://api.github.com/users/sameeragarwal/events{/privacy}","received_events_url":"https://api.github.com/users/sameeragarwal/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2016-04-05T01:07:32Z","updated_at":"2016-04-06T09:59:53Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/12161","html_url":"https://github.com/apache/spark/pull/12161","diff_url":"https://github.com/apache/spark/pull/12161.diff","patch_url":"https://github.com/apache/spark/pull/12161.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR adds support for generating the `AggregateHashMap` class in `TungstenAggregate` if the aggregate group by keys/value are of `LongType`. Note that currently this generate aggregate is not actually used.\r\n\r\n## How was this patch tested?\r\n\r\nManually inspected the generated code. This is what the generated map looks like for 2 keys:\r\n\r\n```java\r\n/* 046 */   public class agg_AggregateHashMap {\r\n/* 047 */     private org.apache.spark.sql.execution.vectorized.ColumnarBatch batch;\r\n/* 048 */     private int[] buckets;\r\n/* 049 */     private int numBuckets;\r\n/* 050 */     private int maxSteps;\r\n/* 051 */     private int numRows = 0;\r\n/* 052 */     private org.apache.spark.sql.types.StructType schema =\r\n/* 053 */     new org.apache.spark.sql.types.StructType()\r\n/* 054 */     .add(\"k1\", org.apache.spark.sql.types.DataTypes.LongType)\r\n/* 055 */     .add(\"k2\", org.apache.spark.sql.types.DataTypes.LongType)\r\n/* 056 */     .add(\"sum\", org.apache.spark.sql.types.DataTypes.LongType);\r\n/* 057 */     \r\n/* 058 */     public agg_AggregateHashMap(int capacity, double loadFactor, int maxSteps) {\r\n/* 059 */       assert (capacity > 0 && ((capacity & (capacity - 1)) == 0));\r\n/* 060 */       this.maxSteps = maxSteps;\r\n/* 061 */       numBuckets = (int) (capacity / loadFactor);\r\n/* 062 */       batch = org.apache.spark.sql.execution.vectorized.ColumnarBatch.allocate(schema,\r\n/* 063 */         org.apache.spark.memory.MemoryMode.ON_HEAP, capacity);\r\n/* 064 */       buckets = new int[numBuckets];\r\n/* 065 */       java.util.Arrays.fill(buckets, -1);\r\n/* 066 */     }\r\n/* 067 */     \r\n/* 068 */     public agg_AggregateHashMap() {\r\n/* 069 */       new agg_AggregateHashMap(1 << 16, 0.25, 5);\r\n/* 070 */     }\r\n/* 071 */     \r\n/* 072 */     public org.apache.spark.sql.execution.vectorized.ColumnarBatch.Row findOrInsert(long k1, long k2) {\r\n/* 073 */       int idx = find(k1, k2);\r\n/* 074 */       if (idx != -1 && buckets[idx] == -1) {\r\n/* 075 */         batch.column(0).putLong(numRows, k1);\r\n/* 076 */         batch.column(1).putLong(numRows, k2);\r\n/* 077 */         batch.column(2).putLong(numRows, 0);\r\n/* 078 */         buckets[idx] = numRows++;\r\n/* 079 */       }\r\n/* 080 */       return batch.getRow(buckets[idx]);\r\n/* 081 */     }\r\n/* 082 */     \r\n/* 083 */     private int find(long k1, long k2) {\r\n/* 084 */       long h = hash(k1, k2);\r\n/* 085 */       int step = 0;\r\n/* 086 */       int idx = (int) h & (numBuckets - 1);\r\n/* 087 */       while (step < maxSteps) {\r\n/* 088 */         // Return bucket index if it's either an empty slot or already contains the key\r\n/* 089 */         if (buckets[idx] == -1) {\r\n/* 090 */           return idx;\r\n/* 091 */         } else if (equals(idx, k1, k2)) {\r\n/* 092 */           return idx;\r\n/* 093 */         }\r\n/* 094 */         idx = (idx + 1) & (numBuckets - 1);\r\n/* 095 */         step++;\r\n/* 096 */       }\r\n/* 097 */       // Didn't find it\r\n/* 098 */       return -1;\r\n/* 099 */     }\r\n/* 100 */     \r\n/* 101 */     private boolean equals(int idx, long k1, long k2) {\r\n/* 102 */       return batch.column(0).getLong(buckets[idx]) == k1 && batch.column(1).getLong(buckets[idx]) == k2;\r\n/* 103 */     }\r\n/* 104 */     \r\n/* 105 */     // TODO: Improve this Hash Function\r\n/* 106 */     private long hash(long k1, long k2) {\r\n/* 107 */       return k1 & k2;\r\n/* 108 */     }\r\n/* 109 */     \r\n/* 110 */   }\r\n```"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/206275246","html_url":"https://github.com/apache/spark/pull/12161#issuecomment-206275246","issue_url":"https://api.github.com/repos/apache/spark/issues/12161","id":206275246,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-04-06T09:59:52Z","updated_at":"2016-04-06T09:59:52Z","body":"Merged build finished. Test PASSed."}}