{"action":"created","issue":{"url":"https://api.github.com/repos/lucidworks/spark-solr/issues/32","repository_url":"https://api.github.com/repos/lucidworks/spark-solr","labels_url":"https://api.github.com/repos/lucidworks/spark-solr/issues/32/labels{/name}","comments_url":"https://api.github.com/repos/lucidworks/spark-solr/issues/32/comments","events_url":"https://api.github.com/repos/lucidworks/spark-solr/issues/32/events","html_url":"https://github.com/lucidworks/spark-solr/issues/32","id":124985006,"number":32,"title":"Accessing collection fails with NullPointerException","user":{"login":"jqnik","id":3527578,"avatar_url":"https://avatars.githubusercontent.com/u/3527578?v=3","gravatar_id":"","url":"https://api.github.com/users/jqnik","html_url":"https://github.com/jqnik","followers_url":"https://api.github.com/users/jqnik/followers","following_url":"https://api.github.com/users/jqnik/following{/other_user}","gists_url":"https://api.github.com/users/jqnik/gists{/gist_id}","starred_url":"https://api.github.com/users/jqnik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jqnik/subscriptions","organizations_url":"https://api.github.com/users/jqnik/orgs","repos_url":"https://api.github.com/users/jqnik/repos","events_url":"https://api.github.com/users/jqnik/events{/privacy}","received_events_url":"https://api.github.com/users/jqnik/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-01-05T15:08:29Z","updated_at":"2016-04-14T16:43:03Z","closed_at":null,"body":"Hi on the current master branch I am running into the following issue, which I did not experience in the 1.1.2 release.\r\n\r\n```scala\r\n[vagrant@test550-master spark-solr-master]$ PROJECT_HOME=`pwd` && spark-shell --jars $PROJECT_HOME/target/spark-solr-1.2.0-SNAPSHOT-shaded.jar --driver-class-path $PROJECT_HOME/target/spark-solr-1.2.0-SNAPSHOT-shaded.jar\r\n\r\nscala> val poems = sqlContext.load(\"solr\", Map(\"zkHost\" -> \"test550-master:2181/solr\", \"collection\" -> \"poems\"))\r\nwarning: there were 1 deprecation warning(s); re-run with -deprecation for details\r\n2016-01-05 14:56:36,987 [main] INFO  SolrZkClient  - Using default ZkCredentialsProvider\r\n2016-01-05 14:56:37,032 [main] INFO  ConnectionManager  - Waiting for client to connect to ZooKeeper\r\n2016-01-05 14:56:37,054 [zkCallback-2-thread-1] INFO  ConnectionManager  - Watcher shaded.apache.solr.common.cloud.ConnectionManager@7d401432 name:ZooKeeperConnection Watcher:test550-master:2181/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None\r\n2016-01-05 14:56:37,055 [main] INFO  ConnectionManager  - Client is connected to ZooKeeper\r\n2016-01-05 14:56:37,055 [main] INFO  SolrZkClient  - Using default ZkACLProvider\r\n2016-01-05 14:56:37,059 [main] INFO  ZkStateReader  - Updating cluster state from ZooKeeper...\r\n2016-01-05 14:56:37,545 [main] WARN  HiveConf  - HiveConf of name hive.enable.spark.execution.engine does not exist\r\npoems: org.apache.spark.sql.DataFrame = [id: string, text: string, _version_: bigint]\r\n\r\nscala> poems.printSchema()\r\nroot\r\n |-- id: string (nullable = false)\r\n |-- text: string (nullable = false)\r\n |-- _version_: long (nullable = true)\r\n\r\nscala> poems.show()\r\n2016-01-05 14:59:25,856 [main] INFO  SolrRelation  - Building Solr scan using fields=[id, text, _version_]\r\n2016-01-05 14:59:25,857 [main] INFO  SolrRelation  - Constructed SolrQuery: q=*%3A*&rows=1000&sort=id+asc&fl=id%2Ctext%2C_version_&collection=poems\r\n2016-01-05 14:59:29,249 [task-result-getter-0] WARN  TaskSetManager  - Lost task 0.0 in stage 0.0 (TID 0, test550-master.fq.dn): java.lang.RuntimeException: shaded.apache.solr.client.solrj.SolrServerException: java.lang.NullPointerException\r\n\tat com.lucidworks.spark.query.StreamingResultsIterator.hasNext(StreamingResultsIterator.java:66)\r\n\tat scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)\r\n\tat scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)\r\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)\r\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)\r\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)\r\n\tat scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:308)\r\n\tat scala.collection.Iterator$class.foreach(Iterator.scala:727)\r\n\tat scala.collection.AbstractIterator.foreach(Iterator.scala:1157)\r\n\tat scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)\r\n\tat scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)\r\n\tat scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)\r\n\tat scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)\r\n\tat scala.collection.AbstractIterator.to(Iterator.scala:1157)\r\n\tat scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)\r\n\tat scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)\r\n\tat scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)\r\n\tat scala.collection.AbstractIterator.toArray(Iterator.scala:1157)\r\n\tat org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:215)\r\n\tat org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:215)\r\n\tat org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)\r\n\tat org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)\r\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:88)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\nCaused by: shaded.apache.solr.client.solrj.SolrServerException: java.lang.NullPointerException\r\n\tat com.lucidworks.spark.SolrRDD.querySolr(SolrRDD.java:956)\r\n\tat com.lucidworks.spark.query.StreamingResultsIterator.fetchNextPage(StreamingResultsIterator.java:91)\r\n\tat com.lucidworks.spark.query.StreamingResultsIterator.hasNext(StreamingResultsIterator.java:61)\r\n\t... 27 more\r\nCaused by: java.lang.NullPointerException\r\n\tat com.lucidworks.spark.SolrRDD.applyFields(SolrRDD.java:622)\r\n\tat com.lucidworks.spark.SolrRDD.querySolr(SolrRDD.java:906)\r\n\t... 29 more\r\n\r\ntrying with tempTable:\r\n\r\nscala> poems.registerTempTable(\"poems\")\r\n\r\nscala> val p = sqlContext.sql(\"SELECT * from poems\")\r\n2016-01-05 15:02:01,588 [main] INFO  ParseDriver  - Parsing command: SELECT * from poems\r\n2016-01-05 15:02:02,286 [main] INFO  ParseDriver  - Parse Completed\r\np: org.apache.spark.sql.DataFrame = [id: string, text: string, _version_: bigint]\r\n\r\nscala> p.show()\r\n2016-01-05 15:02:06,075 [main] INFO  SolrRelation  - Building Solr scan using fields=[id, text, _version_]\r\n2016-01-05 15:02:06,075 [main] INFO  SolrRelation  - Constructed SolrQuery: q=*%3A*&rows=1000&sort=id+asc&fl=id%2Ctext%2C_version_&collection=poems\r\n2016-01-05 15:02:09,242 [task-result-getter-0] WARN  TaskSetManager  - Lost task 0.0 in stage 0.0 (TID 0, test550-master.fq.dn): java.lang.RuntimeException: shaded.apache.solr.client.solrj.SolrServerException: java.lang.NullPointerException\r\n\tat com.lucidworks.spark.query.StreamingResultsIterator.hasNext(StreamingResultsIterator.java:66)\r\n\tat scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)\r\n\tat scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)\r\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)\r\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)\r\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)\r\n\tat scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:308)\r\n\tat scala.collection.Iterator$class.foreach(Iterator.scala:727)\r\n\tat scala.collection.AbstractIterator.foreach(Iterator.scala:1157)\r\n\tat scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)\r\n\tat scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)\r\n\tat scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)\r\n\tat scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)\r\n\tat scala.collection.AbstractIterator.to(Iterator.scala:1157)\r\n\tat scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)\r\n\tat scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)\r\n\tat scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)\r\n\tat scala.collection.AbstractIterator.toArray(Iterator.scala:1157)\r\n\tat org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:215)\r\n\tat org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:215)\r\n\tat org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)\r\n\tat org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)\r\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:88)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\nCaused by: shaded.apache.solr.client.solrj.SolrServerException: java.lang.NullPointerException\r\n\tat com.lucidworks.spark.SolrRDD.querySolr(SolrRDD.java:956)\r\n\tat com.lucidworks.spark.query.StreamingResultsIterator.fetchNextPage(StreamingResultsIterator.java:91)\r\n\tat com.lucidworks.spark.query.StreamingResultsIterator.hasNext(StreamingResultsIterator.java:61)\r\n\t... 27 more\r\nCaused by: java.lang.NullPointerException\r\n\tat com.lucidworks.spark.SolrRDD.applyFields(SolrRDD.java:622)\r\n\tat com.lucidworks.spark.SolrRDD.querySolr(SolrRDD.java:906)\r\n\t... 29 more\r\n```\r\n\r\nThe same does work with the current 1.1.2 release:\r\n\r\n```scala\r\n[vagrant@test550-master spark-solr-1.1.2]$ PROJECT_HOME=`pwd` && spark-shell --jars $PROJECT_HOME/target/spark-solr-1.1.2-shaded.jar --driver-class-path $PROJECT_HOME/target/spark-solr-1.1.2-SNAPSHOT-shaded.jar\r\n\r\nscala> val poems = sqlContext.load(\"solr\", Map(\"zkHost\" -> \"test550-master:2181/solr\", \"collection\" -> \"poems\"))\r\nwarning: there were 1 deprecation warning(s); re-run with -deprecation for details\r\npoems: org.apache.spark.sql.DataFrame = [_version_: bigint, id: string, text: string]\r\n\r\nscala> poems.show()\r\n16/01/05 15:06:35 INFO SolrRelation: Building Solr scan using fields=[_version_, id, text]\r\n16/01/05 15:06:35 INFO SolrRelation: Constructed SolrQuery: q=*%3A*&rows=1000&sort=id+asc&collection=poems&fl=_version_%2Cid%2Ctext\r\n16/01/05 15:06:35 INFO SparkContext: Starting job: show at <console>:22\r\n16/01/05 15:06:35 INFO DAGScheduler: Got job 1 (show at <console>:22) with 1 output partitions\r\n16/01/05 15:06:35 INFO DAGScheduler: Final stage: ResultStage 1(show at <console>:22)\r\n16/01/05 15:06:35 INFO DAGScheduler: Parents of final stage: List()\r\n16/01/05 15:06:35 INFO DAGScheduler: Missing parents: List()\r\n16/01/05 15:06:35 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[9] at show at <console>:22), which has no missing parents\r\n16/01/05 15:06:35 INFO MemoryStore: ensureFreeSpace(3976) called with curMem=6354, maxMem=556038881\r\n16/01/05 15:06:35 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 3.9 KB, free 530.3 MB)\r\n16/01/05 15:06:35 INFO MemoryStore: ensureFreeSpace(2377) called with curMem=10330, maxMem=556038881\r\n16/01/05 15:06:35 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.3 KB, free 530.3 MB)\r\n16/01/05 15:06:35 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.0.4:46263 (size: 2.3 KB, free: 530.3 MB)\r\n16/01/05 15:06:35 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:861\r\n16/01/05 15:06:35 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[9] at show at <console>:22)\r\n16/01/05 15:06:35 INFO YarnScheduler: Adding task set 1.0 with 1 tasks\r\n16/01/05 15:06:35 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, test550-master.fq.dn, partition 0,PROCESS_LOCAL, 2145 bytes)\r\n16/01/05 15:06:36 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on test550-master.fq.dn:54886 (size: 2.3 KB, free: 530.3 MB)\r\n16/01/05 15:06:38 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 2365 ms on test550-master.fq.dn (1/1)\r\n16/01/05 15:06:38 INFO YarnScheduler: Removed TaskSet 1.0, whose tasks have all completed, from pool\r\n16/01/05 15:06:38 INFO DAGScheduler: ResultStage 1 (show at <console>:22) finished in 2.366 s\r\n16/01/05 15:06:38 INFO DAGScheduler: Job 1 finished: show at <console>:22, took 2.387781 s\r\n+-------------------+---+--------------------+\r\n|          _version_| id|                text|\r\n+-------------------+---+--------------------+\r\n|1520710855906295808|  1|Mary had a little...|\r\n|1520710855977598976|  2|The quick brown f...|\r\n+-------------------+---+--------------------+\r\n\r\n\r\nscala>\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/lucidworks/spark-solr/issues/comments/210042847","html_url":"https://github.com/lucidworks/spark-solr/issues/32#issuecomment-210042847","issue_url":"https://api.github.com/repos/lucidworks/spark-solr/issues/32","id":210042847,"user":{"login":"kiranchitturi","id":4766112,"avatar_url":"https://avatars.githubusercontent.com/u/4766112?v=3","gravatar_id":"","url":"https://api.github.com/users/kiranchitturi","html_url":"https://github.com/kiranchitturi","followers_url":"https://api.github.com/users/kiranchitturi/followers","following_url":"https://api.github.com/users/kiranchitturi/following{/other_user}","gists_url":"https://api.github.com/users/kiranchitturi/gists{/gist_id}","starred_url":"https://api.github.com/users/kiranchitturi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kiranchitturi/subscriptions","organizations_url":"https://api.github.com/users/kiranchitturi/orgs","repos_url":"https://api.github.com/users/kiranchitturi/repos","events_url":"https://api.github.com/users/kiranchitturi/events{/privacy}","received_events_url":"https://api.github.com/users/kiranchitturi/received_events","type":"User","site_admin":false},"created_at":"2016-04-14T16:43:03Z","updated_at":"2016-04-14T16:43:03Z","body":"@jqnik Could you try with master again ?\r\n\r\nWe have made a lot of refactoring and bug fixes in the past 2 months. "}}