<p>I'm deploying a Spark Structured Streaming application to Google Kubernetes Engine and while accessing a bucket using <code>gs://</code> URI scheme I'm facing the following exception:</p>
<pre><code>Exception in thread &quot;main&quot; java.lang.NullPointerException: projectId must not be null
    at com.google.cloud.hadoop.repackaged.gcs.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:897)
    at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.createBucket(GoogleCloudStorageImpl.java:437)
    at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorage.createBucket(GoogleCloudStorage.java:88)
    at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem.mkdirsInternal(GoogleCloudStorageFileSystem.java:456)
    at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem.mkdirs(GoogleCloudStorageFileSystem.java:444)
    at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.mkdirs(GoogleHadoopFileSystemBase.java:911)
    at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2275)
    at org.apache.spark.sql.execution.streaming.StreamExecution.&lt;init&gt;(StreamExecution.scala:137)
    at org.apache.spark.sql.execution.streaming.MicroBatchExecution.&lt;init&gt;(MicroBatchExecution.scala:50)
    at org.apache.spark.sql.streaming.StreamingQueryManager.createQuery(StreamingQueryManager.scala:317)
    at org.apache.spark.sql.streaming.StreamingQueryManager.startQuery(StreamingQueryManager.scala:359)
    at org.apache.spark.sql.streaming.DataStreamWriter.startQuery(DataStreamWriter.scala:466)
    at org.apache.spark.sql.streaming.DataStreamWriter.startInternal(DataStreamWriter.scala:456)
    at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:301)
    at meetup.SparkStreamsApp$.delayedEndpoint$meetup$SparkStreamsApp$1(SparkStreamsApp.scala:25)
    at meetup.SparkStreamsApp$delayedInit$body.apply(SparkStreamsApp.scala:7)
</code></pre>
<p>I'm pretty sure it's related to a service account to access and create subdirectories in the bucket that I've been using while <code>spark-submit</code> the Spark app locally using <code>GOOGLE_APPLICATION_CREDENTIALS</code> environment variable and <code>spark.hadoop.google.cloud.auth.service.account.enable=true</code> configuration property.</p>
<p>I'm deploying the Spark application as follows:</p>
<pre><code>./bin/spark-submit \
  --master k8s://$K8S_SERVER \
  --deploy-mode cluster \
  --name $POD_NAME \
  --class meetup.SparkStreamsApp \
  --conf spark.kubernetes.driver.request.cores=400m \
  --conf spark.kubernetes.executor.request.cores=100m \
  --conf spark.kubernetes.container.image=$SPARK_IMAGE \
  --conf spark.kubernetes.driver.pod.name=$POD_NAME \
  --conf spark.kubernetes.namespace=$K8S_NAMESPACE \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
  --conf spark.kubernetes.submission.waitAppCompletion=false \
  --conf spark.hadoop.fs.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem \
  --conf spark.hadoop.google.cloud.auth.service.account.enable=true \
  --verbose \
  local:///opt/spark/jars/meetup.spark-streams-demo-0.1.0.jar $BUCKET_NAME
</code></pre>
<p>How to fix it in a proper Kubernetes / GKE-way?</p>

Jacek Laskowski
<p>Based on your configuration I would suggest you to add the following property <code>fs.gs.project.id</code> as indicated <a href="https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/c0552aa5cfb1fc3fef7a643a1c8957529cbd2909/gcs/INSTALL.md#configure-hadoop" rel="nofollow noreferrer">here</a>. As it is showed up as <code>Required. Google Cloud Project ID with access to configured GCS buckets</code>.</p>
<p>Additional, to this I am agree with the statement from @blackbishop about the secret management.</p>
