<p>I have been trying to switch my application to use Google Cloud logging (aka Stackdriver). I have built a custom Logback appender which uses the Google Java Cloud Logging Library. </p>

<p>When I run my application in the IDE it logs correctly to Google Cloud Logging. When I run it inside a Kubernetes container on GKE, I get an Authentication Error. However in both scenarios I am use the same GCP Service Account. </p>

<p>I have even gone to the length of outputting the Key from the Google credentials object to prove they are the same in each environment. Which they are. </p>

<p>Does anyone have any ideas of what I might be missing? Is there any sort of context related settings that might cause authentication to fail from inside the container? Anyone got any ideas about what to try next?</p>

<hr>

<p><strong>Update:</strong> Here is the stack trace:</p>

<p>java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.google.cloud.logging.LoggingException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Credentials should use fail() instead of throwing exceptions at java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.google.cloud.logging.LoggingException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Credentials should use fail() instead of throwing exceptions at at com.google.cloud.logging.LoggingImpl.flush(LoggingImpl.java:579) at at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:560) at at uk.co.processflows.platform.util.StackdriverAppender.append(StackdriverAppender.kt:135) at at uk.co.processflows.platform.util.StackdriverAppender.append(StackdriverAppender.kt:18) at at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84) at at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51) at at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270) at at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257) at at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421) at at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383) at at ch.qos.logback.classic.Logger.error(Logger.java:538) at at uk.co.processflows.platform.pushnotification.PushNotificationClient.forward(PushNotificationClient.kt:58) at at uk.co.processflows.platform.push.PushMessagingConnectionHandlerImpl.broadcast(PushMessagingConnectionHandlerImpl.kt:257) at at uk.co.processflows.platform.push.PushMessagingConnectionHandlerImpl.broadcastMessageToTenant(PushMessagingConnectionHandlerImpl.kt:153) at at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at at java.base/java.lang.reflect.Method.invoke(Method.java:567) at at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50) at at com.sun.proxy.$Proxy69.broadcastMessageToTenant(Unknown Source) at at uk.co.processflows.platform.workflow.TaskLockControllerImpl$lockTask$2.run(TaskLockControllerImpl.kt:94) at at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at at java.base/java.lang.Thread.run(Thread.java:835) Caused by: java.util.concurrent.ExecutionException: com.google.cloud.logging.LoggingException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Credentials should use fail() instead of throwing exceptions at at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:552) at at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:431) at at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:97) at at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:68) at at com.google.cloud.logging.LoggingImpl.flush(LoggingImpl.java:577) at ... 25 common frames omitted Caused by: com.google.cloud.logging.LoggingException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Credentials should use fail() instead of throwing exceptions at at com.google.cloud.logging.spi.v2.GrpcLoggingRpc$2.apply(GrpcLoggingRpc.java:190) at at com.google.cloud.logging.spi.v2.GrpcLoggingRpc$2.apply(GrpcLoggingRpc.java:184) at at com.google.api.core.ApiFutures$GaxFunctionToGuavaFunction.apply(ApiFutures.java:204) at at com.google.common.util.concurrent.AbstractCatchingFuture$CatchingFuture.doFallback(AbstractCatchingFuture.java:206) at at com.google.common.util.concurrent.AbstractCatchingFuture$CatchingFuture.doFallback(AbstractCatchingFuture.java:194) at at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:107) at at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138) at at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:958) at at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748) at at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95) at at com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77) at at com.google.api.gax.rpc.BatchedFuture.setException(BatchedFuture.java:55) at at com.google.api.gax.rpc.BatchedRequestIssuer.sendResult(BatchedRequestIssuer.java:84) at at com.google.api.gax.rpc.BatchExecutor$1.onFailure(BatchExecutor.java:98) at at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68) at at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1056) at at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138) at at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:958) at at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748) at at com.google.api.gax.retrying.BasicRetryingFuture.handleAttempt(BasicRetryingFuture.java:179) at at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.handle(CallbackChainRetryingFuture.java:135) at at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.run(CallbackChainRetryingFuture.java:117) at at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138) at at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:958) at at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748) at at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95) at at com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77) at at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97) at at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68) at at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1056) at at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138) at at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:958) at at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748) at at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:515) at at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:490) at at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:699) at at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397) at at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459) at at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63) at at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546) at at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467) at at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584) at at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at ... 3 common frames omitted Caused by: com.google.api.gax.rpc.UnauthenticatedException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Credentials should use fail() instead of throwing exceptions at at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:73) at at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72) at at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60) at ... 30 common frames omitted Caused by: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Credentials should use fail() instead of throwing exceptions at at io.grpc.Status.asRuntimeException(Status.java:532) at ... 22 common frames omitted Caused by: java.lang.NoSuchMethodError: io.grpc.MethodDescriptor.getServiceName()Ljava/lang/String; at at io.grpc.auth.GoogleAuthLibraryCallCredentials.serviceUri(GoogleAuthLibraryCallCredentials.java:162) at at io.grpc.auth.GoogleAuthLibraryCallCredentials.applyRequestMetadata(GoogleAuthLibraryCallCredentials.java:103) at at io.grpc.CallCredentials2.applyRequestMetadata(CallCredentials2.java:58) at at io.grpc.internal.CallCredentialsApplyingTransportFactory$CallCredentialsApplyingTransport.newStream(CallCredentialsApplyingTransportFactory.java:107) at at io.grpc.internal.ForwardingConnectionClientTransport.newStream(ForwardingConnectionClientTransport.java:49) at at io.grpc.internal.InternalSubchannel$CallTracingTransport.newStream(InternalSubchannel.java:690) at at io.grpc.internal.ClientCallImpl.start(ClientCallImpl.java:245) at at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1.start(CensusTracingModule.java:392) at at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1.start(CensusStatsModule.java:694) at at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:32) at at com.google.api.gax.grpc.GrpcHeaderInterceptor$1.start(GrpcHeaderInterceptor.java:94) at at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:310) at at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:282) at at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:191) at at com.google.api.gax.grpc.GrpcDirectCallable.futureCall(GrpcDirectCallable.java:58) at at com.google.api.gax.grpc.GrpcExceptionCallable.futureCall(GrpcExceptionCallable.java:64) at at com.google.api.gax.rpc.AttemptCallable.call(AttemptCallable.java:86) at at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:63) at at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:41) at at com.google.api.gax.tracing.TracedBatchingCallable.futureCall(TracedBatchingCallable.java:82) at at com.google.api.gax.rpc.UnaryCallable$1.futureCall(UnaryCallable.java:126) at at com.google.api.gax.rpc.UnaryCallable.futureCall(UnaryCallable.java:87) at at com.google.api.gax.rpc.BatchExecutor.processBatch(BatchExecutor.java:82) at at com.google.api.gax.rpc.BatchExecutor.processBatch(BatchExecutor.java:53) at at com.google.api.gax.batching.ThresholdBatcher.pushCurrentBatch(ThresholdBatcher.java:233) at at com.google.api.gax.batching.ThresholdBatcher$1.run(ThresholdBatcher.java:76) at ... 6 common frames omitted</p>

<hr>

<p><strong>Update 2:</strong></p>

<p>I have just spotted deep inside the caused by hiarchy of the stack trace there is this message.</p>

<p>Caused by: java.lang.NoSuchMethodError: io.grpc.MethodDescriptor.getServiceName()Ljava/lang/String; at at io.grpc.auth.GoogleAuthLibraryCallCredentials.serviceUri(GoogleAuthLibraryCallCredentials.java:162)</p>

<p>That doesn't sound much like an authentication problem.</p>

Charlie M
<p><em>Update</em> I did not see the stack trace when I originally answered this question, and it leads me to be believe your issue is not with authentication at all.  My answer may be relevant to those seeing an authentication issue with GCP APIs in GKE, so I'll leave the original here.</p>

<p><em>Original</em> If you're getting an authentication error, it's likely you're not using the service account key correctly.  You can read this <a href="https://cloud.google.com/kubernetes-engine/docs/tutorials/authenticating-to-cloud-platform" rel="nofollow noreferrer">tutorial</a> to learn about how it should be done. In short, you need to export the service account key as a JSON file and mount it as a secret.  </p>
