<p>I've had a 3 node setup in GKE. And one of my pod creation is in CrashLoopBackOff state and it is not recovering. The log suggests the below java.lang.IllegalArgumentException. But the other 2 pods they have no such issue. They are up and running. I'm completely unsure of the issue, can someone help me?</p>
<p>Is the issue, a by-product of <strong>install-plugins</strong> in the YML file?
If yes, why didn't the same problem occur with other pods? Can you please help me with it?</p>
<p><strong>Exception:</strong></p>
<pre><code>    &quot;type&quot;: &quot;server&quot;, &quot;timestamp&quot;: &quot;2022-08-29T19:52:29,743Z&quot;, &quot;level&quot;: &quot;ERROR&quot;, &quot;component&quot;: &quot;o.e.b.ElasticsearchUncaughtExceptionHandler&quot;, &quot;cluster.name&quot;: &quot;dev&quot;, &quot;node.name&quot;: &quot;dev-es-data-hot-1&quot;, &quot;message&quot;: &quot;uncaught exception in thread [main]&quot;, 
&quot;stacktrace&quot;: [&quot;org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown secure setting [dev-es-snapshot-backup-feeb83405c27.json] please check that any required plugins are installed, or check the breaking changes documentation for removed settings&quot;,
&quot;at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.16.3.jar:7.16.3]&quot;,
&quot;at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-7.16.3.jar:7.16.3]&quot;,
&quot;at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.16.3.jar:7.16.3]&quot;,
&quot;at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.16.3.jar:7.16.3]&quot;,
&quot;at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.16.3.jar:7.16.3]&quot;,
&quot;at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-7.16.3.jar:7.16.3]&quot;,
&quot;at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.16.3.jar:7.16.3]&quot;,
&quot;Caused by: java.lang.IllegalArgumentException: unknown secure setting [dev-es-snapshot-backup-feeb83405c27.json] please check that any required plugins are installed, or check the breaking changes documentation for removed settings&quot;,
&quot;at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:561) ~[elasticsearch-7.16.3.jar:7.16.3]&quot;,
uncaught exception in thread [main]
&quot;at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:507) ~[elasticsearch-7.16.3.jar:7.16.3]&quot;,
&quot;at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:477) ~[elasticsearch-7.16.3.jar:7.16.3]&quot;,
&quot;at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:447) ~[elasticsearch-7.16.3.jar:7.16.3]&quot;,
&quot;at org.elasticsearch.common.settings.SettingsModule.&lt;init&gt;(SettingsModule.java:137) ~[elasticsearch-7.16.3.jar:7.16.3]&quot;,
&quot;at org.elasticsearch.node.Node.&lt;init&gt;(Node.java:500) ~[elasticsearch-7.16.3.jar:7.16.3]&quot;,
&quot;at org.elasticsearch.node.Node.&lt;init&gt;(Node.java:309) ~[elasticsearch-7.16.3.jar:7.16.3]&quot;,
&quot;at org.elasticsearch.bootstrap.Bootstrap$5.&lt;init&gt;(Bootstrap.java:234) ~[elasticsearch-7.16.3.jar:7.16.3]&quot;,
&quot;at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.16.3.jar:7.16.3]&quot;,
&quot;at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.16.3.jar:7.16.3]&quot;,
&quot;at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.16.3.jar:7.16.3]&quot;,
&quot;... 6 more&quot;] }
</code></pre>
<p><strong>Here is my YAML config:</strong></p>
<pre><code> - name: data-hot-ingest
   count: 3
   config:
     node.roles: [&quot;data_hot&quot;, &quot;ingest&quot;, &quot;data_content&quot;]
     node.attr.data: hot
     node.store.allow_mmap: false
     xpack.security.authc:
       anonymous:
         username: anon
         roles: monitoring_user
   podTemplate:
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: type
                 operator: In
                 values:
                 - hot
       initContainers:
       - name: install-plugins
         command:
         - sh
         - -c
         - |
           bin/elasticsearch-plugin install --batch repository-gcs
       - name: set-virtual-mem
         command:
         - sysctl
         - -w
         - vm.max_map_count=262144
       containers:
         - name: elasticsearch
           resources:
             requests:
               memory: &quot;64Gi&quot;
               cpu: &quot;30000m&quot;
             limits:
              memory: &quot;65Gi&quot;
              cpu: &quot;30000m&quot;
           env:
             - name: ES_JAVA_OPTS
               value: -Xms32g -Xmx32g
           readinessProbe:
             httpGet:
               scheme: HTTPS
               port: 8080
   volumeClaimTemplates:
   - metadata:
       name: elasticsearch-data
     spec:
       accessModes:
       - ReadWriteOnce
       resources:
         requests:
           storage: 350Gi
       storageClassName: gold
</code></pre>
<p><strong>EDIT:</strong>
We have this secure setting configured, which is linked to a secret in our</p>
<pre><code>  secureSettings:
  - secretName: credentials
</code></pre>

Deepak
<p><strong>[ANSWERING MY OWN QUESTION]</strong></p>
<p>Trying to resolve the below exception:
<code>java.lang.IllegalArgumentException: unknown secure setting [dev-es-snapshot-backup-feeb83405c27.json]</code></p>
<p>I tried comparing the yaml config of the pods, and I found that the pods running successfully do not have a secure setting. But the pod that was crash looping, had the secure setting under <code>elastic-internal-secure-settings</code></p>
<pre><code>- name: elastic-internal-secure-settings
    secret:
      defaultMode: 420
      optional: false
      secretName: dev-es-secure-settings
</code></pre>
<p>And in the operator yaml, I found this:</p>
<pre><code>secureSettings:
  - secretName: credentials
</code></pre>
<p>Just to confirm the behaviour, I upscaled the statefulset, and found the new pod also crash looping with the same error. So someone had tried the secure setting last month, and it crash looped the pod, and didn't reset it back to normal. Once I removed the secure-setting from the operator yaml, the pods started running without any issue.</p>
