<p>I'm using <a href="https://hub.docker.com/r/splunk/splunk/" rel="nofollow noreferrer">this</a> Splunk image on Kubernetes (testing locally with minikube).</p>

<p>After applying the code below I'm facing the following error:</p>

<blockquote>
  <p>ERROR: Couldn't read "/opt/splunk/etc/splunk-launch.conf" -- maybe
  $SPLUNK_HOME or $SPLUNK_ETC is set wrong?</p>
</blockquote>

<p>My Splunk deployment:</p>

<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: splunk
  labels:
    app: splunk-app
    tier: splunk
spec:
  selector:
    matchLabels:
      app: splunk-app
      track: stable
  replicas: 1
  template:
    metadata:
      labels:
        app: splunk-app
        tier: splunk
        track: stable
    spec:
      volumes:
      - name: configmap-inputs
        configMap:
           name: splunk-config
      containers:
      - name: splunk-client
        image: splunk/splunk:latest
        imagePullPolicy: Always
        env:
        - name: SPLUNK_START_ARGS
          value: --accept-license --answer-yes
        - name: SPLUNK_USER
          value: root
        - name: SPLUNK_PASSWORD
          value: changeme
        - name: SPLUNK_FORWARD_SERVER
          value: splunk-receiver:9997
        ports:
        - name: incoming-logs
          containerPort: 514
        volumeMounts:
          - name: configmap-inputs
            mountPath: /opt/splunk/etc/system/local/inputs.conf
            subPath: "inputs.conf"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: splunk-config
data:
  inputs.conf: |
    [monitor:///opt/splunk/var/log/syslog-logs]
    disabled = 0
    index=my-index
</code></pre>

<p>I tried to add also this env variables - with no success:</p>

<pre><code>    - name: SPLUNK_HOME
      value: /opt/splunk
    - name: SPLUNK_ETC
      value: /opt/splunk/etc
</code></pre>

<p>I've tested the image with the following <strong>docker</strong> configuration - <strong>and it ran successfully</strong>:</p>

<pre><code>version: '3.2'
services:
    splunk-forwarder:
      hostname: splunk-client
      image: splunk/splunk:latest
      environment:
        SPLUNK_START_ARGS: --accept-license --answer-yes
        SPLUNK_USER: root
        SPLUNK_PASSWORD: changeme
      ports:
      - "8089:8089"
      - "9997:9997"
</code></pre>

<hr>

<p>Saw <a href="https://answers.splunk.com/answers/728533/getting-error-couldnt-read-optsplunketcsplunk-laun.html" rel="nofollow noreferrer">this</a> on Splunk forum but the answer did not help in my case. </p>

<p>Any ideas?</p>

<hr>

<p>Edit #1: </p>

<p>Minikube version: Upgraded from<code>v0.33.1</code> to <code>v1.2.0</code>.</p>

<p>Full error log:</p>

<pre><code>$kubectl logs -l tier=splunk

splunk_common : Set first run fact -------------------------------------- 0.04s
splunk_common : Set privilege escalation user --------------------------- 0.04s
splunk_common : Set current version fact -------------------------------- 0.04s
splunk_common : Set splunk install fact --------------------------------- 0.04s
splunk_common : Set docker fact ----------------------------------------- 0.04s
Execute pre-setup playbooks --------------------------------------------- 0.04s
splunk_common : Setting upgrade fact ------------------------------------ 0.04s
splunk_common : Set target version fact --------------------------------- 0.04s
Determine captaincy ----------------------------------------------------- 0.04s
ERROR: Couldn't read "/opt/splunk/etc/splunk-launch.conf" -- maybe $SPLUNK_HOME or $SPLUNK_ETC is set wrong?
</code></pre>

<p>Edit #2: Adding config map to the code (was removed from the original question for the sake of brevity). <strong>This is the cause of failure</strong>.</p>

Rotem jackoby
<p>Based on the direction pointed out by @Amit-Kumar-Gupta I'll try also to give a full solution.</p>

<p>So <a href="https://github.com/kubernetes/kubernetes/pull/58720" rel="nofollow noreferrer">this PR</a> change makes it so that containers cannot write to <code>secret</code>, <code>configMap</code>, <code>downwardAPI</code> and projected volumes since the runtime will now <em>mount them as read-only</em>. <br/>
This change is since <code>v1.9.4</code> and can lead to issues for various applications which chown or otherwise manipulate their configs.</p>

<p>When Splunk boots, it registers all the config files in various locations on the filesystem under <code>${SPLUNK_HOME}</code> which is in our case <code>/opt/splunk</code>.<br/>
The error specified in the my question reflect that splunk failed to manipulate all the relevant files in the <code>/opt/splunk/etc</code> directory because of the change in  the mounting mechanism.</p>

<hr>

<p>Now for the solution.</p>

<p>Instead of mounting the configuration file directly inside the <code>/opt/splunk/etc</code> directory we'll use the following setup:</p>

<p>We'll start the docker container with a <code>default.yml</code> file which will be mounted in <code>/tmp/defaults/default.yml</code>.</p>

<p>For that, we'll create the <code>default.yml</code> file with: <br/> <code>docker run splunk/splunk:latest create-defaults &gt; ./default.yml</code></p>

<p>Then, We'll go to the <code>splunk:</code> block and add a <code>config:</code> sub block under it:</p>

<pre><code>splunk:
  conf:
    inputs:
      directory: /opt/splunk/etc/system/local
      content:
          monitor:///opt/splunk/var/log/syslog-logs:
            disabled : 0
            index : syslog-index
    outputs:
      directory: /opt/splunk/etc/system/local
      content:
          tcpout:splunk-indexer:
            server: splunk-indexer:9997
</code></pre>

<p>This setup will generate two files with a <code>.conf</code> postfix (Remember that the sub block start with <code>conf:</code>) which be owned by the correct Splunk user and group.</p>

<p>The <code>inputs:</code> section will produce the a <code>inputs.conf</code> with the following content:</p>

<pre><code>[monitor:///opt/splunk/var/log/syslog-logs]
disabled = 0
index=syslog-index
</code></pre>

<p>In a similar way, the <code>outputs:</code> block will resemble the following:</p>

<pre><code>[tcpout:splunk-receiver]
server=splunk-receiver:9997
</code></pre>

<p>This is instead of the passing an environment variable directly like I did in the origin code:</p>

<pre><code>SPLUNK_FORWARD_SERVER: splunk-receiver:9997
</code></pre>

<p><strong>Now everything is up and running (:</strong></p>

<hr>

<p>Full setup of the <code>forwarder.yaml</code>:</p>

<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: splunk-forwarder
  labels:
    app: splunk-forwarder-app
    tier: splunk
spec:
  selector:
    matchLabels:
      app: splunk-forwarder-app
      track: stable
  replicas: 1
  template:
    metadata:
      labels:
        app: splunk-forwarder-app
        tier: splunk
        track: stable
    spec:
      volumes:
      - name: configmap-forwarder
        configMap:
          name: splunk-forwarder-config

      containers:
      - name: splunk-forwarder
        image: splunk/splunk:latest
        imagePullPolicy : Always
        env:
        - name: SPLUNK_START_ARGS
          value: --accept-license --answer-yes

        - name: SPLUNK_PASSWORD
          valueFrom:
            secretKeyRef:
              name: splunk-secret
              key: password

        volumeMounts:
        - name: configmap-forwarder
          mountPath: /tmp/defaults/default.yml
          subPath: "default.yml"
</code></pre>

<hr>

<p>For further reading:</p>

<p><a href="https://splunk.github.io/docker-splunk/ADVANCED.html" rel="nofollow noreferrer">https://splunk.github.io/docker-splunk/ADVANCED.html</a></p>

<p><a href="https://github.com/splunk/docker-splunk/blob/develop/docs/ADVANCED.md" rel="nofollow noreferrer">https://github.com/splunk/docker-splunk/blob/develop/docs/ADVANCED.md</a></p>

<p><a href="https://www.splunk.com/blog/2018/12/17/deploy-splunk-enterprise-on-kubernetes-splunk-connect-for-kubernetes-and-splunk-insights-for-containers-beta-part-1.html" rel="nofollow noreferrer">https://www.splunk.com/blog/2018/12/17/deploy-splunk-enterprise-on-kubernetes-splunk-connect-for-kubernetes-and-splunk-insights-for-containers-beta-part-1.html</a></p>

<p><a href="https://splunk.github.io/splunk-ansible/ADVANCED.html#inventory-script" rel="nofollow noreferrer">https://splunk.github.io/splunk-ansible/ADVANCED.html#inventory-script</a></p>

<p><a href="https://static.rainfocus.com/splunk/splunkconf18/sess/1521146368312001VwQc/finalPDF/FN1089_DockerizingSplunkatScale_Final_1538666172485001Loc0.pdf" rel="nofollow noreferrer">https://static.rainfocus.com/splunk/splunkconf18/sess/1521146368312001VwQc/finalPDF/FN1089_DockerizingSplunkatScale_Final_1538666172485001Loc0.pdf</a></p>
