<p>I have an Airflow application running in kubernetes along with Vault. I am already using Vault to manage certain Airflow Connections and it's working fine (which proves the connectivity with Vault is working fine).</p>
<p>I am now trying to configure Airflow's configs <code>core.sql_alchemy_conn</code>, <code>celery.result_backend</code> and <code>celery.broker_url</code> through Vault as mentioned <a href="https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html?highlight=_secret#envvar-AIRFLOW__-SECTION-__-KEY-_SECRET" rel="nofollow noreferrer">here</a>.</p>
<p>In my <code>values.yaml</code> I have</p>
<pre class="lang-yaml prettyprint-override"><code>secrets_backend_kwargs:
  auth_type: 'kubernetes'
  kubernetes_role: 'role-id'
  connections_path: 'applications/secrets/airflow/connections/'
  config_path: 'applications/secrets/airflow/config/'

config:
  core:
    sql_alchemy_conn_secret: &quot;sql_alchemy_conn&quot;
  celery:
    broker_url_secret: 'broker_url_conn'
    result_backend_secret: 'result_backend_conn'
  secrets:
    backend: 'airflow.providers.hashicorp.secrets.vault.VaultBackend'
    backend_kwargs: '{{ .Values.secrets_backend_kwargs | toJson }}'
</code></pre>
<p>In my vault, I have the proper encoded URI (<code>postgresql%2Bpsycopg2%3A%2F%2Fblahblahblah...</code>) stored at the right location:</p>
<p><a href="https://i.stack.imgur.com/V0R2G.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/V0R2G.png" alt="enter image description here" /></a></p>
<p>Now when I deploy the application, <code>run-airflow-migration</code> job fails with AirflowConfigException while validating the configs.</p>
<pre><code># AirflowConfigException
error: cannot use sqlite with the CeleryExecutor
</code></pre>
<p>Which means it is not picking the postgres database connection settings from Vault and using sqlite settings from Airflow configs.</p>
<p>I also tried this by providing these configs in the variables for the containers.</p>
<pre class="lang-yaml prettyprint-override"><code>env:
  - name: VAULT_ADDR
    value: 'https://path.to.vault/'
  - name: AIRFLOW__CORE__SQL_ALCHEMY_CONN_SECRET
    value: 'sql_alchemy_conn'
  - name: AIRFLOW__CELERY__BROKER_URL_SECRET
    value: 'broker_url_conn'
  - name: AIRFLOW__CELERY__RESULT_BACKEND_SECRET
    value: 'result_backend_conn'
</code></pre>
<p>I can see these environment variables in the container but the deployment fails because of the error in migration job.</p>
<p>I verified the Vault path and connectivity by changing the <code>config_path</code> in the secret backend kwargs and if I change that I still reach the Vault but since the key/connection id doesn't exist on the incorrect path so I get error from Vault. Which means that When I am setting the <code>_secret</code> as config, application is reaching Vault to fetch value but not updating or using the value.</p>
<p>I can't figure what am I missing.</p>

<p>Configs are set like Variables in the secret backend, while I was setting them like a Connection with <code>conn_uri</code> as key.</p>
<p>It is not clear in the official Documentation and had to go through the code to find the fix.</p>
<p>So after putting my configs in correct way (where mount point is <code>secret</code> and <code>config_path</code> is <code>applications/secrets/airflow/config/</code>), it worked.</p>
<pre><code>vault kv put secret/applications/secrets/airflow/config/sql_alchemy_conn value=postgresql://user:pass@host:5432/db
</code></pre>
