<p>I'm currently working on a project which runs with .NET Core 2.1 on Kubernetes. My application reading configuration values from <code>appsettings.json</code> files and Environment variables.</p>
<p>Problem is when I tried the read configuration from <code>IConfiguration</code> interface, &quot;most of the time&quot; value returns null, but only in Kubernetes. It's perfectly working on my development machine and test machine with Windows VM &amp; IIS. The difference between these environments is development and testing machines are Windows but productions is Kubernetes running on linux containers (based on <strong>microsoft/dotnet:2.1-aspnetcore-runtime</strong> image) and how I implement environment variables. In my development machine there is no environment variables defined. In Windows VM I do use Environment Variables through <code>web.config</code> entries like these:</p>
<pre class="lang-xml prettyprint-override"><code>&lt;aspNetCore processPath=&quot;dotnet&quot; arguments=&quot;myapp.dll&quot; stdoutLogEnabled=&quot;false&quot; stdoutLogFile=&quot;.\logs\stdout&quot;&gt;
    &lt;environmentVariables&gt;
         &lt;environmentVariable name=&quot;Site:SiteId&quot; value=&quot;1&quot; /&gt;
         &lt;environmentVariable name=&quot;Site:Theme&quot; value=&quot;MyTheme&quot; /&gt;
    &lt;/environmentVariables&gt;
&lt;/aspNetCore&gt;
</code></pre>
<p>I'm using <code>WebHost.CreateDefaultBuilder()</code> for the configuration and documentation says methods implements <code>appsettings.json</code>, <code>appsettings.{EnvironmentName}.json</code>, environment variables and command line arguments for configuration sources.</p>
<p>My <code>appsettings.json</code> file:</p>
<pre class="lang-json prettyprint-override"><code>{
  &quot;Site&quot;: {
    &quot;SiteId&quot;: 1,
    &quot;Theme&quot;: &quot;Limitless&quot;
  },
  &quot;RemoteClient&quot;: {
    &quot;HostUrl&quot;: &quot;http://localhost:59554/management/&quot;,
    &quot;AppKey&quot;: &quot;test&quot;,
    &quot;AppSecret&quot;: &quot;1&quot;
  }
}
</code></pre>
<p>And lastly my Kubernetes deployment file:</p>
<pre class="lang-yml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      component: myapp
  strategy: {}
  template:
    metadata:
      labels:
        component: myapp
    spec:
      nodeSelector:
        beta.kubernetes.io/os: linux
      containers:
      - image: 'myregistry/myapp:latest'
        imagePullPolicy: IfNotPresent
        name: myapp
        env:
        - name: &quot;Site__SiteId&quot;
          value: &quot;1&quot;
        - name: &quot;Site__Theme&quot;
          value: &quot;Limitless&quot;
        - name: &quot;RemoteClient__HostUrl&quot;
          value: &quot;http://apphost/management&quot;
        - name: &quot;RemoteClient__AppKey&quot;
          value: e824b2670e644824bdd5cdee1db9bd4b
        - name: &quot;RemoteClient__AppSecret&quot;
          value: 12d4deadf948
        - name: &quot;ASPNETCORE_ENVIRONMENT&quot;
          value: Production
        ports:
        - containerPort: 80
      restartPolicy: Always
</code></pre>
<p>I' m saying &quot;most of the time&quot; because it behaves different each time pod restarts. Sometimes it reads one variable but not other, other times it reads rest of the variables and eventually it can read all variables and application runs normally. You can see in <code>kubectl describe pod</code> output says</p>
<p><a href="https://i.stack.imgur.com/OHP23.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/OHP23.png" alt="kubectl describe pod output" /></a></p>
<p>But when I read the values from <code>IConfiguration</code> service and output to the console the output looks like this:</p>
<p><a href="https://i.stack.imgur.com/qFaWe.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/qFaWe.png" alt="configuration output 1" /></a></p>
<p>and other restart</p>
<p><a href="https://i.stack.imgur.com/qj0ZI.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/qj0ZI.png" alt="configuration output 2" /></a></p>
<p>I didn't change a thing between these two runs, it just restarted. It didn't read the values even default values defined in appsettings.json.</p>
<p>So what do you think why is this happens?</p>
<p><strong>Edit:</strong></p>
<p>I' m using this code block for initial configuration, <code>CreateDefaultBuilder()</code> responsible for configuration.</p>
<pre class="lang-cs prettyprint-override"><code>public static IWebHostBuilder CreateWebHostBuilder(string[] args) =&gt;
            WebHost.CreateDefaultBuilder(args)
                .UseStartup&lt;Startup&gt;()
                .AddPreConfigureServices();
</code></pre>
<p>and annotations of <code>CreateDefaultBinder()</code>:</p>
<pre class="lang-cs prettyprint-override"><code>//
// Summary:
//     Initializes a new instance of the Microsoft.AspNetCore.Hosting.WebHostBuilder
//     class with pre-configured defaults.
//
// Parameters:
//   args:
//     The command line args.
//
// Returns:
//     The initialized Microsoft.AspNetCore.Hosting.IWebHostBuilder.
//
// Remarks:
//     The following defaults are applied to the returned Microsoft.AspNetCore.Hosting.WebHostBuilder:
//     use Kestrel as the web server and configure it using the application's configuration
//     providers, set the Microsoft.AspNetCore.Hosting.IHostingEnvironment.ContentRootPath
//     to the result of System.IO.Directory.GetCurrentDirectory, load Microsoft.Extensions.Configuration.IConfiguration
//     from 'appsettings.json' and 'appsettings.[Microsoft.AspNetCore.Hosting.IHostingEnvironment.EnvironmentName].json',
//     load Microsoft.Extensions.Configuration.IConfiguration from User Secrets when
//     Microsoft.AspNetCore.Hosting.IHostingEnvironment.EnvironmentName is 'Development'
//     using the entry assembly, load Microsoft.Extensions.Configuration.IConfiguration
//     from environment variables, load Microsoft.Extensions.Configuration.IConfiguration
//     from supplied command line args, configures the Microsoft.Extensions.Logging.ILoggerFactory
//     to log to the console and debug output, enables IIS integration, and enables
//     the ability for frameworks to bind their options to their default configuration
//     sections.
public static IWebHostBuilder CreateDefaultBuilder(string[] args);
</code></pre>
<p><strong>Edit 2:</strong></p>
<p>Reading values in my <code>Startup.cs</code> like this:</p>
<pre class="lang-cs prettyprint-override"><code>services.AddMvc(options =&gt; { })
    .AddEntityBinders()
    .AddRazorOptions(options =&gt;
    {
        string theme = Configuration[&quot;Site:Theme&quot;];

        options.ViewLocationFormats.Clear();
        options.ViewLocationFormats.Add($&quot;~/Views/{theme}Theme/{{1}}/{{0}}.cshtml&quot;);
        options.ViewLocationFormats.Add($&quot;~/Views/{theme}Theme/Shared/{{0}}.cshtml&quot;);
    })
    .SetCompatibilityVersion(Microsoft.AspNetCore.Mvc.CompatibilityVersion.Version_2_1);
</code></pre>

<p>After long and difficult debugging session. I can finally pinpoint the problem.</p>
<p>.Net Core uses <code>:</code> as Configuration key separator but Kubernetes configuration files doesn't support such character when naming environment variables. <code>EnvironmentVariablesConfigurationProvider</code> solve this by accepting <code>__</code> as separator and replacing with <code>:</code> when reading values.</p>
<p>So, I declare my environment variables with __ separator in my Kubernetes deployment file (as you can see above). But Dockerfile didn't have this problem and I write ENV definitions with <code>:</code> separator (below) and this creates duplicate variables which .Net Core reads all of them and creates unstable results.</p>
<pre><code>ENV RemoteClient:HostUrl ${HOST_URL}
ENV RemoteClient:AppKey ${APP_KEY}
ENV RemoteClient:AppSecret ${APP_SECRET}
ENV ASPNETCORE_ENVIRONMENT ${ENVIRONMENT_NAME}
</code></pre>
