<p>I'm deploying my Node.js server on AKS and I'm experiencing a few unexpected behaviours not happening on my local machine. I've tried various things to debug it but I'm not succeeding at all.</p>
<p>I use a Middleware as</p>
<pre><code>exports.clientApiKeyValidation = (req, res, next) =&gt; {

   try {
      const clientApiKey = req.get('api_key');
      console.log(`received api_key is ${clientApiKey}
      and expected API_KEY is ${process.env.API_KEY}`);
      const receivedTypeof = console.log('clientApiKey ', typeof clientApiKey);
      const expectedTypeof = console.log('expected ', typeof process.env.API_KEY);
      console.log('req.headers is: ', req.headers);
      if (!clientApiKey) {
         return res.status(400).send({
            status: false,
            message: &quot;Missing Api Key&quot;
         });
      }
      if (clientApiKey === process.env.API_KEY) {

         // console.log('Api key correct');
         next();
      }
      else {
         return res.status(400).send({
            status: false,
            message: &quot;Invalid Api Key&quot;
         });
      }
   } catch (error) {
      res.status(401).json({
         error: new Error('Invalid request!')
      });
   }

}
</code></pre>
<p>used on routes like</p>
<pre><code>router.get('/users', auth.clientApiKeyValidation, userController.findUsers); 
</code></pre>
<p>In <code>app.js</code> I also set a few things, including a middleware function to allow this params as</p>
<pre><code>app.disable('x-powered-by');
app.use(express.json({ limit: '50mb' }));
app.use(function (req, res, next) {
  // either res.setHeader or res.header works.
  // res.setHeader('Access-Control-Allow-Origin', 'http://localhost:5000');

  res.setHeader(
    'Access-Control-Allow-Origin',
    'https://xxx.westeurope.cloudapp.azure.com'
  );
  res.setHeader(
    'Access-Control-Allow-Methods',
    'GET, POST, PUT, DELETE'
  );
  // res.setHeader('Access-Control-Allow-Methods', 'GET, POST, PUT, DELETE');
  res.setHeader(
    'Access-Control-Allow-Headers',
    'content-type, api_key, AuthToken, apikey'
  );
  // res.setHeader('Access-Control-Allow-Credentials', true);

  // res.header('Access-Control-Allow-Origin', 'http://localhost:5000');
  // res.header('Access-Control-Allow-Methods', 'GET, POST, OPTIONS, PUT, PATCH, DELETE');
  // res.header('Access-Control-Allow-Headers', 'Origin, Content-Type, api_key, Accept');
  // console.log(' res headers added: ', res.getHeaders());
  next();
});

app.use(express.urlencoded({ extended: true }));
app.use(cors());
app.use(express.json());
app.use(express.json({ type: 'application/json' }));


</code></pre>
<p>I'm sending requests to it from my Flutter app adding headers like</p>
<pre><code>    await _firebaseAuth.currentUser?.getIdToken().then((idToken) {
      headers = {
        'Content-Type': 'application/json',
        'api_key': Environment.dbApiKey,
        'AuthToken': idToken
      };
    });
    FixitUser? userDetails;

    // final Uri uri = Uri.http(Environment.dbUrl, '/api/users');
    final Uri uri = Uri.https(Environment.dbUrl, '/server/api/users');
    log('headers are $headers');
    await get(uri, headers: headers).then((resp) {
      log('UserMongoDBRepository.downloadUserDetails resp.body is : ${resp.body}');

...
</code></pre>
<p>When running the server directly middleware logs show it all works as expected</p>
<pre><code>received api_key is hjkdiu-slia7h-si9udd989jw-ols8dh
      and expected API_KEY is hjkdiu-slia7h-si9udd989jw-ols8dh
clientApiKey  string
expected  string
req.headers is:  {
  'user-agent': 'Dart/2.18 (dart:io)',
  'accept-encoding': 'gzip',
  api_key: 'hjkdiu-slia7h-si9udd989jw-ols8dh',
  host: '192.168.1.48:3000',
  'content-type': 'application/json',
  authtoken: 'eyJhbGciOiJSUzI1NiIsImtpZCI6IjVhNTA5ZjAxOWY3MGQ3NzlkODBmMTUyZDFhNWQzMzgxMWFiN2NlZjciLCJ0eXAiOiJKV1QifQ.eyJuYW1lIjoidmluY2Vuem8gY2FsaWEiLCJwaWN0dXJlIjoiaHR0cHM6Ly9saDYuZ29vZ2xldXNlcmNvbnRlbnQuY29tLy13TVNwMUxZd2hPZy9BQUFBQUFBQUFBSS9BQUFBQUFBQUFCSS9OdE43TTlTMEVIUS9zOTYtYy9waG90by5qcGciLCJpc3MiOiJodHRwczovL3NlY3VyZXRva2VuLmdvb2dsZS5jb20vZml4LWl0LWI0YjAwIiwiYXVkIjoiZml4LWl0LWI0YjAwIiwiYXV0aF90aW1lIjoxNjc1Nzc1MTg2LCJ1c2VyX2lkIjoiWnhtejJHSmxNUlBXdjBMRGgyRDg4Y0o3T3V6MSIsInN1YiI6Ilp4bXoyR0psTVJQV3YwTERoMkQ4OGNKN091ejEiLCJpYXQiOjE2NzU4NDY4MDAsImV4cCI6MTY3NTg1MDQwMCwiZW1haWwiOiJ2aW5jZW56by5jYWxpYS4xOTc2QGdtYWlsLmNvbSIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJwaG9uZV9udW1iZXIiOiIrMzkzNjYxNDcxMzEzIiwiZmlyZWJhc2UiOnsiaWRlbnRpdGllcyI6eyJnb29nbGUuY29tIjpbIjExNjkyMTg2MTcwOTA0NTM5MzU5MiJdLCJwaG9uZSI6WyIrMzkzNjYxNDcxMzEzIl0sImVtYWlsIjpbInZpbmNlbnpvLmNhbGlhLjE5NzZAZ21haWwuY29tIl19LCJzaWduX2luX3Byb3ZpZGVyIjoiZ29vZ2xlLmNvbSJ9fQ.ZNJFrxWlycMVgg4VAdWt6Q0WR5yrWPar5_UJwhQ9-hVX25aKC69yDpoM2adx3OIQ-hlsGz1bNvVEUSfvWUWRWF-TaX2TVeLb5z0blAtl8A1cGcvnaryffr0jpCtN_nhDg3WTtHw4B2USDe432TxSXx0ICrk8bx_fum8jsfBvAh_xU8bnr6lLsc0pltcmU-zun0rhZcC6jpxua2d0jIwWyWSXurjNBkLkduzOpUbw6KCLur7wVcxz-HmuD67D0qx5dr37malLbOBt5mcfOtdJEJcEq55XqXqUanopT_OWTWxByMrD_CXyZnLu_q1DgSBMM4kaLrSA14ETD_EvIeqHOQ',
  
}
</code></pre>
<p>When instead is running on AKS cluster <code>api_key</code> is missing from received request headers as the logs show, so the middleware responds with <code>{&quot;status&quot;:false,&quot;message&quot;:&quot;Missing Api Key&quot;}</code></p>
<pre><code>received api_key is undefined
      and expected API_KEY is some api key

req.headers is:  {
  host: 'xxx.westeurope.cloudapp.azure.com',
  'x-request-id': '2ecc2ec74c808cf40f816921374f72d4',
  'x-real-ip': '81.56.11.23',
  'x-forwarded-for': '81.56.11.23',
  'x-forwarded-host': 'xxx.westeurope.cloudapp.azure.com',
  'x-forwarded-port': '443',
  'x-forwarded-proto': 'https',
  'x-forwarded-scheme': 'https',
  'x-scheme': 'https',
  'user-agent': 'Dart/2.18 (dart:io)',
  'accept-encoding': 'gzip',
  'content-type': 'application/json',
  authtoken: 'eyJhbGciOiJSUzI1NiIsImtpZCI6IjVhNTA5ZjAxOWY3MGQ3NzlkODBmMTUyZDFhNWQzMzgxMWFiN2NlZjciLCJ0eXAiOiJKV1QifQ.eyJuYW1lIjoidmluY2Vuem8gY2FsaWEiLCJwaWN0dXJlIjoiaHR0cHM6Ly9saDYuZ29vZ2xldXNlcmNvbnRlbnQuY29tLy13TVNwMUxZd2hPZy9BQUFBQUFBQUFBSS9BQUFBQUFBQUFCSS9OdE43TTlTMEVIUS9zOTYtYy9waG90by5qcGciLCJpc3MiOiJodHRwczovL3NlY3VyZXRva2VuLmdvb2dsZS5jb20vZml4LWl0LWI0YjAwIiwiYXVkIjoiZml4LWl0LWI0YjAwIiwiYXV0aF90aW1lIjoxNjcxODc0Nzg2LCJ1c2VyX2lkIjoiWnhtejJHSmxNUlBXdjBMRGgyRDg4Y0o3T3V6MSIsInN1YiI6Ilp4bXoyR0psTVJQV3YwTERoMkQ4OGNKN091ejEiLCJpYXQiOjE2NzU3NjkyNzksImV4cCI6MTY3NTc3Mjg3OSwiZW1haWwiOiJ2aW5jZW56by5jYWxpYS4xOTc2QGdtYWlsLmNvbSIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJwaG9uZV9udW1iZXIiOiIrMzkzNjYxNDcxMzEzIiwiZmlyZWJhc2UiOnsiaWRlbnRpdGllcyI6eyJnb29nbGUuY29tIjpbIjExNjkyMTg2MTcwOTA0NTM5MzU5MiJdLCJwaG9uZSI6WyIrMzkzNjYxNDcxMzEzIl0sImVtYWlsIjpbInZpbmNlbnpvLmNhbGlhLjE5NzZAZ21haWwuY29tIl19LCJzaWduX2luX3Byb3ZpZGVyIjoiZ29vZ2xlLmNvbSJ9fQ.d8emU2BKNBV5oe3YQVHT8M8otFos_RvEmeyutdmYBDhnzyPgMZTAn_l3JikRAbcTNdDOAVoutZgTb5s8d6di3plAoE240OwwZTuSwxVpSaS7fDPt_rjQf9k2RmVsRa-fq1SWIP2ejdEbma_QngLSpXO0-PSPx4wa7mThjv2enP00TpUB9RDsOPK2QKlwOX9i1gc1_7kOPGJwouG3S3W4_kOXIxSoVjAT0P9k2xtHa99W-_gwn-9YqM1UoHrLkEs-ONKpe5SWLIet9r_PvI2l1zqb-1fGBmoeBhyzSijw_cOLJSayEoImtkCOmAA0rhMNYc--Yxuzd8EMyyp1U9dThg'
}
</code></pre>
<p>The headers prints from Flutter shows the api_key being correctly set.</p>
<pre><code>headers are {Content-Type: application/json, api_key: some api key, AuthToken: eyJhbGciOiJSUzI1NiIsImtpZCI6IjVhNTA5ZjAxOWY3MGQ3NzlkODBmMTUyZDFhNWQzMzgxMWFiN2NlZjciLCJ0eXAiOiJKV1QifQ.eyJuYW1lIjoidmluY2Vuem8gY2FsaWEiLCJwaWN0dXJlIjoiaHR0cHM6Ly9saDYuZ29vZ2xldXNlcmNvbnRlbnQuY29tLy13TVNwMUxZd2hPZy9BQUFBQUFBQUFBSS9BQUFBQUFBQUFCSS9OdE43TTlTMEVIUS9zOTYtYy9waG90by5qcGciLCJpc3MiOiJodHRwczovL3NlY3VyZXRva2VuLmdvb2dsZS5jb20vZml4LWl0LWI0YjAwIiwiYXVkIjoiZml4LWl0LWI0YjAwIiwiYXV0aF90aW1lIjoxNjcxODc0Nzg2LCJ1c2VyX2lkIjoiWnhtejJHSmxNUlBXdjBMRGgyRDg4Y0o3T3V6MSIsInN1YiI6Ilp4bXoyR0psTVJQV3YwTERoMkQ4OGNKN091ejEiLCJpYXQiOjE2NzU3NjkyNzksImV4cCI6MTY3NTc3Mjg3OSwiZW1haWwiOiJ2aW5jZW56by5jYWxpYS4xOTc2QGdtYWlsLmNvbSIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJwaG9uZV9udW1iZXIiOiIrMzkzNjYxNDcxMzEzIiwiZmlyZWJhc2UiOnsiaWRlbnRpdGllcyI6eyJnb29nbGUuY29tIjpbIjExNjkyMTg2MTcwOTA0NTM5MzU5MiJdLCJwaG9uZSI6WyIrMzkzNjYxNDcxMzEzIl0sImVtYWlsIjpbInZpbmNlbnpvLmNhbGlhLjE5NzZAZ21haWwuY29tIl19LCJzaWduX2luX3Byb3ZpZGVyIjoiZ29vZ2xlLmNvbSJ9fQ.d8emU2BKNBV5oe3YQVHT8M8otFos_RvEmeyutdmYBDhnzyPgMZTAn_l3JikRAbcTNdDOAVoutZgTb5s8d6di3plAoE240OwwZTuSwxVpSaS7fDPt_rjQf9k2RmVsRa-fq1SWIP2ejdEbma_QngLSpXO0-PSPx4wa7mThjv2enP00TpUB9RDsOPK2QKlwOX9i1gc1_7kOPGJwouG3S3W4_kOXIxSoVjAT0P9k2xtHa99W-_gwn-9YqM1UoHrLkEs-ONKpe5SWLIet9r_PvI2l1zqb-1fGBmoeBhyzSijw_cOLJSayEoImtkCOmAA0rhMNYc--Yxuzd8EMyyp1U9dThg}
</code></pre>
<p>So as a test instead of <code>api_key</code> I used <code>apikey</code> and it gets retrieved in the request, but then the <code>===</code> check fails so the middleware responds with <code>{&quot;status&quot;:false,&quot;message&quot;:&quot;Invalid Api Key&quot;}</code> even it their type and value are identical, as checked here <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Strict_equality" rel="nofollow noreferrer">https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Strict_equality</a></p>
<pre><code>received api_key is hjkdiu-slia7h-si9udd989jw-ols8dh
      and expected API_KEY is hjkdiu-slia7h-si9udd989jw-ols8dh

clientApiKey  string
expected  string
req.headers is:  {
  host: 'fixit1.westeurope.cloudapp.azure.com',
  'x-request-id': '515ad2a00b1a3db24a69e09f6d181036',
  'x-real-ip': '81.56.11.23',
  'x-forwarded-for': '81.56.11.23',
  'x-forwarded-host': 'fixit1.westeurope.cloudapp.azure.com',
  'x-forwarded-port': '443',
  'x-forwarded-proto': 'https',
  'x-forwarded-scheme': 'https',
  'x-scheme': 'https',
  'user-agent': 'Dart/2.18 (dart:io)',
  'accept-encoding': 'gzip',
  'content-type': 'application/json',
  authtoken: 'eyJhbGciOiJSUzI1NiIsImtpZCI6IjVhNTA5ZjAxOWY3MGQ3NzlkODBmMTUyZDFhNWQzMzgxMWFiN2NlZjciLCJ0eXAiOiJKV1QifQ.eyJuYW1lIjoidmluY2Vuem8gY2FsaWEiLCJwaWN0dXJlIjoiaHR0cHM6Ly9saDYuZ29vZ2xldXNlcmNvbnRlbnQuY29tLy13TVNwMUxZd2hPZy9BQUFBQUFBQUFBSS9BQUFBQUFBQUFCSS9OdE43TTlTMEVIUS9zOTYtYy9waG90by5qcGciLCJpc3MiOiJodHRwczovL3NlY3VyZXRva2VuLmdvb2dsZS5jb20vZml4LWl0LWI0YjAwIiwiYXVkIjoiZml4LWl0LWI0YjAwIiwiYXV0aF90aW1lIjoxNjc1Nzc1MTg2LCJ1c2VyX2lkIjoiWnhtejJHSmxNUlBXdjBMRGgyRDg4Y0o3T3V6MSIsInN1YiI6Ilp4bXoyR0psTVJQV3YwTERoMkQ4OGNKN091ejEiLCJpYXQiOjE2NzU4NDMxNzgsImV4cCI6MTY3NTg0Njc3OCwiZW1haWwiOiJ2aW5jZW56by5jYWxpYS4xOTc2QGdtYWlsLmNvbSIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJwaG9uZV9udW1iZXIiOiIrMzkzNjYxNDcxMzEzIiwiZmlyZWJhc2UiOnsiaWRlbnRpdGllcyI6eyJnb29nbGUuY29tIjpbIjExNjkyMTg2MTcwOTA0NTM5MzU5MiJdLCJwaG9uZSI6WyIrMzkzNjYxNDcxMzEzIl0sImVtYWlsIjpbInZpbmNlbnpvLmNhbGlhLjE5NzZAZ21haWwuY29tIl19LCJzaWduX2luX3Byb3ZpZGVyIjoiZ29vZ2xlLmNvbSJ9fQ.lDMgmTJkrm6ZnoyEU1F7Sjoo7Y8or8ZKzIrwBJ9ssQiR8yN5KD2ZhyM6yyR_Arscmyg1ZV_6RsTnFgGsVsmjiMzyX6TOXmYkmRlvvMjjjFsV8rW_W_gIdVld6vSg-JMrOlLcCeBknFDJC50bbNGYBSwQ2_C_MZIKlbFWWrqME988MOiUBlyT86t5Oofc5uVMETrpBf0a-wsFRdyEX-3uj-T3MRHza62PTcpHURoptQdIzYsBSc6WxR6WCINVjx__DbWlWrGt612Mw4iLv1XReiGriQEjTDc9cXbG0ngbiRsn0ojvZ-Jb8Pb6kj7gWRYDRsKFg2nxxFMhVeSDuIeO-w',
  apikey: 'hjkdiu-slia7h-si9udd989jw-ols8dh'
}
</code></pre>
<p>As a second test I changed the <code>===</code> operator to <code>==</code> in the middleware but still it returns false and responds with <code>{&quot;status&quot;:false,&quot;message&quot;:&quot;Invalid Api Key&quot;}</code>.</p>
<p>Then I tried to use <code>const serverApiKey = JSON.stringify(process.env.API_KEY);</code> and <code>const clientApiKey = JSON.stringify(req.get('apikey'));</code> for the comparison and they actually do yield different results</p>
<pre><code>clientApiKey json string is:  &quot;hjkdiu-slia7h-si9udd989jw-ols8dh&quot;
serverApiKey json string is:  &quot;hjkdiu-slia7h-si9udd989jw-ols8dh\n&quot;
</code></pre>
<p><code>env.API_KEY</code> does get it's value from a k8s secret, base64 encoded with <code>echo -n hjkdiu-slia7h-si9udd989jw-ols8dh |base64</code> command. To see if it was a matter of an empty space in the encoded value from the secret I tried encoding it with and without the <code>-n</code> flag but they result in the same json encoded string.</p>
<p>I'm not sure it could have something to do with the docker image I create so here's the <code>Dockrfile</code></p>
<pre><code>FROM node:18.3.0
WORKDIR /usr/app
# where available (npm@5+)
COPY ./ package.json ./
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true
# Istall the dependencies

RUN npm install
# If you are building your code for production
# RUN npm ci --only=production
COPY ./ ./


# Add the following lines for create-react-app bug, workaround
ENV CI=true
ENV WDS_SOCKET_PORT=0

CMD [&quot;npm&quot;, &quot;start&quot;]
</code></pre>
<ol>
<li>Why <code>api_key</code> param is not present while <code>apikey</code> param is in the received request headers?</li>
<li>Why the <code>\n</code> in the <code>serverApiKey</code> but not in the <code>clientApiKey</code> ?</li>
<li>Why is this happening only on AKS and not on my local machine?</li>
</ol>
<p>Thank you very much as always.</p>

Vincenzo
<p>I don't know anything about AKS, but some HTTP server implementations (for instance <a href="https://nginx.org/en/docs/http/ngx_http_core_module.html#underscores_in_headers" rel="nofollow noreferrer">nginx</a>) don't accept underscores in HTTP header names.</p>
<p>As for why the server API keys contains a trailing newline: it may be a platform thing. I would just be pragmatic about it and remove it before use: <code>process.API_KEY.trim()</code></p>
