<p>I have a testing Kubernetes cluster and I created elasticsearch on AWS which include Kibana for the log management.</p>

<p>Endpoint: <a href="https://search-this-is-my-es-wuktx5la4txs7avvo6ypuuyri.ca-central-1.es.amazonaws.com" rel="nofollow noreferrer">https://search-this-is-my-es-wuktx5la4txs7avvo6ypuuyri.ca-central-1.es.amazonaws.com</a></p>

<p>As far as I googled, I have to send logs from fluentd. Then I tried to implement DaemonSet using this <a href="https://medium.com/@while1eq1/logging-to-aws-elasticsearch-service-from-kubernetes-855ad0959251" rel="nofollow noreferrer">article</a>. No luck.</p>

<p>Could you please share any good documentation to me, please</p>

<p>I had a similar problem. Below are the full details of how I got it working.</p>

<p>Setup:</p>

<ul>
<li>AWS ES instance accessible via a VPC.</li>
<li>Using <a href="https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/fluentd-daemonset-elasticsearch.yaml" rel="nofollow noreferrer">this yaml file</a> as a template.</li>
<li>k8s client version v1.9.2</li>
<li>k8s server version v1.8.7</li>
</ul>

<p><strong>Host problem:</strong></p>

<p>The main problem I had was with defining the environment variables correctly. For <code>FLUENT_ELASTICSEARCH_HOST</code>, I was including the <code>https://</code> prefix on the host URL. Once I removed that, my connection problems went away.</p>

<p><strong>Authentication:</strong></p>

<p>There's no username or password configured for AWS ES. Per <a href="https://github.com/fluent/fluentd-kubernetes-daemonset/issues/33" rel="nofollow noreferrer">this discussion</a>, I set the <code>FLUENT_ELASTICSEARCH_USER</code> and <code>FLUENT_ELASTICSEARCH_PASSWORD</code> values to null.</p>

<p><strong>Sample configuration:</strong></p>

<p>Here's the full set of environment variables in my daemonset yaml file:</p>

<pre><code>- name:  FLUENT_ELASTICSEARCH_HOST
  value: "vpc-MY-DOMAIN.REGION.es.amazonaws.com"
- name:  FLUENT_ELASTICSEARCH_PORT
  value: "443"
- name: FLUENT_ELASTICSEARCH_SCHEME
  value: "https"
- name: FLUENT_ELASTICSEARCH_USER
  value: null
- name: FLUENT_ELASTICSEARCH_PASSWORD
  value: null
</code></pre>

<p><strong>Bonus: connecting to Kibana</strong></p>

<p>Instead of setting up AWS Cognito, I created an nginx pod in my kubernetes cluster that I use as a proxy to reach Kibana. I use the <code>kubectl port-foward</code> command to reach the nginx server from my local machine.</p>

<p>Here's my nginx.conf:</p>

<pre><code>server {
  listen 80;
  listen [::]:80;

  server_name MY-DOMAIN;

  location /_plugin/kibana {
      proxy_pass https://vpc-MY-DOMAIN.REGION.es.amazonaws.com/_plugin/kibana;
  }
  location / {
      proxy_pass https://vpc-MY-DOMAIN.REGION.es.amazonaws.com;
  }
}
</code></pre>

<p>Once the nginx pod is deployed, I run this command:</p>

<pre><code>kubectl port-forward POD_NAME 8888:80
</code></pre>

<p>Now the Kibana is accessible at <a href="http://localhost:8888/_plugin/kibana" rel="nofollow noreferrer">http://localhost:8888/_plugin/kibana</a></p>

<p>I'm still having a timeout issue with the port-foward command and a problem with nginx caching the ES service IP (since that can change), but I'll update my response once I resolve those issues.</p>
