<p>In attempt to setup airflow logging to localstack s3 buckets, for local and kubernetes dev environments, I am following the <a href="https://airflow.apache.org/docs/1.10.1/howto/write-logs.html" rel="noreferrer">airflow documentation for logging to s3</a>. To give a little context, <a href="https://github.com/localstack/localstack" rel="noreferrer">localstack</a> is a local AWS cloud stack with AWS services including s3 running locally.</p>

<p>I added the following environment variables to my airflow containers <a href="https://stackoverflow.com/questions/44780736/setting-up-s3-for-logs-in-airflow#answer-48194903">similar to this other stack overflow post</a> in attempt to log to my local s3 buckets. This is what I added to <code>docker-compose.yaml</code> for all airflow containers:</p>

<pre><code>       - AIRFLOW__CORE__REMOTE_LOGGING=True
       - AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER=s3://local-airflow-logs
       - AIRFLOW__CORE__REMOTE_LOG_CONN_ID=MyS3Conn
       - AIRFLOW__CORE__ENCRYPT_S3_LOGS=False
</code></pre>

<p>I've also added my localstack s3 creds to <code>airflow.cfg</code></p>

<pre class="lang-sh prettyprint-override"><code>[MyS3Conn]
aws_access_key_id = foo
aws_secret_access_key = bar
aws_default_region = us-east-1
host = http://localstack:4572    # s3 port. not sure if this is right place for it 
</code></pre>

<p>Additionally, I've installed apache-airflow[hooks], and apache-airflow[s3], though it's not clear which one is really needed based on the <a href="https://airflow.apache.org/docs/1.10.1/howto/write-logs.html" rel="noreferrer">documentation</a>.</p>

<p>I've followed the steps in <a href="https://stackoverflow.com/questions/48817258/setting-up-s3-logging-in-airflow">a previous stack overflow post</a> in attempt verify if the S3Hook can write to my localstack s3 instance: </p>

<pre class="lang-py prettyprint-override"><code>from airflow.hooks import S3Hook
s3 = S3Hook(aws_conn_id='MyS3Conn')
s3.load_string('test','test',bucket_name='local-airflow-logs')
</code></pre>

<p>But I get <code>botocore.exceptions.NoCredentialsError: Unable to locate credentials</code>. </p>

<p>After adding credentials to airflow console under <code>/admin/connection/edit</code> as depicted: 
<a href="https://i.stack.imgur.com/oRQFR.png" rel="noreferrer"><img src="https://i.stack.imgur.com/oRQFR.png" alt="enter image description here"></a>
this is the new exception, <code>botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the PutObject operation: The AWS Access Key Id you provided does not exist in our records.</code> is returned. <a href="https://github.com/localstack/localstack/issues/1568" rel="noreferrer">Other people have encountered this same issue</a> and it may have been related to networking.</p>

<p>Regardless, a programatic setup is needed, not a manual one.</p>

<p>I was able to access the bucket using a standalone Python script (entering AWS credentials explicitly with boto), but it needs to work as part of airflow.</p>

<p>Is there a proper way to set up host / port / credentials for <code>S3Hook</code> by adding <code>MyS3Conn</code> to <code>airflow.cfg</code>? </p>

<p>Based on the <a href="https://github.com/apache/airflow/blob/58c3542ed25061320ce61dbe0adf451a44c738dd/airflow/providers/amazon/aws/hooks/s3.py" rel="noreferrer">airflow s3 hooks source code</a>, it seems a custom s3 URL may not yet be supported by airflow. However, based on the airflow <a href="https://github.com/apache/airflow/blob/58c3542ed25061320ce61dbe0adf451a44c738dd/airflow/providers/amazon/aws/hooks/aws_hook.py" rel="noreferrer">aws_hook source code</a> (parent) it seems it should be possible to set the endpoint_url including port, and it should be read from <code>airflow.cfg</code>.</p>

<p>I am able to inspect and write to my s3 bucket in localstack using boto alone. Also, <code>curl http://localstack:4572/local-mochi-airflow-logs</code> returns the contents of the bucket from the airflow container. And <code>aws --endpoint-url=http://localhost:4572 s3 ls</code> returns <code>Could not connect to the endpoint URL: "http://localhost:4572/"</code>.</p>

<p>What other steps might be needed to log to localstack s3 buckets from airflow running in docker, with automated setup and is this even supported yet?  </p>

oasisPolo
<p>I think you're supposed to use <code>localhost</code> not <code>localstack</code> for the endpoint, e.g. <code>host = http://localhost:4572</code>.</p>

<p>In Airflow 1.10 you can override the endpoint on a per-connection basis but unfortunately it only supports one endpoint at a time so you'd be changing it for <em>all</em> AWS hooks using the connection. To override it, edit the relevant connection and in the "Extra" field put:</p>

<pre><code>{"host": "http://localhost:4572"}
</code></pre>

<p>I believe this will fix it?</p>
