title: Setting up S3 logging in Airflow
body: <p>This is driving me nuts.</p>
  
  <p>I'm setting up airflow in a cloud environment. I have one server running the scheduler and the webserver and one server as a celery worker, and I'm using airflow 1.8.0.</p>
  
  <p>Running jobs works fine. What <em>refuses</em> to work is logging.</p>
  
  <p>I've set up the correct path in airflow.cfg on both servers:</p>
  
  <blockquote>
    <p>remote_base_log_folder = s3://my-bucket/airflow_logs/</p>
    
    <p>remote_log_conn_id = s3_logging_conn</p>
  </blockquote>
  
  <p>I've set up s3_logging_conn in the airflow UI, with the access key and the secret key as described <a href="https://stackoverflow.com/questions/44780736/setting-up-s3-for-logs-in-airflow">here</a>.</p>
  
  <p>I checked the connection using</p>
  
  <blockquote>
    <p>s3 = airflow.hooks.S3Hook('s3_logging_conn')</p>
    
    <p>s3.load_string('test','test',bucket_name='my-bucket')</p>
  </blockquote>
  
  <p>This <em>works</em> on both servers. So the connection is properly set up. Yet all I get whenever I run a task is </p>
  
  <blockquote>
    <p>*** Log file isn't local.</p>
    
    <p>*** Fetching here: http://*******</p>
    
    <p>*** Failed to fetch log file from worker.</p>
    
    <p>*** Reading remote logs...</p>
    
    <p>Could not read logs from s3://my-bucket/airflow_logs/my-dag/my-task/2018-02-15T21:46:47.577537</p>
  </blockquote>
  
  <p>I tried manually uploading the log following the expected conventions and the webserver still can't pick it up - so the problem is on both ends. I'm at a loss at what to do, everything I've read so far tells me this <em>should</em> be working. I'm close to just installing the 1.9.0 which I hear changes logging and see if I'm more lucky.</p>
  
  <p>UPDATE: I made a clean install of Airflow 1.9 and followed the specific instructions <a href="https://stackoverflow.com/a/48194903/5378460">here</a>.</p>
  
  <p>Webserver won't even start now with the following error:</p>
  
  <blockquote>
    <p>airflow.exceptions.AirflowConfigException: section/key [core/remote_logging] not found in config</p>
  </blockquote>
  
  <p>There <em>is</em> an explicit reference to this section in <a href="https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/airflow_local_settings.py" rel="nofollow noreferrer">this config template</a>.</p>
  
  <p>So I tried removing it and just loading the S3 handler without checking first and I got the following error message instead:</p>
  
  <blockquote>
    <p>Unable to load the config, contains a configuration error.</p>
    
    <p>Traceback (most recent call last):</p>
    
    <p>File "/usr/lib64/python3.6/logging/config.py", line 384, in resolve:</p>
    
    <p>self.importer(used)</p>
    
    <p>ModuleNotFoundError: No module named </p>
    
    <p>'airflow.utils.log.logging_mixin.RedirectStdHandler'; </p>
    
    <p>'airflow.utils.log.logging_mixin' is not a package</p>
  </blockquote>
  
  <p>I get the feeling that this shouldn't be <em>this</em> hard.</p>
  
  <p>Any help would be much appreciated, cheers</p>