title: GitLab docker image does not reconfigure certain parameters from gitlab.rb after a change has been made
body: <h3>Environment</h3>
  
  <ul>
  <li>Host System: Ubuntu 16.04 LTS</li>
  <li><p>Apache 2.24 on host w/ reverse proxy to the container</p>
  
  <pre><code>$ sudo docker version
  Client:
   Version:      17.06.0-ce
   API version:  1.30
   Go version:   go1.8.3
   Git commit:   02c1d87
   Built:        Fri Jun 23 21:23:31 2017
   OS/Arch:      linux/amd64
  
  Server:
   Version:      17.06.0-ce
   API version:  1.30 (minimum version 1.12)
   Go version:   go1.8.3
   Git commit:   02c1d87
   Built:        Fri Jun 23 21:19:04 2017
   OS/Arch:      linux/amd64
   Experimental: false
  
  $ sudo docker info
  Containers: 1
   Running: 1
   Paused: 0
   Stopped: 0
  Images: 3
  Server Version: 17.06.0-ce
  Storage Driver: aufs
   Root Dir: /var/lib/docker/aufs
   Backing Filesystem: extfs
   Dirs: 30
   Dirperm1 Supported: true
  Logging Driver: json-file
  Cgroup Driver: cgroupfs
  Plugins:
   Volume: local
   Network: bridge host macvlan null overlay
   Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
  Swarm: inactive
  Runtimes: runc
  Default Runtime: runc
  Init Binary: docker-init
  containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
  runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
  init version: 949e6fa
  Security Options:
   apparmor
   seccomp
    Profile: default
  Kernel Version: 4.4.0-47-generic
  Operating System: Ubuntu 16.04.2 LTS
  OSType: linux
  Architecture: x86_64
  CPUs: 4
  Total Memory: 15.67GiB
  Name: &lt;masked&gt;
  ID: CWMI:WONN:2RZP:5HFD:GN52:BO3S:GNNW:P6WJ:5MQJ:PQLG:ZTWR:4JHM
  Docker Root Dir: /var/lib/docker
  Debug Mode (client): false
  Debug Mode (server): false
  Registry: https://index.docker.io/v1/
  Experimental: false
  Insecure Registries:
   127.0.0.0/8
  Live Restore Enabled: false
  </code></pre></li>
  </ul>
  
  <h3>Issue</h3>
  
  <p>I have torn down and rebuilt this server so many times in the last week getting various other items to function in my particular setup. It seems like when I first built the server, it saved a lot of my configuration but now does not allow me to change it.</p>
  
  <p>Here is the SMTP/Email portion of my <code>gitlab.rb</code> (with my actual fqdn replaced by "mydomain.com":</p>
  
  <pre><code>### Email Settings
  # gitlab_rails['gitlab_email_enabled'] = true
  # gitlab_rails['gitlab_email_from'] = 'gitlab@mydomain.com'
  # gitlab_rails['gitlab_email_display_name'] = 'GitLab'
  # gitlab_rails['gitlab_email_reply_to'] = 'noreply@mydomain.com'
  # gitlab_rails['gitlab_email_subject_suffix'] = ''
  
  ### GitLab email server settings
  ###! Docs: https://docs.gitlab.com/omnibus/settings/smtp.html
  ###! **Use smtp instead of sendmail/postfix.**
  
  # gitlab_rails['smtp_enable'] = true
  # gitlab_rails['smtp_address'] = "mysmtp"
  # gitlab_rails['smtp_port'] = 587
  # gitlab_rails['smtp_user_name'] = "&lt;username&gt;"
  # gitlab_rails['smtp_password'] = "&lt;pass&gt;"
  # gitlab_rails['smtp_domain'] = "mysmtp"
  # gitlab_rails['smtp_authentication'] = "plain"
  # gitlab_rails['smtp_enable_starttls_auto'] = true
  # gitlab_rails['smtp_tls'] = true
  
  ###! **Can be: 'none', 'peer', 'client_once', 'fail_if_no_peer_cert'**
  ###! Docs: http://api.rubyonrails.org/classes/ActionMailer/Base.html
  # gitlab_rails['smtp_openssl_verify_mode'] = 'none'
  
  # gitlab_rails['smtp_ca_path'] = "/etc/ssl/certs"
  # gitlab_rails['smtp_ca_file'] = "/etc/ssl/certs/ca-certificates.crt"
  </code></pre>
  
  <p>Based on the information from GitLab's SMTP documentation, I should be able to test sending mail via the ruby console like this:</p>
  
  <pre><code>gitlab-rails console
  irb(main):007:0&gt; Notify.test_email('adam@adamwgay.com', 'Message Subject', 'Message Body').deliver_now
  </code></pre>
  
  <p>However, the output from the console is unexpected. None of my settings from <code>gitlab.rb</code> are present, and it seems to be using some default set of options (including attempting to use <code>sendmail</code>, which it spits up an error about right at the beginning).</p>
  
  <pre><code>irb(main):007:0&gt; Notify.test_email('adam@adamwgay.com', 'Message Subject', 'Message Body').deliver_now
  
  Notify#test_email: processed outbound mail in 1.2ms
  sh: 1: /usr/sbin/sendmail: not found
  
  Sent mail to adam@adamwgay.com (5.8ms)
  Date: Tue, 25 Jul 2017 19:16:15 +0000
  From: GitLab &lt;gitlab@gitlab.mydomain.com&gt;
  Reply-To: GitLab &lt;noreply@gitlab.mydomain.com&gt;
  To: adam@adamwgay.com
  Message-ID: &lt;597798ff6423a_9593fdebe2c710863b8@gitlab.mydomain.com&gt;
  Subject: Message Subject
  Mime-Version: 1.0
  Content-Type: text/html;
   charset=UTF-8
  Content-Transfer-Encoding: 7bit
  Auto-Submitted: auto-generated
  X-Auto-Response-Suppress: All
  
  &lt;!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd"&gt;
  &lt;html&gt;&lt;body&gt;&lt;p&gt;Message Body&lt;/p&gt;&lt;/body&gt;&lt;/html&gt;
  
  =&gt; #&lt;Mail::Message:70225704413280, Multipart: false, Headers: &lt;Date: Tue, 25 Jul 2017 19:16:15 +0000&gt;, &lt;From: GitLab &lt;gitlab@gitlab.mydomain.com&gt;&gt;, &lt;Reply-To: GitLab &lt;noreply@gitlab.mydomain.com&gt;&gt;, &lt;To: adam@adamwgay.com&gt;, &lt;Message-ID: &lt;597798ff6423a_9593fdebe2c710863b8@gitlab.mydomain.com.mail&gt;&gt;, &lt;Subject: Message Subject&gt;, &lt;Mime-Version: 1.0&gt;, &lt;Content-Type: text/html; charset=UTF-8&gt;, &lt;Content-Transfer-Encoding: 7bit&gt;, &lt;Auto-Submitted: auto-generated&gt;, &lt;X-Auto-Response-Suppress: All&gt;&gt;
  </code></pre>
  
  <p>There are a couple of issues that I see with this output, and something tells me that the SMTP settings from gitlab.rb are not being parsed/delivered to the GitLab system properly:</p>
  
  <ul>
  <li>All the <code>From</code> and <code>Reply-To</code> fields in this output DO NOT match what I have set in <code>gitlab.rb</code></li>
  <li>This line, <code>sh: 1: /usr/sbin/sendmail: not found</code>, seems to indicate the system is still trying to use sendmail. Again, not what I have set in <code>gitlab.rb</code></li>
  </ul>
  
  <p>Looking further, It seems that nothing in my <code>gitlab.rb</code> file is being updated upon running <code>gitlab-ctl reconfigure</code> then restarting the container.</p>
  
  <ul>
  <li>If I change settings directly (even if there's a volume on my host), say the nginx conf files, then a restart always reverts the settings, despite what I have set in the <code>gitlab.rb</code> file.</li>
  <li>If I add a file to try and force the SMTP settings on the container, then again a restart removes this file.</li>
  </ul>
  
  <p>Is there another way to apply the settings from <code>gitlab.rb</code> to the container that I have missed?</p>
  
  <p>My order of ops:</p>
  
  <ol>
  <li>Make my changes in <code>gitlab.rb</code></li>
  <li>Run <code>sudo docker exec gitlab gitlab-ctl reconfigure</code> or just <code>gitlab-ctl reconfigure</code> inside the container</li>
  <li>Run <code>sudo docker restart gitlab</code> or <code>gitlab-ctl restart</code> inside the container</li>
  </ol>
  
  <p>Even though both ways give me a successful reconfigure message, the settings are not changed. At some point, changes I made to <code>gitlab.rb</code> got stored into the container, as every time I rebuild it certain hostnames and settings are unchanged, and my instance returns fairly quickly with all the necessary parameters for certs and HTTPS. I just can't seem to make any further changes to my gitlab container via the config files.</p>
  
  <p>Here's how I build the container, maybe this is part of the issue? Am I missing something simple?</p>
  
  <pre><code>sudo docker run --detach \
     --hostname gitlab.mydomain.com \
     --publish 10443:443 --publish 10080:80 --publish 10022:22 --publish 10587:587 --publish 10025:25 \
     --name gitlab \
     --restart always \
     --volume /srv/gitlab/config:/etc/gitlab \
     --volume /srv/gitlab/logs:/var/log/gitlab \
     --volume /srv/gitlab/data:/var/opt/gitlab \
     gitlab/gitlab-ce:latest
  </code></pre>
  
  <h3>UPDATE 1</h3>
  
  <p>If I run <code>gitlab-ctl diff-config</code> in the container, it seems to tell me that there is indeed another gitlab.rb that it is pulling from, and that it is not pulling from mine in the <code>/etc/gitlab/</code> directory within the container. It even shows me the values from my config compared to the incorrect values I see in the "package available config."</p>
  
  <h3>UPDATE 2</h3>
  
  <p>Went down the path thinking this may be a permissions issue. I'm not sure exactly what the suggested ownership should be for any persistent volumes that live on the host system. Mine end up owned by <code>root:root</code> every time I restart the container. Even if I make the folders owned by <code>root:docker</code>, then reconfigure/reset, it reverts back to being <code>root:root</code>. No idea if permissions is the issue, but just posting the update for information purposes in case it helps me track down the issue.</p>
  
  <h3>UPDATE 3</h3>
  
  <p>Ok, found a really roundabout way to bandaid my issue. Again, I think this is just a bandaid and that my <code>gitlab.rb</code> should be handling this re-config for me, I'll have to track that down when I have the time.</p>
  
  <p>The gist of the bandaid: I edited each .yml and .rb file individually on the container file system that I needed to edit (just 2 for now), and have made my server rebuild script as follows. I had to do the <code>gitlab-ctl restart</code> command, as restarting the whole container overwrites these two files again.</p>
  
  <pre><code>sudo docker run --detach \
      --hostname gitlab.mydomain.com \
      --publish 10443:443 --publish 10080:80 --publish 10022:22 --publish 10587:587 \
      --name gitlab \
      --restart always \
      --volume /srv/gitlab/config:/etc/gitlab \
      --volume /srv/gitlab/logs:/var/log/gitlab \
      --volume /srv/gitlab/data:/var/opt/gitlab \
      gitlab/gitlab-ce:latest
  sudo docker exec gitlab cp /etc/gitlab/gitlab.yml /var/opt/gitlab/gitlab-rails/etc/gitlab.yml
  sudo docker exec gitlab cp /etc/gitlab/smtp_settings.rb /opt/gitlab/embedded/service/gitlab-rails/config/initializers/smtp_settings.rb
  sudo docker exec gitlab gitlab-ctl restart
  </code></pre>
  
  <h3>UPDATE 4 (tldr;)</h3>
  
  <p>I had my config files and all parameters commented out... removed the single <code>#</code> from what I needed and now all is well.</p>