title: How to distribute files in a Python sdist that are not VCS tracked?
body: <p>I would like to find the proper way to include files in a python sdist that are not tracked by git.</p>
  
  <h1>Context</h1>
  
  <p>The <code>.mo</code> files from my project are not tracked by <code>git</code> (like some other <code>.txt</code> files that need to be created at install time).</p>
  
  <p>I have written a small function in <code>setup.py</code> to create them at install time, that I call in <code>setup()</code>:</p>
  
  <pre><code>setup(
      .
      .
      .
      data_files=create_extra_files(),
      include_package_data=True,
      .
      .
      .
  )
  </code></pre>
  
  <p>Note that they should belong to <code>data_dir</code> because <a href="https://docs.python.org/3.4/distutils/setupscript.html#installing-additional-files" rel="noreferrer">the documentation says</a>:</p>
  
  <blockquote>
    <p>The data_files option can be used to specify additional files needed
    by the module distribution: configuration files, <strong>message catalogs</strong>,
    data files, anything which doesn’t fit in the previous categories.</p>
  </blockquote>
  
  <p>So, this works well with <code>python3 setup.py install</code> (and <code>bdist</code> too). The <code>.mo</code> files are generated and stored at the right place.</p>
  
  <p>But if I want it to work with <code>sdist</code>, then I must include them in <code>MANIFEST.in</code> (e.g. <code>recursive-include mathmaker *.mo</code>). <a href="https://docs.python.org/3.4/distutils/setupscript.html#installing-additional-files" rel="noreferrer">Documentation says indeed</a>:</p>
  
  <blockquote>
    <p>Changed in version 3.1:  All the files that match data_files will be added to the MANIFEST file
    <strong>if no template is provided</strong>. See <a href="https://docs.python.org/3.4/distutils/sourcedist.html#manifest" rel="noreferrer">Specifying the files to distribute</a>.</p>
  </blockquote>
  
  <p>(The link doesn't help much).</p>
  
  <p>I am reluctant to include <code>*.mo</code> files in <code>MANIFEST.in</code> as they are not tracked by git. And <a href="https://pypi.python.org/pypi/check-manifest" rel="noreferrer">check-manifest</a> doesn't like this kind of situation, it complains about the fact that <code>lists of files in version control and sdist do not match!</code></p>
  
  <p>So, is there a way to fix this ugly situation?</p>
  
  <h1>Steps to reproduce the situation</h1>
  
  <h2>Environment and project</h2>
  
  <p>To avoid polluting your environment, create and activate a dedicated virtual environment (python3.4+) in the directory of your choice:</p>
  
  
  
  <pre class="lang-none prettyprint-override"><code>$ pyvenv-3.4 v0
  $ source v0/bin/activate
  (v0) $
  </code></pre>
  
  <p>Reproduce following tree in a <code>project0</code> directory:</p>
  
  <pre class="lang-none prettyprint-override"><code>.
  ├── .gitignore
  ├── MANIFEST.in
  ├── README.rst
  ├── setup.py
  └── project0
      ├── __init__.py
      ├── main.py
      └── data
          └── dummy_versioned.po
  </code></pre>
  
  <p>Where <code>README.rst</code>, <code>__init__.py</code> and <code>dummy_versioned.po</code> are empty. </p>
  
  <p>Content of the other files:</p>
  
  <ul>
  <li><p><code>.gitignore</code>:</p>
  
  <pre class="lang-none prettyprint-override"><code>build/
  dist/
  *.egg-info
  project0/data/*.txt
  *~
  </code></pre></li>
  <li><p><code>MANIFEST.in</code>:</p>
  
  <pre class="lang-none prettyprint-override"><code>recursive-include project0 *.po
  recursive-include project0 *.txt
  </code></pre></li>
  <li><p><code>main.py</code>:</p>
  
  <pre class="lang-py prettyprint-override"><code>#!/usr/bin/env python3
  # -*- coding: utf-8 -*-
  
  
  def entry_point():
      with open('project0/data/a_file.txt', mode='rt') as f:
          print(f.read())
  </code></pre></li>
  <li><p><code>setup.py</code>:</p>
  
  <pre class="lang-py prettyprint-override"><code>#!/usr/bin/env python3
  # -*- coding: utf-8 -*-
  
  import platform
  from setuptools import setup, find_packages
  
  
  def create_files():
      txt_file_path = 'project0/data/a_file.txt'
      with open(txt_file_path, mode='w+') as f:
          f.write("Some dummy platform information: " + platform.platform())
      return [('project0/data', [txt_file_path])]
  
  
  setup(
      name='project0',
      version='0.0.1',
      author='J. Doe',
      author_email='j.doe@someprovider.com',
      url='http://myproject.url',
      packages=find_packages(),
      data_files=create_files(),
      include_package_data=True,
      entry_points={
          'console_scripts': ['myscript0 = project0.main:entry_point'],
      }
  )
  </code></pre></li>
  </ul>
  
  <p>Start a local <code>git</code> repo:</p>
  
  <pre class="lang-none prettyprint-override"><code>(v0) $ git init
  (v0) $ git add .
  </code></pre>
  
  <p>Install <code>check-manifest</code>:</p>
  
  <pre class="lang-none prettyprint-override"><code>(v0) $ pip3 install check-manifest
  </code></pre>
  
  <h2>Install and test</h2>
  
  <p><code>install</code> works:</p>
  
  <pre class="lang-none prettyprint-override"><code>(v0) $ python3 setup.py install
  .
  .
  .
  copying project0/data/a_file.txt -&gt; build/lib/project0/data
  .
  .
  .
  Finished processing dependencies for project0==0.0.1
  (v0) $ myscript0 
  Some dummy platform information: Linux-3.16.0-29-generic-x86_64-with-Ubuntu-14.04-trusty
  </code></pre>
  
  <p>If you <code>rm project0/data/a_file.txt</code>, then <code>myscript0</code> doesn't work anymore, but reinstall it and it works again, as expected.</p>
  
  <p>Building the sdist also includes <code>a_file.txt</code>:</p>
  
  <pre class="lang-none prettyprint-override"><code>(v0) $ python3 setup.py sdist
  .
  .
  .
  hard linking project0/data/a_file.txt -&gt; project0-0.0.1/project0/data
  .
  .
  .
  </code></pre>
  
  <p>Note that to have this file included in the sdist, it looks necessary (as explained in the "context" part below) to have <code>recursive-include project0 *.txt</code> in <code>MANIFEST.in</code>. Would you remove this line, <code>python3 setup.py sdist</code> woudln't mention <code>a_file.txt</code> any more (do not forget to remove any previous <code>build/</code> or <code>dist/</code> directories to observe this).</p>
  
  <h2>Conclusion</h2>
  
  <p>So, everything works as it is, but there is this discrepancy: <code>a_file.txt</code> is not tracked by <code>git</code>, but is included in <code>MANIFEST.in</code>.</p>
  
  <p><code>check-manifest</code> tells clearly:</p>
  
  <pre class="lang-none prettyprint-override"><code>lists of files in version control and sdist do not match!
  missing from VCS:
    project0/data/a_file.txt
  </code></pre>
  
  <p>So, is there a proper way to handle this situation?</p>