pax_global_header00006660000000000000000000000064132050143310014502gustar00rootroot0000000000000052 comment=69b8a2954ae8be7417e5b27babfa16c195c6f73b check-manifest-0.36/000077500000000000000000000000001320501433100143135ustar00rootroot00000000000000check-manifest-0.36/.coveragerc000066400000000000000000000002051320501433100164310ustar00rootroot00000000000000[report] exclude_lines = pragma: nocover except ImportError: if __name__ == '__main__': if sys.platform == 'darwin': check-manifest-0.36/.gitignore000066400000000000000000000001061320501433100163000ustar00rootroot00000000000000*.pyc __pycache__/ .tox/ *.egg-info/ dist/ tmp/ .coverage build/ tags check-manifest-0.36/.travis.yml000066400000000000000000000007411320501433100164260ustar00rootroot00000000000000language: python sudo: false python: - 2.7 - 3.3 - 3.4 - 3.5 - 3.6 - pypy - pypy3 env: - FORCE_TEST_VCS=bzr - FORCE_TEST_VCS=git - FORCE_TEST_VCS=hg - FORCE_TEST_VCS=svn install: - pip install coverage coveralls mock script: - SKIP_NO_TESTS=1 coverage run --source=check_manifest setup.py test -q - coverage run --source=check_manifest --append check_manifest.py after_script: - coveralls notifications: email: false check-manifest-0.36/CHANGES.rst000066400000000000000000000225071320501433100161230ustar00rootroot00000000000000Changelog ========= 0.36 (2017-11-21) ----------------- - Handle empty VCS repositories more gracefully (`#84 `__). 0.35 (2017-01-30) ----------------- - Python 3.6 support. 0.34 (2016-09-14) ----------------- - Fix WindowsError due to presence of read-only files (`#74 `__). 0.33 (2016-08-29) ----------------- - Fix WindowsError due to git submodules in subdirectories (`#73 `__). Contributed by Loren Gordon. 0.32 (2016-08-16) ----------------- * New config/command line option to ignore bad ideas (ignore-bad-ideas) (`issue #67 `__). Contributed by Brecht Machiels. * Files named ``.hgsigs`` are ignored by default. Contributed by Jakub Wilk. 0.31 (2016-01-28) ----------------- - Drop Python 3.2 support. - Ignore commented-out lines in MANIFEST.in (`issue #66 `__). 0.30 (2015-12-10) ----------------- * Support git submodules (`issue #61 `__). * Revert the zc.buildout support hack from 0.26 because it causes breakage (`issue #56 `__). * Improve non-ASCII filename handling with Bazaar on Windows. 0.29 (2015-11-21) ----------------- * Fix --python with just a command name, to be found in path (`issue #57 `__). 0.28 (2015-11-11) ----------------- * Fix detection of git repositories when .git is a file and not a directory (`#53 `__). One situation where this occurs is when the project is checked out as a git submodule. * Apply ignore patterns in subdirectories too (`#54 `__). 0.27 (2015-11-02) ----------------- * Fix utter breakage on Windows, introduced in 0.26 (`issue #52 `__). (The bug -- clearing the environment unnecessarily -- could probably also cause locale-related problems on other OSes.) 0.26 (2015-10-30) ----------------- * Do not complain about missing ``.gitattributes`` file (`PR #50 `__). * Normalize unicode representation and case of filenames. (`issue #47 `__). * Support installation via zc.buildout better (`issue #35 `__). * Drop Python 2.6 support because one of our test dependencies (mock) dropped it. This also means we no longer use environment markers. 0.25 (2015-05-27) ----------------- * Stop dynamic computation of install_requires in setup.py: this doesn't work well in the presence of the pip 7 wheel cache. Use PEP-426 environment markers instead (this means we now require setuptools >= 0.7, and pip >= 6.0, and wheel >= 0.24). 0.24 (2015-03-26) ----------------- * Make sure ``setup.py`` not being added to the VCS doesn't cause hard-to-understand errors (`issue #46 `__). 0.23 (2015-02-12) ----------------- * More reliable svn status parsing; now handles svn externals (`issue #45 `__). * The test suite now skips tests for version control systems that aren't installed (`issue #42 `__). 0.22 (2014-12-23) ----------------- * More terse output by default; use the new ``-v`` (``--verbose``) flag to see all the details. * Warn the user if MANIFEST.in is missing (`issue #31 `__). * Fix IOError when files listed under version control are missing (`issue #32 `__). * Improved wording of the match/do not match messages (`issue #34 `__). * Handle a relative --python path (`issue #36 `__). * Warn about leading and trailing slashes in MANIFEST.in (`issue #37 `__). * Ignore .travis.yml by default (`issue #39 `__). * Suggest a rule for Makefile found deeper in the source tree. 0.21 (2014-06-13) ----------------- * Don't drop setup.cfg when copying version-controlled files into a clean temporary directory (`issue #29 `__). 0.20 (2014-05-14) ----------------- * Restore warning about files included in the sdist but not added to the version control system (`issue #27 `__). * Fix ``check-manifest relative/pathname`` (`issue #28 `__). 0.19 (2014-02-09) ----------------- * More correct MANIFEST.in parsing for exclusion rules. * Some effort was expended towards Windows compatibility. * Handles non-ASCII filenames, as long as they're valid in your locale (`issue #23 `__, `#25 `__). 0.18 (2014-01-30) ----------------- * Friendlier error message when an external command cannot be found (`issue #21 `__). * Add suggestion pattern for `.coveragerc`. * Python 2.6 support (`issue #22 `__). 0.17 (2013-10-10) ----------------- * Read the existing MANIFEST.in file for files to ignore (`issue #19 `__). 0.16 (2013-10-01) ----------------- * Fix Subversion status parsing in the presence of svn usernames longer than 12 characters (`issue #18 `__). 0.15 (2013-09-20) ----------------- * Normalize the paths of all files, avoiding some duplicate misses of directories. (`issue #16 `__). [maurits] 0.14 (2013-08-28) ----------------- * Supports packages that do not live in the root of a version control repository (`issue #15 `__). * More reliable svn support: detect files that have been added but not committed (or committed but not updated). * Licence changed from GPL (v2 or later) to MIT (`issue #12 `__). 0.13 (2013-07-31) ----------------- * New command line option: --ignore (`issue #11 `__). Contributed by Steven Myint. * New command line option: -p, --python. Defaults to the Python you used to run check-manifest. Fixes issues with packages that require Python 3 to run setup.py (`issue #13 `__). 0.12 (2013-05-15) ----------------- * Add suggestion pattern for `Makefile`. * More generic suggestion patterns, should cover almost anything. * zest.releaser_ integration: skip check-release for non-Python packages (`issue #9 `__). 0.11 (2013-03-20) ----------------- * Make sure ``MANIFEST.in`` is not ignored even if it hasn't been added to the VCS yet (`issue #7 `__). 0.10 (2013-03-17) ----------------- * ``check-manifest --version`` now prints the version number. * Don't apologize for not adding rules for directories (especially after adding rules that include files inside that directory). * Python 3 support contributed by Steven Myint. * Default ignore patterns can be configured in ``setup.cfg`` (`issue #3 `_). 0.9 (2013-03-06) ---------------- * Add suggestion pattern for `.travis.yml`. * When check-manifest -u (or -c) doesn't know how to write a rule matching a particular file, it now apologizes explicitly. * Copy the source tree to a temporary directory before running python setup.py sdist to avoid side effects from setuptools plugins or stale \*.egg-info/SOURCES.txt files (`issue #1 `_). * Warn if `*.egg-info` or `*.mo` is actually checked into the VCS. * Don't complain if `*.mo` files are present in the sdist but not in the VCS (`issue #2 `_). 0.8 (2013-03-06) ---------------- * Entry point for zest.releaser_. If you install both zest.releaser and check-manifest, you will be asked if you want to check your manifest during ``fullrelease``. .. _zest.releaser: https://pypi.python.org/pypi/zest.releaser 0.7 (2013-03-05) ---------------- * First release available from the Python Package Index. * Moved from https://gist.github.com/4277075 to https://github.com/mgedmin/check-manifest * Added README.rst, CHANGES.rst, setup.py, tox.ini (but no real tests yet), MANIFEST.in, and a Makefile. * Fixed a bug in error reporting (when setup.py failed, the user would get `TypeError: descriptor '__init__' requires an 'exceptions.Exception' object but received a 'str'`). check-manifest-0.36/LICENSE.rst000066400000000000000000000021131320501433100161240ustar00rootroot00000000000000The MIT License (MIT) Copyright (c) 2013 Marius Gedminas and contributors Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. check-manifest-0.36/MANIFEST.in000066400000000000000000000002141320501433100160460ustar00rootroot00000000000000include *.rst include *.py include tox.ini include Makefile include .gitignore include .travis.yml include appveyor.yml include .coveragerc check-manifest-0.36/Makefile000077500000000000000000000043651320501433100157660ustar00rootroot00000000000000PYTHON = python FILE_WITH_VERSION = check_manifest.py FILE_WITH_CHANGELOG = CHANGES.rst .PHONY: all all: @echo "Nothing to build. Try 'make test' perhaps?" .PHONY: check test test: detox check: SKIP_NO_TESTS=1 tox .PHONY: coverage coverage: tox -e coverage .PHONY: dist dist: $(PYTHON) setup.py -q sdist bdist_wheel .PHONY: distcheck distcheck: $(PYTHON) check_manifest.py # Bit of a chicken-and-egg here, but if the tree is unclean, make # distcheck will fail. @test -z "`git status -s 2>&1`" || { echo; echo "Your working tree is not clean" 1>&2; git status; exit 1; } make dist pkg_and_version=`$(PYTHON) setup.py --name`-`$(PYTHON) setup.py --version` && \ rm -rf tmp && \ mkdir tmp && \ git archive --format=tar --prefix=tmp/tree/ HEAD | tar -xf - && \ cd tmp && \ tar xvzf ../dist/$$pkg_and_version.tar.gz && \ diff -ur $$pkg_and_version tree -x PKG-INFO -x setup.cfg -x '*.egg-info' && \ cd $$pkg_and_version && \ make dist check && \ cd .. && \ mkdir one two && \ cd one && \ tar xvzf ../../dist/$$pkg_and_version.tar.gz && \ cd ../two/ && \ tar xvzf ../$$pkg_and_version/dist/$$pkg_and_version.tar.gz && \ cd .. && \ diff -ur one two -x SOURCES.txt && \ cd .. && \ rm -rf tmp && \ echo "sdist seems to be ok" .PHONY: releasechecklist releasechecklist: @$(PYTHON) setup.py --version | grep -qv dev || { \ echo "Please remove the 'dev' suffix from the version number in $(FILE_WITH_VERSION)"; exit 1; } @$(PYTHON) setup.py --long-description | rst2html --exit-status=2 > /dev/null @ver_and_date="`$(PYTHON) setup.py --version` (`date +%Y-%m-%d`)" && \ grep -q "^$$ver_and_date$$" $(FILE_WITH_CHANGELOG) || { \ echo "$(FILE_WITH_CHANGELOG) has no entry for $$ver_and_date"; exit 1; } make distcheck .PHONY: release release: releasechecklist # I'm chicken so I won't actually do these things yet @echo "Please run" @echo @echo " rm -rf dist && $(PYTHON) setup.py -q sdist bdist_wheel && twine upload dist/* && git tag `$(PYTHON) setup.py --version`" @echo @echo "Please increment the version number in $(FILE_WITH_VERSION)" @echo "and add a new empty entry at the top of the changelog in $(FILE_WITH_CHANGELOG), then" @echo @echo ' git commit -a -m "Post-release version bump" && git push && git push --tags' @echo check-manifest-0.36/README.rst000066400000000000000000000075221320501433100160100ustar00rootroot00000000000000check-manifest ============== |buildstatus|_ |appveyor|_ |coverage|_ Are you a Python developer? Have you uploaded packages to the Python Package Index? Have you accidentally uploaded *broken* packages with some files missing? If so, check-manifest is for you. Quick start ----------- :: $ pip install check-manifest $ cd ~/src/mygreatpackage $ check-manifest You can ask the script to help you update your MANIFEST.in:: $ check-manifest -u -v listing source files under version control: 6 files and directories building an sdist: check-manifest-0.7.tar.gz: 4 files and directories lists of files in version control and sdist do not match! missing from sdist: tests.py tox.ini suggested MANIFEST.in rules: include *.py include tox.ini updating MANIFEST.in $ cat MANIFEST.in include *.rst # added by check_manifest.py include *.py include tox.ini Command-line reference ---------------------- :: $ check-manifest --help usage: check-manifest [-h] [--version] [-v] [-c] [-u] [-p PYTHON] [--ignore patterns] [source_tree] Check a Python MANIFEST.in file for completeness positional arguments: source_tree location for the source tree (default: .) optional arguments: -h, --help show this help message and exit --version show program's version number and exit -v, --verbose more verbose output (default: False) -c, --create create a MANIFEST.in if missing (default: False) -u, --update append suggestions to MANIFEST.in (implies --create) (default: False) -p PYTHON, --python PYTHON use this Python interpreter for running setup.py sdist (default: /home/mg/.venv/bin/python) --ignore patterns ignore files/directories matching these comma- separated patterns (default: None) --ignore-bad-ideas patterns ignore bad idea files/directories matching these comma-separated patterns (default: []) Configuration ------------- You can tell check-manifest to ignore certain file patterns by adding a ``check-manifest`` section to your package's ``setup.cfg``. Example:: [check-manifest] ignore = .travis.yml The following options are recognized: ignore A list of newline separated filename patterns that will be ignored by check-manifest. Use this if you want to keep files in your version control system that shouldn't be included in your source distributions. The default ignore list is :: PKG-INFO *.egg-info *.egg-info/* setup.cfg .hgtags .hgsigs .hgignore .gitignore .bzrignore .gitattributes .travis.yml Jenkinsfile *.mo ignore-default-rules If set to ``true``, your ``ignore`` patterns will replace the default ignore list instead of adding to it. ignore-bad-ideas A list of newline separated filename patterns that will be ignored by check-manifest's generated files check. Use this if you want to keep generated files in your version control system, even though it is generally a bad idea. .. |buildstatus| image:: https://api.travis-ci.org/mgedmin/check-manifest.svg?branch=master .. _buildstatus: https://travis-ci.org/mgedmin/check-manifest .. |appveyor| image:: https://ci.appveyor.com/api/projects/status/github/mgedmin/check-manifest?branch=master&svg=true .. _appveyor: https://ci.appveyor.com/project/mgedmin/check-manifest .. |coverage| image:: https://coveralls.io/repos/mgedmin/check-manifest/badge.svg?branch=master .. _coverage: https://coveralls.io/r/mgedmin/check-manifest check-manifest-0.36/appveyor.yml000066400000000000000000000010531320501433100167020ustar00rootroot00000000000000version: build-{build}-{branch} environment: matrix: # https://www.appveyor.com/docs/installed-software#python lists available # versions - PYTHON: "C:\\Python27" - PYTHON: "C:\\Python33" - PYTHON: "C:\\Python34" - PYTHON: "C:\\Python35" - PYTHON: "C:\\Python36" init: - "echo %PYTHON%" install: - "set PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%" - python --version - pip install tox - choco install bzr - "set PATH=C:\\Program Files (x86)\\Bazaar;%PATH%" - bzr --version build: off test_script: - tox -e py check-manifest-0.36/check_manifest.py000077500000000000000000001041121320501433100176320ustar00rootroot00000000000000#!/usr/bin/env python """Check the MANIFEST.in file in a Python source package for completeness. This script works by building a source distribution archive (by running setup.py sdist), then checking the file list in the archive against the file list in version control (Subversion, Git, Mercurial, Bazaar are supported). Since the first check can fail to catch missing MANIFEST.in entries when you've got the right setuptools version control system support plugins installed, the script copies all the versioned files into a temporary directory and building the source distribution again. This also avoids issues with stale egg-info/SOURCES.txt files that may cause files not mentioned in MANIFEST.in to be included nevertheless. """ from __future__ import print_function import argparse import codecs import fnmatch import locale import os import posixpath import re import shutil import stat import subprocess import sys import tarfile import tempfile import unicodedata import zipfile from distutils.filelist import glob_to_re from contextlib import contextmanager, closing from distutils.text_file import TextFile from xml.etree import cElementTree as ET try: import ConfigParser except ImportError: # Python 3.x import configparser as ConfigParser __version__ = '0.36' __author__ = 'Marius Gedminas ' __licence__ = 'MIT' __url__ = 'https://github.com/mgedmin/check-manifest' class Failure(Exception): """An expected failure (as opposed to a bug in this script).""" # # User interface # VERBOSE = False _to_be_continued = False def _check_tbc(): global _to_be_continued if _to_be_continued: print() _to_be_continued = False def info(message): _check_tbc() print(message) def info_begin(message): if not VERBOSE: return _check_tbc() sys.stdout.write(message) sys.stdout.flush() global _to_be_continued _to_be_continued = True def info_continue(message): if not VERBOSE: return sys.stdout.write(message) sys.stdout.flush() global _to_be_continued _to_be_continued = True def info_end(message): if not VERBOSE: return print(message) global _to_be_continued _to_be_continued = False def error(message): _check_tbc() print(message, file=sys.stderr) def warning(message): _check_tbc() print(message, file=sys.stderr) def format_list(list_of_strings): return "\n".join(" " + s for s in list_of_strings) def format_missing(missing_from_a, missing_from_b, name_a, name_b): res = [] if missing_from_a: res.append("missing from %s:\n%s" % (name_a, format_list(sorted(missing_from_a)))) if missing_from_b: res.append("missing from %s:\n%s" % (name_b, format_list(sorted(missing_from_b)))) return '\n'.join(res) # # Filesystem/OS utilities # class CommandFailed(Failure): def __init__(self, command, status, output): Failure.__init__(self, "%s failed (status %s):\n%s" % ( command, status, output)) def run(command, encoding=None, decode=True, cwd=None): """Run a command [cmd, arg1, arg2, ...]. Returns the output (stdout + stderr). Raises CommandFailed in cases of error. """ if not encoding: encoding = locale.getpreferredencoding() try: with open(os.devnull, 'rb') as devnull: pipe = subprocess.Popen(command, stdin=devnull, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, cwd=cwd) except OSError as e: raise Failure("could not run %s: %s" % (command, e)) output = pipe.communicate()[0] if decode: output = output.decode(encoding) status = pipe.wait() if status != 0: raise CommandFailed(command, status, output) return output @contextmanager def cd(directory): """Change the current working directory, temporarily. Use as a context manager: with cd(d): ... """ old_dir = os.getcwd() try: os.chdir(directory) yield finally: os.chdir(old_dir) @contextmanager def mkdtemp(hint=''): """Create a temporary directory, then clean it up. Use as a context manager: with mkdtemp('-purpose'): ... """ dirname = tempfile.mkdtemp(prefix='check-manifest-', suffix=hint) try: yield dirname finally: rmtree(dirname) def chmod_plus(path, add_bits=stat.S_IWUSR): """Change a file's mode by adding a few bits. Like chmod + in a Unix shell. """ try: os.chmod(path, stat.S_IMODE(os.stat(path).st_mode) | add_bits) except OSError: # pragma: nocover pass # well, we tried def rmtree(path): """A version of rmtree that can deal with read-only files and directories. Needed because the stock shutil.rmtree() fails with an access error when there are read-only files in the directory on Windows, or when the directory itself is read-only on Unix. """ def onerror(func, path, exc_info): # Did you know what on Python 3.3 on Windows os.remove() and # os.unlink() are distinct functions? if func is os.remove or func is os.unlink or func is os.rmdir: if sys.platform != 'win32': chmod_plus(os.path.dirname(path), stat.S_IWUSR | stat.S_IXUSR) chmod_plus(path) func(path) else: raise shutil.rmtree(path, onerror=onerror) def copy_files(filelist, destdir): """Copy a list of files to destdir, preserving directory structure. File names should be relative to the current working directory. """ for filename in filelist: destfile = os.path.join(destdir, filename) # filename should not be absolute, but let's double-check assert destfile.startswith(destdir + os.path.sep) destfiledir = os.path.dirname(destfile) if not os.path.isdir(destfiledir): os.makedirs(destfiledir) if os.path.isdir(filename): os.mkdir(destfile) else: shutil.copy2(filename, destfile) def get_one_file_in(dirname): """Return the pathname of the one file in a directory. Raises if the directory has no files or more than one file. """ files = os.listdir(dirname) if len(files) > 1: raise Failure('More than one file exists in %s:\n%s' % (dirname, '\n'.join(sorted(files)))) elif not files: raise Failure('No files found in %s' % dirname) return os.path.join(dirname, files[0]) def unicodify(filename): """Make sure filename is Unicode. Because the tarfile module on Python 2 doesn't return Unicode. """ if isinstance(filename, bytes): return filename.decode(locale.getpreferredencoding()) else: return filename def get_archive_file_list(archive_filename): """Return the list of files in an archive. Supports .tar.gz and .zip. """ if archive_filename.endswith('.zip'): with closing(zipfile.ZipFile(archive_filename)) as zf: return add_directories(zf.namelist()) elif archive_filename.endswith(('.tar.gz', '.tar.bz2', '.tar')): with closing(tarfile.open(archive_filename)) as tf: return add_directories(list(map(unicodify, tf.getnames()))) else: ext = os.path.splitext(archive_filename)[-1] raise Failure('Unrecognized archive type: %s' % ext) def strip_toplevel_name(filelist): """Strip toplevel name from a file list. >>> strip_toplevel_name(['a', 'a/b', 'a/c', 'a/c/d']) ['b', 'c', 'c/d'] >>> strip_toplevel_name(['a/b', 'a/c', 'a/c/d']) ['b', 'c', 'c/d'] """ if not filelist: return filelist prefix = filelist[0] if '/' in prefix: prefix = prefix.partition('/')[0] + '/' names = filelist else: prefix = prefix + '/' names = filelist[1:] for name in names: if not name.startswith(prefix): raise Failure("File doesn't have the common prefix (%s): %s" % (name, prefix)) return [name[len(prefix):] for name in names] def add_prefix_to_each(prefix, filelist): """Add a prefix to each name in a file list. >>> filelist = add_prefix_to_each('foo/bar', ['a', 'b', 'c/d']) >>> [f.replace(os.path.sep, '/') for f in filelist] ['foo/bar/a', 'foo/bar/b', 'foo/bar/c/d'] """ return [posixpath.join(prefix, name) for name in filelist] class VCS(object): @classmethod def detect(cls, location): return os.path.isdir(os.path.join(location, cls.metadata_name)) class Git(VCS): metadata_name = '.git' # Git for Windows uses UTF-8 instead of the locale encoding. # Git on POSIX systems uses the locale encoding. _encoding = 'UTF-8' if sys.platform == 'win32' else None @classmethod def detect(cls, location): # .git can be a file for submodules return os.path.exists(os.path.join(location, cls.metadata_name)) @classmethod def get_versioned_files(cls): """List all files versioned by git in the current directory.""" files = cls._git_ls_files() submodules = cls._list_submodules() for subdir in submodules: subdir = os.path.relpath(subdir).replace(os.path.sep, '/') files += add_prefix_to_each(subdir, cls._git_ls_files(subdir)) return add_directories(files) @classmethod def _git_ls_files(cls, cwd=None): output = run(['git', 'ls-files', '-z'], encoding=cls._encoding, cwd=cwd) return output.split('\0')[:-1] @classmethod def _list_submodules(cls): # This is incredibly expensive on my Jenkins instance (50 seconds for # each invocation, even when there are no submodules whatsoever). # Curiously, I cannot reproduce that in Appveyor, or even on the same # Jenkins machine but when I run the tests manually. Still, 2-hour # Jenkins runs are bad, so let's avoid running 'git submodule' when # there's no .gitmodules file. if not os.path.exists('.gitmodules'): return [] return run(['git', 'submodule', '--quiet', 'foreach', '--recursive', 'printf "%s/%s\\n" $toplevel $path'], encoding=cls._encoding).splitlines() class Mercurial(VCS): metadata_name = '.hg' @staticmethod def get_versioned_files(): """List all files under Mercurial control in the current directory.""" output = run(['hg', 'status', '-ncamd', '.']) return add_directories(output.splitlines()) class Bazaar(VCS): metadata_name = '.bzr' @classmethod def _get_terminal_encoding(self): # Python 3.6 lets us name the OEM codepage directly, which is lucky # because it also breaks our old method of OEM codepage detection # (PEP-528 changed sys.stdout.encoding to UTF-8). try: codecs.lookup('oem') except LookupError: pass else: # pragma: nocover return 'oem' # Based on bzrlib.osutils.get_terminal_encoding() encoding = getattr(sys.stdout, 'encoding', None) if not encoding: encoding = getattr(sys.stdin, 'encoding', None) if encoding == 'cp0': # "no codepage" encoding = None # NB: bzrlib falls back on bzrlib.osutils.get_user_encoding(), # which is like locale.getpreferredencoding() on steroids, and # also includes a fallback from 'ascii' to 'utf-8' when # sys.platform is 'darwin'. This is probably something we might # want to do in run(), but I'll wait for somebody to complain # first, since I don't have a Mac OS X machine and cannot test. return encoding @classmethod def get_versioned_files(cls): """List all files versioned in Bazaar in the current directory.""" encoding = cls._get_terminal_encoding() output = run(['bzr', 'ls', '-VR'], encoding=encoding) return output.splitlines() class Subversion(VCS): metadata_name = '.svn' @classmethod def get_versioned_files(cls): """List all files under SVN control in the current directory.""" output = run(['svn', 'st', '-vq', '--xml'], decode=False) tree = ET.XML(output) return sorted(entry.get('path') for entry in tree.findall('.//entry') if cls.is_interesting(entry)) @staticmethod def is_interesting(entry): """Is this entry interesting? ``entry`` is an XML node representing one entry of the svn status XML output. It looks like this:: mg 2015-02-06T07:52:38.163516Z """ if entry.get('path') == '.': return False status = entry.find('wc-status') if status is None: warning('svn status --xml parse error: without' ' ' % entry.get('path')) return False # For SVN externals we get two entries: one mentioning the # existence of the external, and one about the status of the external. if status.get('item') in ('unversioned', 'external'): return False return True def detect_vcs(): """Detect the version control system used for the current directory.""" location = os.path.abspath('.') while True: for vcs in Git, Mercurial, Bazaar, Subversion: if vcs.detect(location): return vcs parent = os.path.dirname(location) if parent == location: raise Failure("Couldn't find version control data" " (git/hg/bzr/svn supported)") location = parent def get_vcs_files(): """List all files under version control in the current directory.""" vcs = detect_vcs() return normalize_names(vcs.get_versioned_files()) def normalize_names(names): """Normalize file names.""" return list(map(normalize_name, names)) def normalize_name(name): """Some VCS print directory names with trailing slashes. Strip them. Easiest is to normalize the path. And encodings may trip us up too, especially when comparing lists of files. Plus maybe lowercase versus uppercase. """ name = os.path.normpath(name) name = unicodify(name) if sys.platform == 'darwin': # Mac OSX may have problems comparing non-ascii filenames, so # we convert them. name = unicodedata.normalize('NFC', name) return name def add_directories(names): """Git/Mercurial/zip files omit directories, let's add them back.""" res = list(names) seen = set(names) for name in names: while True: name = os.path.dirname(name) if not name or name in seen: break res.append(name) seen.add(name) return sorted(res) # # Packaging logic # # it's fine if any of these are missing in the VCS or in the sdist IGNORE = [ 'PKG-INFO', # always generated '*.egg-info', # always generated '*.egg-info/*', # always generated 'setup.cfg', # always generated, sometimes also kept in source control # it's not a problem if the sdist is lacking these files: '.hgtags', '.hgsigs', '.hgignore', '.gitignore', '.bzrignore', '.gitattributes', '.travis.yml', 'Jenkinsfile', # it's convenient to ship compiled .mo files in sdists, but they shouldn't # be checked in '*.mo', ] IGNORE_REGEXPS = [ # Regular expressions for filename to ignore. This is useful for # filename patterns where the '*' part must not search in # directories. ] WARN_ABOUT_FILES_IN_VCS = [ # generated files should not be committed into the VCS 'PKG-INFO', '*.egg-info', '*.mo', '*.py[co]', '*.so', '*.pyd', '*~', '.*.sw[po]', '.#*', ] IGNORE_BAD_IDEAS = [] _sep = r'\\' if os.path.sep == '\\' else os.path.sep SUGGESTIONS = [(re.compile(pattern.replace('/', _sep)), suggestion) for pattern, suggestion in [ # regexp -> suggestion ('^([^/]+[.](cfg|ini))$', r'include \1'), ('^([.]travis[.]yml)$', r'include \1'), ('^([.]coveragerc)$', r'include \1'), ('^([A-Z]+)$', r'include \1'), ('^(Makefile)$', r'include \1'), ('^[^/]+[.](txt|rst|py)$', r'include *.\1'), ('^([a-zA-Z_][a-zA-Z_0-9]*)/' '.*[.](py|zcml|pt|mako|xml|html|txt|rst|css|png|jpg|dot|po|pot|mo|ui|desktop|bat)$', r'recursive-include \1 *.\2'), ('^([a-zA-Z_][a-zA-Z_0-9]*)(?:/.*)?/(Makefile)$', r'recursive-include \1 \2'), # catch-all rules that actually cover some of the above; somewhat # experimental: I fear false positives ('^([a-zA-Z_0-9]+)$', r'include \1'), ('^[^/]+[.]([a-zA-Z_0-9]+)$', r'include *.\1'), ('^([a-zA-Z_][a-zA-Z_0-9]*)/.*[.]([a-zA-Z_0-9]+)$', r'recursive-include \1 *.\2'), ]] CFG_SECTION_CHECK_MANIFEST = 'check-manifest' CFG_IGNORE_DEFAULT_RULES = (CFG_SECTION_CHECK_MANIFEST, 'ignore-default-rules') CFG_IGNORE = (CFG_SECTION_CHECK_MANIFEST, 'ignore') CFG_IGNORE_BAD_IDEAS = (CFG_SECTION_CHECK_MANIFEST, 'ignore-bad-ideas') def read_config(): """Read configuration from setup.cfg.""" # XXX modifies global state, which is kind of evil config = ConfigParser.ConfigParser() config.read(['setup.cfg']) if not config.has_section(CFG_SECTION_CHECK_MANIFEST): return if (config.has_option(*CFG_IGNORE_DEFAULT_RULES) and config.getboolean(*CFG_IGNORE_DEFAULT_RULES)): del IGNORE[:] if config.has_option(*CFG_IGNORE): patterns = [p.strip() for p in config.get(*CFG_IGNORE).splitlines()] IGNORE.extend(p for p in patterns if p) if config.has_option(*CFG_IGNORE_BAD_IDEAS): lines = config.get(*CFG_IGNORE_BAD_IDEAS).splitlines() patterns = [p.strip() for p in lines] IGNORE_BAD_IDEAS.extend(p for p in patterns if p) def read_manifest(): """Read existing configuration from MANIFEST.in. We use that to ignore anything the MANIFEST.in ignores. """ # XXX modifies global state, which is kind of evil if not os.path.isfile('MANIFEST.in'): return ignore, ignore_regexps = _get_ignore_from_manifest('MANIFEST.in') IGNORE.extend(ignore) IGNORE_REGEXPS.extend(ignore_regexps) def _glob_to_regexp(pat): """Compile a glob pattern into a regexp. We need to do this because fnmatch allows * to match /, which we don't want. E.g. a MANIFEST.in exclude of 'dirname/*css' should match 'dirname/foo.css' but not 'dirname/subdir/bar.css'. """ return glob_to_re(pat) def _get_ignore_from_manifest(filename): """Gather the various ignore patterns from a MANIFEST.in. Returns a list of standard ignore patterns and a list of regular expressions to ignore. """ class MyTextFile(TextFile): def error(self, msg, line=None): # pragma: nocover # (this is never called by TextFile in current versions of CPython) raise Failure(self.gen_error(msg, line)) def warn(self, msg, line=None): warning(self.gen_error(msg, line)) template = MyTextFile(filename, strip_comments=True, skip_blanks=True, join_lines=True, lstrip_ws=True, rstrip_ws=True, collapse_join=True) try: lines = template.readlines() finally: template.close() return _get_ignore_from_manifest_lines(lines) def _get_ignore_from_manifest_lines(lines): """Gather the various ignore patterns from a MANIFEST.in. 'lines' should be a list of strings with comments removed and continuation lines joined. Returns a list of standard ignore patterns and a list of regular expressions to ignore. """ ignore = [] ignore_regexps = [] for line in lines: try: cmd, rest = line.split(None, 1) except ValueError: # no whitespace, so not interesting continue for part in rest.split(): # distutils enforces these warnings on Windows only if part.startswith('/'): warning("ERROR: Leading slashes are not allowed in MANIFEST.in on Windows: %s" % part) if part.endswith('/'): warning("ERROR: Trailing slashes are not allowed in MANIFEST.in on Windows: %s" % part) if cmd == 'exclude': # An exclude of 'dirname/*css' can match 'dirname/foo.css' # but not 'dirname/subdir/bar.css'. We need a regular # expression for that, since fnmatch doesn't pay attention to # directory separators. for pat in rest.split(): if '*' in pat or '?' in pat or '[!' in pat: ignore_regexps.append(_glob_to_regexp(pat)) else: # No need for special handling. ignore.append(pat) elif cmd == 'global-exclude': ignore.extend(rest.split()) elif cmd == 'recursive-exclude': try: dirname, patterns = rest.split(None, 1) except ValueError: # Wrong MANIFEST.in line. warning("You have a wrong line in MANIFEST.in: %r\n" "'recursive-exclude' expects " " ..." % line) continue # Strip path separator for clarity. dirname = dirname.rstrip(os.path.sep) for pattern in patterns.split(): if pattern.startswith('*'): ignore.append(dirname + os.path.sep + pattern) else: # 'recursive-exclude plone metadata.xml' should # exclude plone/metadata.xml and # plone/*/metadata.xml, where * can be any number # of sub directories. We could use a regexp, but # two ignores seems easier. ignore.append(dirname + os.path.sep + pattern) ignore.append(dirname + os.path.sep + '*' + os.path.sep + pattern) elif cmd == 'prune': # rest is considered to be a directory name. It should # not contain a path separator, as it actually has no # effect in that case, but that could differ per python # version. We strip it here to avoid double separators. # XXX: mg: I'm not 100% sure the above is correct, AFAICS # all pythons from 2.6 complain if the path has a leading or # trailing slash -- on Windows, that is. rest = rest.rstrip('/\\') ignore.append(rest) ignore.append(rest + os.path.sep + '*') return ignore, ignore_regexps def file_matches(filename, patterns): """Does this filename match any of the patterns?""" return any(fnmatch.fnmatch(filename, pat) or fnmatch.fnmatch(os.path.basename(filename), pat) for pat in patterns) def file_matches_regexps(filename, patterns): """Does this filename match any of the regular expressions?""" return any(re.match(pat, filename) for pat in patterns) def strip_sdist_extras(filelist): """Strip generated files that are only present in source distributions. We also strip files that are ignored for other reasons, like command line arguments, setup.cfg rules or MANIFEST.in rules. """ return [name for name in filelist if not file_matches(name, IGNORE) and not file_matches_regexps(name, IGNORE_REGEXPS)] def find_bad_ideas(filelist): """Find files matching WARN_ABOUT_FILES_IN_VCS patterns.""" return [name for name in filelist if file_matches(name, WARN_ABOUT_FILES_IN_VCS)] def find_suggestions(filelist): """Suggest MANIFEST.in patterns for missing files.""" suggestions = set() unknowns = [] for filename in filelist: if os.path.isdir(filename): # it's impossible to add empty directories via MANIFEST.in anyway, # and non-empty directories will be added automatically when we # specify patterns for files inside them continue for pattern, suggestion in SUGGESTIONS: m = pattern.match(filename) if m is not None: suggestions.add(pattern.sub(suggestion, filename)) break else: unknowns.append(filename) return sorted(suggestions), unknowns def is_package(source_tree='.'): """Is the directory the root of a Python package? Note: the term "package" here refers to a collection of files with a setup.py, not to a directory with an __init__.py. """ return os.path.exists(os.path.join(source_tree, 'setup.py')) def check_manifest(source_tree='.', create=False, update=False, python=sys.executable): """Compare a generated source distribution with list of files in a VCS. Returns True if the manifest is fine. """ all_ok = True if os.path.sep in python: python = os.path.abspath(python) with cd(source_tree): if not is_package(): raise Failure('This is not a Python project (no setup.py).') read_config() read_manifest() info_begin("listing source files under version control") all_source_files = sorted(get_vcs_files()) source_files = strip_sdist_extras(all_source_files) info_continue(": %d files and directories" % len(source_files)) if not all_source_files: raise Failure('There are no files added to version control!') info_begin("building an sdist") with mkdtemp('-sdist') as tempdir: run([python, 'setup.py', 'sdist', '-d', tempdir]) sdist_filename = get_one_file_in(tempdir) info_continue(": %s" % os.path.basename(sdist_filename)) sdist_files = sorted(normalize_names(strip_sdist_extras( strip_toplevel_name(get_archive_file_list(sdist_filename))))) info_continue(": %d files and directories" % len(sdist_files)) existing_source_files = list(filter(os.path.exists, all_source_files)) missing_source_files = sorted(set(all_source_files) - set(existing_source_files)) if missing_source_files: warning("some files listed as being under source control are missing:\n%s" % format_list(missing_source_files)) info_begin("copying source files to a temporary directory") with mkdtemp('-sources') as tempsourcedir: copy_files(existing_source_files, tempsourcedir) if os.path.exists('MANIFEST.in') and 'MANIFEST.in' not in source_files: # See https://github.com/mgedmin/check-manifest/issues/7 # if do this, we will emit a warning about MANIFEST.in not # being in source control, if we don't do this, the user # gets confused about their new manifest rules being # ignored. copy_files(['MANIFEST.in'], tempsourcedir) if 'setup.py' not in source_files: # See https://github.com/mgedmin/check-manifest/issues/46 # if do this, we will emit a warning about setup.py not # being in source control, if we don't do this, the user # gets a scary error copy_files(['setup.py'], tempsourcedir) info_begin("building a clean sdist") with cd(tempsourcedir): with mkdtemp('-sdist') as tempdir: run([python, 'setup.py', 'sdist', '-d', tempdir]) sdist_filename = get_one_file_in(tempdir) info_continue(": %s" % os.path.basename(sdist_filename)) clean_sdist_files = sorted(normalize_names(strip_sdist_extras( strip_toplevel_name(get_archive_file_list(sdist_filename))))) info_continue(": %d files and directories" % len(clean_sdist_files)) missing_from_manifest = set(source_files) - set(clean_sdist_files) missing_from_VCS = set(sdist_files + clean_sdist_files) - set(source_files) if not missing_from_manifest and not missing_from_VCS: info("lists of files in version control and sdist match") else: error("lists of files in version control and sdist do not match!\n%s" % format_missing(missing_from_VCS, missing_from_manifest, "VCS", "sdist")) suggestions, unknowns = find_suggestions(missing_from_manifest) user_asked_for_help = update or (create and not os.path.exists('MANIFEST.in')) if 'MANIFEST.in' not in existing_source_files: if suggestions and not user_asked_for_help: info("no MANIFEST.in found; you can run 'check-manifest -c' to create one") else: info("no MANIFEST.in found") if suggestions: info("suggested MANIFEST.in rules:\n%s" % format_list(suggestions)) if user_asked_for_help: existed = os.path.exists('MANIFEST.in') with open('MANIFEST.in', 'a') as f: if not existed: info("creating MANIFEST.in") else: info("updating MANIFEST.in") f.write('\n# added by check_manifest.py\n') f.write('\n'.join(suggestions) + '\n') if unknowns: info("don't know how to come up with rules matching\n%s" % format_list(unknowns)) elif user_asked_for_help: info("don't know how to come up with rules" " matching any of the files, sorry!") all_ok = False bad_ideas = find_bad_ideas(all_source_files) filtered_bad_ideas = [bad_idea for bad_idea in bad_ideas if not file_matches(bad_idea, IGNORE_BAD_IDEAS)] if filtered_bad_ideas: warning("you have %s in source control!\nthat's a bad idea:" " auto-generated files should not be versioned" % filtered_bad_ideas[0]) if len(filtered_bad_ideas) > 1: warning("this also applies to the following:\n%s" % format_list(filtered_bad_ideas[1:])) all_ok = False return all_ok # # Main script # def main(): parser = argparse.ArgumentParser( description="Check a Python MANIFEST.in file for completeness", formatter_class=argparse.ArgumentDefaultsHelpFormatter) parser.add_argument('source_tree', default='.', nargs='?', help='location for the source tree') parser.add_argument('--version', action='version', version='%(prog)s version ' + __version__) parser.add_argument('-v', '--verbose', action='store_true', help='more verbose output') parser.add_argument('-c', '--create', action='store_true', help='create a MANIFEST.in if missing') parser.add_argument('-u', '--update', action='store_true', help='append suggestions to MANIFEST.in (implies --create)') parser.add_argument('-p', '--python', default=sys.executable, help='use this Python interpreter for running setup.py sdist') parser.add_argument('--ignore', metavar='patterns', default=None, help='ignore files/directories matching these' ' comma-separated patterns') parser.add_argument('--ignore-bad-ideas', metavar='patterns', default=[], help='ignore bad idea files/directories ' 'matching these comma-separated patterns') args = parser.parse_args() if args.ignore: IGNORE.extend(args.ignore.split(',')) if args.ignore_bad_ideas: IGNORE_BAD_IDEAS.extend(args.ignore_bad_ideas.split(',')) if args.verbose: global VERBOSE VERBOSE = True try: if not check_manifest(args.source_tree, create=args.create, update=args.update, python=args.python): sys.exit(1) except Failure as e: error(str(e)) sys.exit(2) # # zest.releaser integration # def zest_releaser_check(data): """Check the completeness of MANIFEST.in before the release. This is an entry point for zest.releaser. See the documentation at https://zestreleaser.readthedocs.io/en/latest/entrypoints.html """ from zest.releaser.utils import ask source_tree = data['workingdir'] if not is_package(source_tree): # You can use zest.releaser on things that are not Python packages. # It's pointless to run check-manifest in those circumstances. # See https://github.com/mgedmin/check-manifest/issues/9 for details. return if not ask("Do you want to run check-manifest?"): return try: if not check_manifest(source_tree): if not ask("MANIFEST.in has problems. " " Do you want to continue despite that?", default=False): sys.exit(1) except Failure as e: error(str(e)) if not ask("Something bad happened. " " Do you want to continue despite that?", default=False): sys.exit(2) if __name__ == '__main__': main() check-manifest-0.36/setup.cfg000066400000000000000000000012051320501433100161320ustar00rootroot00000000000000[bdist_wheel] universal = 1 [nosetests] with-doctest = 1 exe = 1 [pytest] norecursedirs = dist build tmp .* *.egg-info python_files = tests.py check_manifest.py addopts = --doctest-modules --ignore=setup.py [zest.releaser] python-file-with-version = check_manifest.py [flake8] ignore=E241,E501,E261,E126,E127,E128,E302 # E241: multiple spaces after ',' # E501: line too long # E261: at least two spaces before inline comment # E126: continuation line over-indented for hanging indent # E127: continuation line over-indented for visual indent # E128: continuation line under-indented for visual indent # E302: expected 2 blank lines, found 0 check-manifest-0.36/setup.py000077500000000000000000000052511320501433100160330ustar00rootroot00000000000000#!/usr/bin/env python import os, re, ast, email.utils, sys from setuptools import setup if sys.version_info < (2, 7): sys.exit("Python 2.7 or newer is required for check-manifest") if (3, 0) <= sys.version_info < (3, 3): sys.exit("Python 3.3 or newer is required for check-manifest") here = os.path.dirname(__file__) with open(os.path.join(here, 'README.rst')) as readme: with open(os.path.join(here, 'CHANGES.rst')) as changelog: long_description = readme.read() + '\n\n' + changelog.read() metadata = {} with open(os.path.join(here, 'check_manifest.py')) as f: rx = re.compile('(__version__|__author__|__url__|__licence__) = (.*)') for line in f: m = rx.match(line) if m: metadata[m.group(1)] = ast.literal_eval(m.group(2)) version = metadata['__version__'] author, author_email = email.utils.parseaddr(metadata['__author__']) url = metadata['__url__'] licence = metadata['__licence__'] setup( name='check-manifest', version=version, author=author, author_email=author_email, url=url, description='Check MANIFEST.in in a Python source package for completeness', long_description=long_description, keywords=['distutils', 'setuptools', 'packaging', 'manifest', 'checker', 'linter'], classifiers=[ 'Development Status :: 4 - Beta', ## 'Development Status :: 5 - Production/Stable', eventually... 'Environment :: Console', 'Intended Audience :: Developers', 'License :: OSI Approved :: GNU General Public License (GPL)' if licence.startswith('GPL') else 'License :: OSI Approved :: MIT License' if licence.startswith('MIT') else 'License :: uhh, dunno', # fail PyPI upload intentionally until fixed 'Operating System :: OS Independent', 'Programming Language :: Python', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3.3', 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: Implementation :: CPython', 'Programming Language :: Python :: Implementation :: PyPy', ], license=licence, py_modules=['check_manifest'], zip_safe=False, test_suite='tests.test_suite', install_requires=[], extras_require={ 'test': ['mock'], }, tests_require=['mock'], entry_points={ 'console_scripts': [ 'check-manifest = check_manifest:main', ], 'zest.releaser.prereleaser.before': [ 'check-manifest = check_manifest:zest_releaser_check', ], }, ) check-manifest-0.36/tests.py000066400000000000000000001622541320501433100160410ustar00rootroot00000000000000import codecs import doctest import locale import os import subprocess import sys import tarfile import tempfile import textwrap import unittest import zipfile from contextlib import closing from io import BytesIO from xml.etree import cElementTree as ET try: from cStringIO import StringIO # Python 2.x except ImportError: from io import StringIO # Python 3.x import mock from check_manifest import rmtree CAN_SKIP_TESTS = os.getenv('SKIP_NO_TESTS', '') == '' try: codecs.lookup('oem') except LookupError: HAS_OEM_CODEC = False else: # Python >= 3.6 on Windows HAS_OEM_CODEC = True class Tests(unittest.TestCase): def setUp(self): import check_manifest self.warnings = [] self._real_warning = check_manifest.warning check_manifest.warning = self.warnings.append def tearDown(self): import check_manifest check_manifest.warning = self._real_warning def make_temp_dir(self): tmpdir = tempfile.mkdtemp(prefix='test-', suffix='-check-manifest') self.addCleanup(rmtree, tmpdir) return tmpdir def create_file(self, filename, contents): with open(filename, 'w') as f: f.write(contents) def create_zip_file(self, filename, filenames): with closing(zipfile.ZipFile(filename, 'w')) as zf: for fn in filenames: zf.writestr(fn, '') def create_tar_file(self, filename, filenames): with closing(tarfile.TarFile(filename, 'w')) as tf: for fn in filenames: tf.addfile(tarfile.TarInfo(fn), BytesIO()) def test_run_success(self): from check_manifest import run self.assertEqual(run(["true"]), "") def test_run_failure(self): from check_manifest import run, CommandFailed with self.assertRaises(CommandFailed) as cm: run(["false"]) self.assertEqual(str(cm.exception), "['false'] failed (status 1):\n") def test_run_no_such_program(self): from check_manifest import run, Failure with self.assertRaises(Failure) as cm: run(["there-is-really-no-such-program"]) # Linux says "[Errno 2] No such file or directory" # Windows says "[Error 2] The system cannot find the file specified" # but on 3.x it's "[WinErr 2] The system cannot find the file specified" should_start_with = "could not run ['there-is-really-no-such-program']:" self.assertTrue( str(cm.exception).startswith(should_start_with), '\n%r does not start with\n%r' % (str(cm.exception), should_start_with)) def test_mkdtemp_readonly_files(self): from check_manifest import mkdtemp with mkdtemp(hint='-test-readonly') as d: fn = os.path.join(d, 'file.txt') with open(fn, 'w'): pass os.chmod(fn, 0o444) # readonly assert not os.path.exists(d) @unittest.skipIf(sys.platform == 'win32', "No POSIX-like unreadable directories on Windows") def test_rmtree_unreadable_directories(self): d = self.make_temp_dir() sd = os.path.join(d, 'subdir') os.mkdir(sd) os.chmod(sd, 0) # a bad mode for a directory, oops # The onerror API of shutil.rmtree doesn't let us recover from # os.listdir() failures. with self.assertRaises(OSError): rmtree(sd) os.chmod(sd, 0o755) # so we can clean up def test_rmtree_readonly_directories(self): d = self.make_temp_dir() sd = os.path.join(d, 'subdir') fn = os.path.join(sd, 'file.txt') os.mkdir(sd) open(fn, 'w').close() os.chmod(sd, 0o444) # a bad mode for a directory, oops rmtree(sd) assert not os.path.exists(sd) def test_rmtree_readonly_directories_and_files(self): d = self.make_temp_dir() sd = os.path.join(d, 'subdir') fn = os.path.join(sd, 'file.txt') os.mkdir(sd) open(fn, 'w').close() os.chmod(fn, 0o444) # readonly os.chmod(sd, 0o444) # a bad mode for a directory, oops rmtree(sd) assert not os.path.exists(sd) def test_copy_files(self): from check_manifest import copy_files actions = [] n = os.path.normpath with mock.patch('os.path.isdir', lambda d: d in ('b', n('/dest/dir'))): with mock.patch('os.makedirs', lambda d: actions.append('makedirs %s' % d)): with mock.patch('os.mkdir', lambda d: actions.append('mkdir %s' % d)): with mock.patch('shutil.copy2', lambda s, d: actions.append('cp %s %s' % (s, d))): copy_files(['a', 'b', n('c/d/e')], n('/dest/dir')) self.assertEqual( actions, [ 'cp a %s' % n('/dest/dir/a'), 'mkdir %s' % n('/dest/dir/b'), 'makedirs %s' % n('/dest/dir/c/d'), 'cp %s %s' % (n('c/d/e'), n('/dest/dir/c/d/e')), ]) def test_get_one_file_in(self): from check_manifest import get_one_file_in with mock.patch('os.listdir', lambda dir: ['a']): self.assertEqual(get_one_file_in(os.path.normpath('/some/dir')), os.path.normpath('/some/dir/a')) def test_get_one_file_in_empty_directory(self): from check_manifest import get_one_file_in, Failure with mock.patch('os.listdir', lambda dir: []): with self.assertRaises(Failure) as cm: get_one_file_in('/some/dir') self.assertEqual(str(cm.exception), "No files found in /some/dir") def test_get_one_file_in_too_many(self): from check_manifest import get_one_file_in, Failure with mock.patch('os.listdir', lambda dir: ['b', 'a']): with self.assertRaises(Failure) as cm: get_one_file_in('/some/dir') self.assertEqual(str(cm.exception), "More than one file exists in /some/dir:\na\nb") def test_unicodify(self): from check_manifest import unicodify nonascii = b'\xc3\xa9.txt'.decode('UTF-8') # because Py3.2 lacks u'' self.assertEqual(unicodify(nonascii), nonascii) self.assertEqual( unicodify(nonascii.encode(locale.getpreferredencoding())), nonascii) def test_get_archive_file_list_unrecognized_archive(self): from check_manifest import get_archive_file_list, Failure with self.assertRaises(Failure) as cm: get_archive_file_list('archive.rar') self.assertEqual(str(cm.exception), 'Unrecognized archive type: .rar') def test_get_archive_file_list_zip(self): from check_manifest import get_archive_file_list filename = os.path.join(self.make_temp_dir(), 'archive.zip') self.create_zip_file(filename, ['a', 'b/c']) self.assertEqual(get_archive_file_list(filename), ['a', 'b', 'b/c']) def test_get_archive_file_list_zip_nonascii(self): from check_manifest import get_archive_file_list filename = os.path.join(self.make_temp_dir(), 'archive.zip') nonascii = b'\xc3\xa9.txt'.decode('UTF-8') # because Py3.2 lacks u'' self.create_zip_file(filename, [nonascii]) self.assertEqual(get_archive_file_list(filename), [nonascii]) def test_get_archive_file_list_tar(self): from check_manifest import get_archive_file_list filename = os.path.join(self.make_temp_dir(), 'archive.tar') self.create_tar_file(filename, ['a', 'b/c']) self.assertEqual(get_archive_file_list(filename), ['a', 'b', 'b/c']) def test_get_archive_file_list_tar_nonascii(self): from check_manifest import get_archive_file_list filename = os.path.join(self.make_temp_dir(), 'archive.tar') nonascii = b'\xc3\xa9.txt'.decode('UTF-8') # because Py3.2 lacks u'' self.create_tar_file(filename, [nonascii]) self.assertEqual(get_archive_file_list(filename), [nonascii]) def test_format_list(self): from check_manifest import format_list self.assertEqual(format_list([]), "") self.assertEqual(format_list(['a']), " a") self.assertEqual(format_list(['a', 'b']), " a\n b") def test_format_missing(self): from check_manifest import format_missing self.assertEqual( format_missing(set(), set(), "1st", "2nd"), "") self.assertEqual( format_missing(set(["c"]), set(["a"]), "1st", "2nd"), "missing from 1st:\n" " c\n" "missing from 2nd:\n" " a") def test_strip_toplevel_name_empty_list(self): from check_manifest import strip_toplevel_name self.assertEqual(strip_toplevel_name([]), []) def test_strip_toplevel_name_no_common_prefix(self): from check_manifest import strip_toplevel_name, Failure self.assertRaises(Failure, strip_toplevel_name, ["a/b", "c/d"]) def test_detect_vcs_no_vcs(self): from check_manifest import detect_vcs, Failure with mock.patch('check_manifest.VCS.detect', staticmethod(lambda *a: False)): with mock.patch('check_manifest.Git.detect', staticmethod(lambda *a: False)): with self.assertRaises(Failure) as cm: detect_vcs() self.assertEqual(str(cm.exception), "Couldn't find version control data" " (git/hg/bzr/svn supported)") def test_normalize_names(self): from check_manifest import normalize_names j = os.path.join self.assertEqual(normalize_names(["a", j("b", ""), j("c", "d"), j("e", "f", ""), j("g", "h", "..", "i")]), ["a", "b", j("c", "d"), j("e", "f"), j("g", "i")]) def test_add_directories(self): from check_manifest import add_directories j = os.path.join self.assertEqual(add_directories(['a', 'b', j('c', 'd'), j('e', 'f')]), ['a', 'b', 'c', j('c', 'd'), 'e', j('e', 'f')]) def test_file_matches(self): from check_manifest import file_matches # On Windows we might get the pattern list from setup.cfg using / as # the directory separator, but the filenames we're matching against # will use os.path.sep patterns = ['setup.cfg', '*.egg-info', '*.egg-info/*'] j = os.path.join self.assertFalse(file_matches('setup.py', patterns)) self.assertTrue(file_matches('setup.cfg', patterns)) self.assertTrue(file_matches(j('src', 'zope.foo.egg-info'), patterns)) self.assertTrue( file_matches(j('src', 'zope.foo.egg-info', 'SOURCES.txt'), patterns)) def test_strip_sdist_extras(self): from check_manifest import strip_sdist_extras filelist = list(map(os.path.normpath, [ '.gitignore', '.travis.yml', 'setup.py', 'setup.cfg', 'README.txt', 'src', 'src/.gitignore', 'src/zope', 'src/zope/__init__.py', 'src/zope/foo', 'src/zope/foo/__init__.py', 'src/zope/foo/language.po', 'src/zope/foo/language.mo', 'src/zope.foo.egg-info', 'src/zope.foo.egg-info/SOURCES.txt', ])) expected = list(map(os.path.normpath, [ 'setup.py', 'README.txt', 'src', 'src/zope', 'src/zope/__init__.py', 'src/zope/foo', 'src/zope/foo/__init__.py', 'src/zope/foo/language.po', ])) self.assertEqual(strip_sdist_extras(filelist), expected) def test_strip_sdist_extras_with_manifest(self): import check_manifest from check_manifest import strip_sdist_extras from check_manifest import _get_ignore_from_manifest_lines as parse orig_ignore = check_manifest.IGNORE[:] orig_ignore_regexps = check_manifest.IGNORE_REGEXPS[:] manifest_in = textwrap.dedent(""" graft src exclude *.cfg global-exclude *.mo prune src/dump recursive-exclude src/zope *.sh """) filelist = list(map(os.path.normpath, [ '.gitignore', 'setup.py', 'setup.cfg', 'MANIFEST.in', 'README.txt', 'src', 'src/helper.sh', 'src/dump', 'src/dump/__init__.py', 'src/zope', 'src/zope/__init__.py', 'src/zope/zopehelper.sh', 'src/zope/foo', 'src/zope/foo/__init__.py', 'src/zope/foo/language.po', 'src/zope/foo/language.mo', 'src/zope/foo/config.cfg', 'src/zope/foo/foohelper.sh', 'src/zope.foo.egg-info', 'src/zope.foo.egg-info/SOURCES.txt', ])) expected = list(map(os.path.normpath, [ 'setup.py', 'MANIFEST.in', 'README.txt', 'src', 'src/helper.sh', 'src/zope', 'src/zope/__init__.py', 'src/zope/foo', 'src/zope/foo/__init__.py', 'src/zope/foo/language.po', 'src/zope/foo/config.cfg', ])) # This will change the definitions. try: # This is normally done in read_manifest: ignore, ignore_regexps = parse(manifest_in.splitlines()) check_manifest.IGNORE.extend(ignore) check_manifest.IGNORE_REGEXPS.extend(ignore_regexps) # Filter the file list. result = strip_sdist_extras(filelist) finally: # Restore the original definitions check_manifest.IGNORE[:] = orig_ignore check_manifest.IGNORE_REGEXPS[:] = orig_ignore_regexps self.assertEqual(result, expected) def test_find_bad_ideas(self): from check_manifest import find_bad_ideas filelist = list(map(os.path.normpath, [ '.gitignore', 'setup.py', 'setup.cfg', 'README.txt', 'src', 'src/zope', 'src/zope/__init__.py', 'src/zope/foo', 'src/zope/foo/__init__.py', 'src/zope/foo/language.po', 'src/zope/foo/language.mo', 'src/zope.foo.egg-info', 'src/zope.foo.egg-info/SOURCES.txt', ])) expected = list(map(os.path.normpath, [ 'src/zope/foo/language.mo', 'src/zope.foo.egg-info', ])) self.assertEqual(find_bad_ideas(filelist), expected) def test_find_suggestions(self): from check_manifest import find_suggestions self.assertEqual(find_suggestions(['buildout.cfg']), (['include buildout.cfg'], [])) self.assertEqual(find_suggestions(['unknown.file~']), ([], ['unknown.file~'])) self.assertEqual(find_suggestions(['README.txt', 'CHANGES.txt']), (['include *.txt'], [])) filelist = list(map(os.path.normpath, [ 'docs/index.rst', 'docs/image.png', 'docs/Makefile', 'docs/unknown-file', 'src/etc/blah/blah/Makefile', ])) expected_rules = [ 'recursive-include docs *.png', 'recursive-include docs *.rst', 'recursive-include docs Makefile', 'recursive-include src Makefile', ] expected_unknowns = [os.path.normpath('docs/unknown-file')] self.assertEqual(find_suggestions(filelist), (expected_rules, expected_unknowns)) def test_find_suggestions_generic_fallback_rules(self): from check_manifest import find_suggestions n = os.path.normpath self.assertEqual(find_suggestions(['Changelog']), (['include Changelog'], [])) self.assertEqual(find_suggestions(['id-lang.map']), (['include *.map'], [])) self.assertEqual(find_suggestions([n('src/id-lang.map')]), (['recursive-include src *.map'], [])) def test_find_suggestions_ignores_directories(self): from check_manifest import find_suggestions with mock.patch('os.path.isdir', lambda d: True): self.assertEqual(find_suggestions(['SOMEDIR']), ([], [])) def test_is_package(self): from check_manifest import is_package j = os.path.join with mock.patch('os.path.exists', lambda fn: fn == j('a', 'setup.py')): self.assertTrue(is_package('a')) self.assertFalse(is_package('b')) def test_glob_to_regexp(self): from check_manifest import _glob_to_regexp as g2r sep = os.path.sep.replace('\\', '\\\\') if sys.version_info >= (3, 6): self.assertEqual(g2r('foo.py'), r'(?s:foo\.py)\Z') self.assertEqual(g2r('foo/bar.py'), r'(?s:foo\/bar\.py)\Z') self.assertEqual(g2r('foo*.py'), r'(?s:foo[^%s]*\.py)\Z' % sep) self.assertEqual(g2r('foo?.py'), r'(?s:foo[^%s]\.py)\Z' % sep) self.assertEqual(g2r('foo[123].py'), r'(?s:foo[123]\.py)\Z') self.assertEqual(g2r('foo[!123].py'), r'(?s:foo[^123]\.py)\Z') self.assertEqual(g2r('foo/*.py'), r'(?s:foo\/[^%s]*\.py)\Z' % sep) else: self.assertEqual(g2r('foo.py'), r'foo\.py\Z(?ms)') self.assertEqual(g2r('foo/bar.py'), r'foo\/bar\.py\Z(?ms)') self.assertEqual(g2r('foo*.py'), r'foo[^%s]*\.py\Z(?ms)' % sep) self.assertEqual(g2r('foo?.py'), r'foo[^%s]\.py\Z(?ms)' % sep) self.assertEqual(g2r('foo[123].py'), r'foo[123]\.py\Z(?ms)') self.assertEqual(g2r('foo[!123].py'), r'foo[^123]\.py\Z(?ms)') self.assertEqual(g2r('foo/*.py'), r'foo\/[^%s]*\.py\Z(?ms)' % sep) def test_get_ignore_from_manifest_lines(self): from check_manifest import _get_ignore_from_manifest_lines as parse from check_manifest import _glob_to_regexp as g2r j = os.path.join # The return value is a tuple with two lists: # ([], []) self.assertEqual(parse([]), ([], [])) self.assertEqual(parse(['', ' ']), ([], [])) self.assertEqual(parse(['exclude *.cfg']), ([], [g2r('*.cfg')])) self.assertEqual(parse(['exclude *.cfg']), ([], [g2r('*.cfg')])) self.assertEqual(parse(['\texclude\t*.cfg foo.* bar.txt']), (['bar.txt'], [g2r('*.cfg'), g2r('foo.*')])) self.assertEqual(parse(['exclude some/directory/*.cfg']), ([], [g2r('some/directory/*.cfg')])) self.assertEqual(parse(['include *.cfg']), ([], [])) self.assertEqual(parse(['global-exclude *.pyc']), (['*.pyc'], [])) self.assertEqual(parse(['global-exclude *.pyc *.sh']), (['*.pyc', '*.sh'], [])) self.assertEqual(parse(['recursive-exclude dir *.pyc']), ([j('dir', '*.pyc')], [])) self.assertEqual(parse(['recursive-exclude dir *.pyc foo*.sh']), ([j('dir', '*.pyc'), j('dir', 'foo*.sh'), j('dir', '*', 'foo*.sh')], [])) self.assertEqual(parse(['recursive-exclude dir nopattern.xml']), ([j('dir', 'nopattern.xml'), j('dir', '*', 'nopattern.xml')], [])) # We should not fail when a recursive-exclude line is wrong: self.assertEqual(parse(['recursive-exclude dirwithoutpattern']), ([], [])) self.assertEqual(parse(['prune dir']), (['dir', j('dir', '*')], [])) # You should not add a slash at the end of a prune, but let's # not fail over it or end up with double slashes. self.assertEqual(parse(['prune dir/']), (['dir', j('dir', '*')], [])) # You should also not have a leading slash self.assertEqual(parse(['prune /dir']), (['/dir', j('/dir', '*')], [])) # And a mongo test case of everything at the end text = textwrap.dedent(""" exclude *.02 exclude *.03 04.* bar.txt exclude *.05 exclude some/directory/*.cfg global-exclude *.10 *.11 global-exclude *.12 include *.20 prune 30 recursive-exclude 40 *.41 recursive-exclude 42 *.43 44.* """).splitlines() self.assertEqual( parse(text), ([ 'bar.txt', '*.10', '*.11', '*.12', '30', j('30', '*'), j('40', '*.41'), j('42', '*.43'), j('42', '44.*'), j('42', '*', '44.*'), ], [ g2r('*.02'), g2r('*.03'), g2r('04.*'), g2r('*.05'), g2r('some/directory/*.cfg'), ])) def test_get_ignore_from_manifest(self): from check_manifest import _get_ignore_from_manifest as parse filename = os.path.join(self.make_temp_dir(), 'MANIFEST.in') self.create_file(filename, textwrap.dedent(''' exclude \\ # yes, this is allowed! test.dat # https://github.com/mgedmin/check-manifest/issues/66 # docs/ folder ''')) self.assertEqual(parse(filename), (['test.dat'], [])) self.assertEqual(self.warnings, []) def test_get_ignore_from_manifest_warnings(self): from check_manifest import _get_ignore_from_manifest as parse filename = os.path.join(self.make_temp_dir(), 'MANIFEST.in') self.create_file(filename, textwrap.dedent(''' # this is bad: a file should not end with a backslash exclude test.dat \\ ''')) self.assertEqual(parse(filename), (['test.dat'], [])) self.assertEqual(self.warnings, [ "%s, line 2: continuation line immediately precedes end-of-file" % filename, ]) class TestConfiguration(unittest.TestCase): def setUp(self): import check_manifest self.oldpwd = os.getcwd() self.tmpdir = tempfile.mkdtemp(prefix='test-', suffix='-check-manifest') os.chdir(self.tmpdir) self.OLD_IGNORE = check_manifest.IGNORE self.OLD_IGNORE_REGEXPS = check_manifest.IGNORE_REGEXPS self.OLD_IGNORE_BAD_IDEAS = check_manifest.IGNORE_BAD_IDEAS check_manifest.IGNORE = ['default-ignore-rules'] check_manifest.IGNORE_REGEXPS = ['default-ignore-regexps'] check_manifest.IGNORE_BAD_IDEAS = [] def tearDown(self): import check_manifest check_manifest.IGNORE = self.OLD_IGNORE check_manifest.IGNORE_REGEXPS = self.OLD_IGNORE_REGEXPS check_manifest.IGNORE_BAD_IDEAS = self.OLD_IGNORE_BAD_IDEAS os.chdir(self.oldpwd) rmtree(self.tmpdir) def test_read_config_no_config(self): import check_manifest check_manifest.read_config() self.assertEqual(check_manifest.IGNORE, ['default-ignore-rules']) def test_read_config_no_section(self): import check_manifest with open('setup.cfg', 'w') as f: f.write('[pep8]\nignore =\n') check_manifest.read_config() self.assertEqual(check_manifest.IGNORE, ['default-ignore-rules']) def test_read_config_no_option(self): import check_manifest with open('setup.cfg', 'w') as f: f.write('[check-manifest]\n') check_manifest.read_config() self.assertEqual(check_manifest.IGNORE, ['default-ignore-rules']) def test_read_config_extra_ignores(self): import check_manifest with open('setup.cfg', 'w') as f: f.write('[check-manifest]\nignore = foo\n bar\n') check_manifest.read_config() self.assertEqual(check_manifest.IGNORE, ['default-ignore-rules', 'foo', 'bar']) def test_read_config_override_ignores(self): import check_manifest with open('setup.cfg', 'w') as f: f.write('[check-manifest]\nignore = foo\n\n bar\n') f.write('ignore-default-rules = yes\n') check_manifest.read_config() self.assertEqual(check_manifest.IGNORE, ['foo', 'bar']) def test_read_config_ignore_bad_ideas(self): import check_manifest with open('setup.cfg', 'w') as f: f.write('[check-manifest]\n' 'ignore-bad-ideas = \n' ' foo\n' ' bar\n') check_manifest.read_config() self.assertEqual(check_manifest.IGNORE_BAD_IDEAS, ['foo', 'bar']) def test_read_manifest_no_manifest(self): import check_manifest check_manifest.read_manifest() self.assertEqual(check_manifest.IGNORE, ['default-ignore-rules']) def test_read_manifest(self): import check_manifest from check_manifest import _glob_to_regexp as g2r with open('MANIFEST.in', 'w') as f: f.write('exclude *.gif\n') f.write('global-exclude *.png\n') check_manifest.read_manifest() self.assertEqual(check_manifest.IGNORE, ['default-ignore-rules', '*.png']) self.assertEqual(check_manifest.IGNORE_REGEXPS, ['default-ignore-regexps', g2r('*.gif')]) class TestMain(unittest.TestCase): def setUp(self): import check_manifest self._cm_patcher = mock.patch('check_manifest.check_manifest') self._check_manifest = self._cm_patcher.start() self._se_patcher = mock.patch('sys.exit') self._sys_exit = self._se_patcher.start() self._error_patcher = mock.patch('check_manifest.error') self._error = self._error_patcher.start() self._orig_sys_argv = sys.argv sys.argv = ['check-manifest'] self.OLD_IGNORE = check_manifest.IGNORE self.OLD_IGNORE_REGEXPS = check_manifest.IGNORE_REGEXPS self.OLD_IGNORE_BAD_IDEAS = check_manifest.IGNORE_BAD_IDEAS check_manifest.IGNORE = ['default-ignore-rules'] check_manifest.IGNORE_REGEXPS = ['default-ignore-regexps'] check_manifest.IGNORE_BAD_IDEAS = [] def tearDown(self): import check_manifest check_manifest.IGNORE = self.OLD_IGNORE check_manifest.IGNORE_REGEXPS = self.OLD_IGNORE_REGEXPS check_manifest.IGNORE_BAD_IDEAS = self.OLD_IGNORE_BAD_IDEAS sys.argv = self._orig_sys_argv self._se_patcher.stop() self._cm_patcher.stop() self._error_patcher.stop() def test(self): from check_manifest import main sys.argv.append('-v') main() def test_exit_code_1_on_error(self): from check_manifest import main self._check_manifest.return_value = False main() self._sys_exit.assert_called_with(1) def test_exit_code_2_on_failure(self): from check_manifest import main, Failure self._check_manifest.side_effect = Failure('msg') main() self._error.assert_called_with('msg') self._sys_exit.assert_called_with(2) def test_extra_ignore_args(self): import check_manifest sys.argv.append('--ignore=x,y,z') check_manifest.main() self.assertEqual(check_manifest.IGNORE, ['default-ignore-rules', 'x', 'y', 'z']) def test_ignore_bad_ideas_args(self): import check_manifest sys.argv.append('--ignore-bad-ideas=x,y,z') check_manifest.main() self.assertEqual(check_manifest.IGNORE_BAD_IDEAS, ['x', 'y', 'z']) class TestZestIntegration(unittest.TestCase): def setUp(self): sys.modules['zest'] = mock.Mock() sys.modules['zest.releaser'] = mock.Mock() sys.modules['zest.releaser.utils'] = mock.Mock() self.ask = sys.modules['zest.releaser.utils'].ask def tearDown(self): del sys.modules['zest.releaser.utils'] del sys.modules['zest.releaser'] del sys.modules['zest'] @mock.patch('check_manifest.is_package', lambda d: False) @mock.patch('check_manifest.check_manifest') def test_zest_releaser_check_not_a_package(self, check_manifest): from check_manifest import zest_releaser_check zest_releaser_check(dict(workingdir='.')) check_manifest.assert_not_called() @mock.patch('check_manifest.is_package', lambda d: True) @mock.patch('check_manifest.check_manifest') def test_zest_releaser_check_user_disagrees(self, check_manifest): from check_manifest import zest_releaser_check self.ask.return_value = False zest_releaser_check(dict(workingdir='.')) check_manifest.assert_not_called() @mock.patch('check_manifest.is_package', lambda d: True) @mock.patch('sys.exit') @mock.patch('check_manifest.check_manifest') def test_zest_releaser_check_all_okay(self, check_manifest, sys_exit): from check_manifest import zest_releaser_check self.ask.return_value = True check_manifest.return_value = True zest_releaser_check(dict(workingdir='.')) sys_exit.assert_not_called() @mock.patch('check_manifest.is_package', lambda d: True) @mock.patch('check_manifest.error') @mock.patch('sys.exit') @mock.patch('check_manifest.check_manifest') def test_zest_releaser_check_error_user_aborts(self, check_manifest, sys_exit, error): from check_manifest import zest_releaser_check self.ask.side_effect = [True, False] check_manifest.return_value = False zest_releaser_check(dict(workingdir='.')) sys_exit.assert_called_with(1) @mock.patch('check_manifest.is_package', lambda d: True) @mock.patch('check_manifest.error') @mock.patch('sys.exit') @mock.patch('check_manifest.check_manifest') def test_zest_releaser_check_error_user_plods_on(self, check_manifest, sys_exit, error): from check_manifest import zest_releaser_check self.ask.side_effect = [True, True] check_manifest.return_value = False zest_releaser_check(dict(workingdir='.')) sys_exit.assert_not_called() @mock.patch('check_manifest.is_package', lambda d: True) @mock.patch('check_manifest.error') @mock.patch('sys.exit') @mock.patch('check_manifest.check_manifest') def test_zest_releaser_check_failure_user_aborts(self, check_manifest, sys_exit, error): from check_manifest import zest_releaser_check, Failure self.ask.side_effect = [True, False] check_manifest.side_effect = Failure('msg') zest_releaser_check(dict(workingdir='.')) error.assert_called_with('msg') sys_exit.assert_called_with(2) @mock.patch('check_manifest.is_package', lambda d: True) @mock.patch('check_manifest.error') @mock.patch('sys.exit') @mock.patch('check_manifest.check_manifest') def test_zest_releaser_check_failure_user_plods_on(self, check_manifest, sys_exit, error): from check_manifest import zest_releaser_check, Failure self.ask.side_effect = [True, True] check_manifest.side_effect = Failure('msg') zest_releaser_check(dict(workingdir='.')) error.assert_called_with('msg') sys_exit.assert_not_called() class VCSHelper(object): command = None # override in subclasses def is_installed(self): try: p = subprocess.Popen([self.command, '--version'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT) stdout, stderr = p.communicate() rc = p.wait() return (rc == 0) except OSError: return False def _run(self, *command): # Windows doesn't like Unicode arguments to subprocess.Popen(), on Py2: # https://github.com/mgedmin/check-manifest/issues/23#issuecomment-33933031 if str is bytes: command = [s.encode(locale.getpreferredencoding()) for s in command] p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) stdout, stderr = p.communicate() rc = p.wait() if rc: print(' '.join(command)) print(stdout) raise subprocess.CalledProcessError(rc, command[0], output=stdout) class VCSMixin(object): def setUp(self): if not self.vcs.is_installed() and CAN_SKIP_TESTS: self.skipTest("%s is not installed" % self.vcs.command) self.tmpdir = tempfile.mkdtemp(prefix='test-', suffix='-check-manifest') self.olddir = os.getcwd() os.chdir(self.tmpdir) def tearDown(self): os.chdir(self.olddir) rmtree(self.tmpdir) def _create_file(self, filename): assert not os.path.isabs(filename) basedir = os.path.dirname(filename) if basedir and not os.path.isdir(basedir): os.makedirs(basedir) open(filename, 'w').close() def _create_files(self, filenames): for filename in filenames: self._create_file(filename) def _init_vcs(self): self.vcs._init_vcs() def _add_to_vcs(self, filenames): self.vcs._add_to_vcs(filenames) def _commit(self): self.vcs._commit() def _create_and_add_to_vcs(self, filenames): self._create_files(filenames) self._add_to_vcs(filenames) def test_get_vcs_files(self): from check_manifest import get_vcs_files self._init_vcs() self._create_and_add_to_vcs(['a.txt', 'b/b.txt', 'b/c/d.txt']) self._commit() self._create_files(['b/x.txt', 'd/d.txt', 'i.txt']) j = os.path.join self.assertEqual(get_vcs_files(), ['a.txt', 'b', j('b', 'b.txt'), j('b', 'c'), j('b', 'c', 'd.txt')]) def test_get_vcs_files_added_but_uncommitted(self): from check_manifest import get_vcs_files self._init_vcs() self._create_and_add_to_vcs(['a.txt', 'b/b.txt', 'b/c/d.txt']) self._create_files(['b/x.txt', 'd/d.txt', 'i.txt']) j = os.path.join self.assertEqual(get_vcs_files(), ['a.txt', 'b', j('b', 'b.txt'), j('b', 'c'), j('b', 'c', 'd.txt')]) def test_get_vcs_files_deleted_but_not_removed(self): if self.vcs.command == 'bzr': self.skipTest("this cosmetic feature is not supported with bzr") # see the longer explanation in test_missing_source_files from check_manifest import get_vcs_files self._init_vcs() self._create_and_add_to_vcs(['a.txt']) self._commit() os.unlink('a.txt') self.assertEqual(get_vcs_files(), ['a.txt']) def test_get_vcs_files_in_a_subdir(self): from check_manifest import get_vcs_files self._init_vcs() self._create_and_add_to_vcs(['a.txt', 'b/b.txt', 'b/c/d.txt']) self._commit() self._create_files(['b/x.txt', 'd/d.txt', 'i.txt']) os.chdir('b') j = os.path.join self.assertEqual(get_vcs_files(), ['b.txt', 'c', j('c', 'd.txt')]) def test_get_vcs_files_nonascii_filenames(self): # This test will fail if your locale is incapable of expressing # "eacute". UTF-8 or Latin-1 should work. from check_manifest import get_vcs_files self._init_vcs() # A spelling of u"\xe9.txt" that works on Python 3.2 too filename = b'\xc3\xa9.txt'.decode('UTF-8') self._create_and_add_to_vcs([filename]) self.assertEqual(get_vcs_files(), [filename]) def test_get_vcs_files_empty(self): from check_manifest import get_vcs_files self._init_vcs() self.assertEqual(get_vcs_files(), []) class GitHelper(VCSHelper): command = 'git' def _init_vcs(self): self._run('git', 'init') self._run('git', 'config', 'user.name', 'Unit Test') self._run('git', 'config', 'user.email', 'test@example.com') def _add_to_vcs(self, filenames): # Note that we use --force to prevent errors when we want to # add foo.egg-info and the user running the tests has # '*.egg-info' in her global .gitignore file. self._run('git', 'add', '--force', '--', *filenames) def _commit(self): self._run('git', 'commit', '-m', 'Initial') class TestGit(VCSMixin, unittest.TestCase): vcs = GitHelper() def _init_repo_with_files(self, dirname, filenames): os.mkdir(dirname) os.chdir(dirname) self._init_vcs() self._create_and_add_to_vcs(filenames) self._commit() os.chdir(self.tmpdir) def _add_submodule(self, repo, subdir, subrepo): os.chdir(repo) self.vcs._run('git', 'submodule', 'add', subrepo, subdir) self._commit() os.chdir(self.tmpdir) def test_detect_git_submodule(self): from check_manifest import detect_vcs, Failure with self.assertRaises(Failure) as cm: detect_vcs() self.assertEqual(str(cm.exception), "Couldn't find version control data" " (git/hg/bzr/svn supported)") # now create a .git file like in a submodule open(os.path.join(self.tmpdir, '.git'), 'w').close() self.assertEqual(detect_vcs().metadata_name, '.git') def test_get_versioned_files_with_git_submodules(self): from check_manifest import get_vcs_files self._init_repo_with_files('repo1', ['file1', 'file2']) self._init_repo_with_files('repo2', ['file3']) self._init_repo_with_files('repo3', ['file4']) self._add_submodule('repo2', 'sub3', '../repo3') self._init_repo_with_files('main', ['file5', 'subdir/file6']) self._add_submodule('main', 'sub1', '../repo1') self._add_submodule('main', 'subdir/sub2', '../repo2') os.chdir('main') self.vcs._run('git', 'submodule', 'update', '--init', '--recursive') self.assertEqual( get_vcs_files(), [fn.replace('/', os.path.sep) for fn in [ '.gitmodules', 'file5', 'sub1', 'sub1/file1', 'sub1/file2', 'subdir', 'subdir/file6', 'subdir/sub2', 'subdir/sub2/.gitmodules', 'subdir/sub2/file3', 'subdir/sub2/sub3', 'subdir/sub2/sub3/file4', ]]) class BzrHelper(VCSHelper): command = 'bzr' def _init_vcs(self): self._run('bzr', 'init') self._run('bzr', 'whoami', '--branch', 'Unit Test ') def _add_to_vcs(self, filenames): self._run('bzr', 'add', '--', *filenames) def _commit(self): self._run('bzr', 'commit', '-m', 'Initial') class TestBzr(VCSMixin, unittest.TestCase): vcs = BzrHelper() @unittest.skipIf(HAS_OEM_CODEC, "Python 3.6 lets us use 'oem' codec instead of guessing") class TestBzrTerminalCharsetDetectionOnOldPythons(unittest.TestCase): @mock.patch('sys.stdin') @mock.patch('sys.stdout') def test_terminal_encoding_not_known(self, mock_stdout, mock_stdin): from check_manifest import Bazaar mock_stdout.encoding = None mock_stdin.encoding = None self.assertEqual(Bazaar._get_terminal_encoding(), None) @mock.patch('sys.stdout') def test_terminal_encoding_stdout_known(self, mock_stdout): from check_manifest import Bazaar mock_stdout.encoding = 'UTF-8' self.assertEqual(Bazaar._get_terminal_encoding(), 'UTF-8') @mock.patch('sys.stdin') @mock.patch('sys.stdout') def test_terminal_encoding_stdin_known(self, mock_stdout, mock_stdin): from check_manifest import Bazaar mock_stdout.encoding = None mock_stdin.encoding = 'UTF-8' self.assertEqual(Bazaar._get_terminal_encoding(), 'UTF-8') @mock.patch('sys.stdout') def test_terminal_encoding_cp0(self, mock_stdout): from check_manifest import Bazaar mock_stdout.encoding = 'cp0' self.assertEqual(Bazaar._get_terminal_encoding(), None) @unittest.skipIf(not HAS_OEM_CODEC, "'oem' codec not available on Python before 3.6") class TestBzrTerminalCharsetDetectionOnNewPythons(unittest.TestCase): def test_terminal_encoding_cp0(self): from check_manifest import Bazaar self.assertEqual(Bazaar._get_terminal_encoding(), "oem") class HgHelper(VCSHelper): command = 'hg' def _init_vcs(self): self._run('hg', 'init') with open('.hg/hgrc', 'a') as f: f.write('\n[ui]\nusername = Unit Test ') self.assertFalse(Subversion.is_interesting(entry)) self.assertEqual( sys.stderr.getvalue(), 'svn status --xml parse error: ' ' without \n') class TestUserInterface(UIMixin, unittest.TestCase): def test_info(self): import check_manifest check_manifest.VERBOSE = False check_manifest.info("Reticulating splines") self.assertEqual(sys.stdout.getvalue(), "Reticulating splines\n") def test_info_verbose(self): import check_manifest check_manifest.VERBOSE = True check_manifest.info("Reticulating splines") self.assertEqual(sys.stdout.getvalue(), "Reticulating splines\n") def test_info_begin_continue_end(self): import check_manifest check_manifest.VERBOSE = False check_manifest.info_begin("Reticulating splines...") check_manifest.info_continue(" nearly done...") check_manifest.info_continue(" almost done...") check_manifest.info_end(" done!") self.assertEqual(sys.stdout.getvalue(), "") def test_info_begin_continue_end_verbose(self): import check_manifest check_manifest.VERBOSE = True check_manifest.info_begin("Reticulating splines...") check_manifest.info_continue(" nearly done...") check_manifest.info_continue(" almost done...") check_manifest.info_end(" done!") self.assertEqual( sys.stdout.getvalue(), "Reticulating splines... nearly done... almost done... done!\n") def test_info_emits_newline_when_needed(self): import check_manifest check_manifest.VERBOSE = False check_manifest.info_begin("Computering...") check_manifest.info("Forgot to turn the gas off!") self.assertEqual( sys.stdout.getvalue(), "Forgot to turn the gas off!\n") def test_info_emits_newline_when_needed_verbose(self): import check_manifest check_manifest.VERBOSE = True check_manifest.info_begin("Computering...") check_manifest.info("Forgot to turn the gas off!") self.assertEqual( sys.stdout.getvalue(), "Computering...\n" "Forgot to turn the gas off!\n") def test_warning(self): import check_manifest check_manifest.VERBOSE = False check_manifest.info_begin("Computering...") check_manifest.warning("Forgot to turn the gas off!") self.assertEqual(sys.stdout.getvalue(), "") self.assertEqual( sys.stderr.getvalue(), "Forgot to turn the gas off!\n") def test_warning_verbose(self): import check_manifest check_manifest.VERBOSE = True check_manifest.info_begin("Computering...") check_manifest.warning("Forgot to turn the gas off!") self.assertEqual( sys.stdout.getvalue(), "Computering...\n") self.assertEqual( sys.stderr.getvalue(), "Forgot to turn the gas off!\n") def test_error(self): import check_manifest check_manifest.VERBOSE = False check_manifest.info_begin("Computering...") check_manifest.error("Forgot to turn the gas off!") self.assertEqual(sys.stdout.getvalue(), "") self.assertEqual( sys.stderr.getvalue(), "Forgot to turn the gas off!\n") def test_error_verbose(self): import check_manifest check_manifest.VERBOSE = True check_manifest.info_begin("Computering...") check_manifest.error("Forgot to turn the gas off!") self.assertEqual( sys.stdout.getvalue(), "Computering...\n") self.assertEqual( sys.stderr.getvalue(), "Forgot to turn the gas off!\n") def pick_installed_vcs(): preferred_order = [GitHelper, HgHelper, BzrHelper, SvnHelper] force = os.getenv('FORCE_TEST_VCS') if force: for cls in preferred_order: if force == cls.command: return cls() raise ValueError('Unsupported FORCE_TEST_VCS=%s (supported: %s)' % (force, '/'.join(cls.command for cls in preferred_order))) for cls in preferred_order: vcs = cls() if vcs.is_installed(): return vcs return None class TestCheckManifest(unittest.TestCase): _vcs = pick_installed_vcs() def setUp(self): if self._vcs is None: self.fail('at least one version control system should be installed') self.oldpwd = os.getcwd() self.tmpdir = tempfile.mkdtemp(prefix='test-', suffix='-check-manifest') os.chdir(self.tmpdir) self._stdout_patcher = mock.patch('sys.stdout', StringIO()) self._stdout_patcher.start() self._stderr_patcher = mock.patch('sys.stderr', StringIO()) self._stderr_patcher.start() def tearDown(self): self._stderr_patcher.stop() self._stdout_patcher.stop() os.chdir(self.oldpwd) rmtree(self.tmpdir) def _create_repo_with_code(self, add_to_vcs=True): self._vcs._init_vcs() with open('setup.py', 'w') as f: f.write("from setuptools import setup\n") f.write("setup(name='sample', py_modules=['sample'])\n") with open('sample.py', 'w') as f: f.write("# wow. such code. so amaze\n") if add_to_vcs: self._vcs._add_to_vcs(['setup.py', 'sample.py']) def _create_repo_with_code_in_subdir(self): os.mkdir('subdir') os.chdir('subdir') self._create_repo_with_code() # NB: when self._vcs is SvnHelper, we're actually in # ./subdir/checkout rather than in ./subdir subdir = os.path.basename(os.getcwd()) os.chdir(os.pardir) return subdir def _add_to_vcs(self, filename, content=''): if os.path.sep in filename and not os.path.isdir(os.path.dirname(filename)): os.makedirs(os.path.dirname(filename)) with open(filename, 'w') as f: f.write(content) self._vcs._add_to_vcs([filename]) def test_not_python_project(self): from check_manifest import check_manifest, Failure with self.assertRaises(Failure) as cm: check_manifest() self.assertEqual(str(cm.exception), "This is not a Python project (no setup.py).") def test_forgot_to_git_add_anything(self): from check_manifest import check_manifest, Failure self._create_repo_with_code(add_to_vcs=False) with self.assertRaises(Failure) as cm: check_manifest() self.assertEqual(str(cm.exception), "There are no files added to version control!") def test_all_is_well(self): from check_manifest import check_manifest self._create_repo_with_code() self.assertTrue(check_manifest()) def test_relative_pathname(self): from check_manifest import check_manifest subdir = self._create_repo_with_code_in_subdir() self.assertTrue(check_manifest(subdir)) def test_relative_python(self): # https://github.com/mgedmin/check-manifest/issues/36 from check_manifest import check_manifest subdir = self._create_repo_with_code_in_subdir() python = os.path.relpath(sys.executable) self.assertTrue(check_manifest(subdir, python=python)) def test_python_from_path(self): # https://github.com/mgedmin/check-manifest/issues/57 # NB: this test assumes you have a 'python' executable somewhere # in your path. from check_manifest import check_manifest subdir = self._create_repo_with_code_in_subdir() self.assertTrue(check_manifest(subdir, python='python')) def test_suggestions(self): from check_manifest import check_manifest self._create_repo_with_code() self._add_to_vcs('unrelated.txt') self.assertFalse(check_manifest()) self.assertIn("missing from sdist:\n unrelated.txt", sys.stderr.getvalue()) self.assertIn("suggested MANIFEST.in rules:\n include *.txt", sys.stdout.getvalue()) def test_suggestions_create(self): from check_manifest import check_manifest self._create_repo_with_code() self._add_to_vcs('unrelated.txt') self.assertFalse(check_manifest(create=True)) self.assertIn("missing from sdist:\n unrelated.txt", sys.stderr.getvalue()) self.assertIn("suggested MANIFEST.in rules:\n include *.txt", sys.stdout.getvalue()) self.assertIn("creating MANIFEST.in", sys.stdout.getvalue()) with open('MANIFEST.in') as f: self.assertEqual(f.read(), "include *.txt\n") def test_suggestions_update(self): from check_manifest import check_manifest self._create_repo_with_code() self._add_to_vcs('unrelated.txt') self._add_to_vcs('MANIFEST.in', '#tbd') self.assertFalse(check_manifest(update=True)) self.assertIn("missing from sdist:\n unrelated.txt", sys.stderr.getvalue()) self.assertIn("suggested MANIFEST.in rules:\n include *.txt", sys.stdout.getvalue()) self.assertIn("updating MANIFEST.in", sys.stdout.getvalue()) with open('MANIFEST.in') as f: self.assertEqual( f.read(), "#tbd\n# added by check_manifest.py\ninclude *.txt\n") def test_suggestions_all_unknown_patterns(self): from check_manifest import check_manifest self._create_repo_with_code() self._add_to_vcs('.dunno-what-to-do-with-this') self.assertFalse(check_manifest(update=True)) self.assertIn("missing from sdist:\n .dunno-what-to-do-with-this", sys.stderr.getvalue()) self.assertIn( "don't know how to come up with rules matching any of the files, sorry!", sys.stdout.getvalue()) def test_suggestions_some_unknown_patterns(self): from check_manifest import check_manifest self._create_repo_with_code() self._add_to_vcs('.dunno-what-to-do-with-this') self._add_to_vcs('unrelated.txt') self.assertFalse(check_manifest(update=True)) self.assertIn( "don't know how to come up with rules matching\n .dunno-what-to-do-with-this", sys.stdout.getvalue()) self.assertIn("creating MANIFEST.in", sys.stdout.getvalue()) with open('MANIFEST.in') as f: self.assertEqual(f.read(), "include *.txt\n") def test_MANIFEST_in_does_not_need_to_be_added_to_be_considered(self): from check_manifest import check_manifest self._create_repo_with_code() self._add_to_vcs('unrelated.txt') with open('MANIFEST.in', 'w') as f: f.write("include *.txt\n") self.assertFalse(check_manifest()) self.assertIn("missing from VCS:\n MANIFEST.in", sys.stderr.getvalue()) self.assertNotIn("missing from sdist", sys.stderr.getvalue()) def test_setup_py_does_not_need_to_be_added_to_be_considered(self): from check_manifest import check_manifest self._create_repo_with_code(add_to_vcs=False) self._add_to_vcs('sample.py') self.assertFalse(check_manifest()) self.assertIn("missing from VCS:\n setup.py", sys.stderr.getvalue()) self.assertNotIn("missing from sdist", sys.stderr.getvalue()) def test_bad_ideas(self): from check_manifest import check_manifest self._create_repo_with_code() self._add_to_vcs('foo.egg-info') self._add_to_vcs('moo.mo') self.assertFalse(check_manifest()) self.assertIn("you have foo.egg-info in source control!", sys.stderr.getvalue()) self.assertIn("this also applies to the following:\n moo.mo", sys.stderr.getvalue()) def test_ignore_bad_ideas(self): from check_manifest import check_manifest self._create_repo_with_code() with open('setup.cfg', 'w') as f: f.write('[check-manifest]\n' 'ignore =\n' ' subdir/bar.egg-info\n' 'ignore-bad-ideas =\n' ' *.mo\n' ' subdir/bar.egg-info\n') self._add_to_vcs('foo.egg-info') self._add_to_vcs('moo.mo') self._add_to_vcs(os.path.join('subdir', 'bar.egg-info')) self.assertFalse(check_manifest()) self.assertIn("you have foo.egg-info in source control!", sys.stderr.getvalue()) self.assertNotIn("moo.mo", sys.stderr.getvalue()) self.assertNotIn("bar.egg-info", sys.stderr.getvalue()) def test_missing_source_files(self): # https://github.com/mgedmin/check-manifest/issues/32 from check_manifest import check_manifest self._create_repo_with_code() self._add_to_vcs('missing.py') os.unlink('missing.py') check_manifest() if self._vcs.command != 'bzr': # 'bzr ls' doesn't list files that were deleted but not # marked for deletion. 'bzr st' does, but it doesn't list # unmodified files. Importing bzrlib and using the API to # get the file list we need is (a) complicated, (b) opens # the optional dependency can of worms, and (c) not viable # under Python 3 unless we fork off a Python 2 subprocess. # Manually combining 'bzr ls' and 'bzr st' outputs just to # produce a cosmetic warning message seems like overkill. self.assertIn("some files listed as being under source control are missing:\n missing.py", sys.stderr.getvalue()) def test_suite(): return unittest.TestSuite([ unittest.defaultTestLoader.loadTestsFromName(__name__), doctest.DocTestSuite('check_manifest'), ]) check-manifest-0.36/tox.ini000066400000000000000000000011661320501433100156320ustar00rootroot00000000000000[tox] envlist = py27,py33,py34,py35,py36,pypy,pypy3 [testenv] passenv = LANG LC_CTYPE LC_ALL MSYSTEM deps = mock nose commands = nosetests {posargs} ## if I run check-manifest here, it breaks my 'make release' :( ## check-manifest [testenv:coverage] deps = {[testenv]deps} coverage commands = coverage run --source=check_manifest -m nose ## since I'm not running check-manifest during regular tests, let's not lie ## about the coverage numbers ## coverage run --append check_manifest.py coverage report -m --fail-under=100 [testenv:py] commands = python --version nosetests {posargs}