importlab-0.8.1/0000755000175000017500000000000014567445061013425 5ustar faunrisfaunrisimportlab-0.8.1/PKG-INFO0000640000175000017500000000442514567445061014523 0ustar faunrisfaunrisMetadata-Version: 2.1 Name: importlab Version: 0.8.1 Summary: A library to calculate python dependency graphs. Home-page: https://github.com/google/importlab Maintainer: Google Inc. Maintainer-email: pytype-dev@google.com License: Apache 2.0 Classifier: Development Status :: 4 - Beta Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: Apache Software License Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: Implementation :: CPython Classifier: Topic :: Software Development Requires-Python: >=3.6.0 License-File: LICENSE importlab --------- Importlab is a library for Python that automatically infers dependencies and calculates a dependency graph. It can perform dependency ordering of a set of files, including cycle detection. Importlab's main use case is to work with static analysis tools that process one file at a time, ensuring that a file's dependencies are analysed before it is. (This is not an official Google product.) License ------- Apache 2.0 Installation ------------ Importlab can be installed from pip :: pip install importlab To check out and install the latest source code :: git clone https://github.com/google/importlab.git cd importlab python setup.py install Usage ----- Importlab is primarily intended to be used as a library. It takes one or more python files as arguments, and generates an import graph, typically used to process files in dependency order. It is currently integrated into `pytype `__ Command-line tool ----------------- Importlab ships with a small command-line tool, also called ``importlab``, which can display some information about a project's import graph. :: usage: importlab [-h] [--tree] [--unresolved] [filename [filename ...]] positional arguments: filename input file(s) optional arguments: -h, --help show this help message and exit --tree Display import tree. --unresolved Display unresolved dependencies. Roadmap ------- - ``Makefile`` generation, to take advantage of ``make``'s incremental update and parallel execution features - Integration with other static analysis tools importlab-0.8.1/testdata/0000755000175000017500000000000014567445061015236 5ustar faunrisfaunrisimportlab-0.8.1/testdata/test.py0000640000175000017500000000002314567445061016556 0ustar faunrisfaunrisfrom .pkg import a importlab-0.8.1/testdata/pkg/0000755000175000017500000000000014567445061016017 5ustar faunrisfaunrisimportlab-0.8.1/testdata/pkg/a.py0000640000175000017500000000006114567445061016602 0ustar faunrisfaunrisfrom . import b from . import c from . import d importlab-0.8.1/testdata/pkg/d.py0000640000175000017500000000000514567445061016603 0ustar faunrisfaunrispass importlab-0.8.1/testdata/pkg/c.py0000640000175000017500000000002014567445061016577 0ustar faunrisfaunrisfrom . import b importlab-0.8.1/testdata/pkg/b.py0000640000175000017500000000002014567445061016576 0ustar faunrisfaunrisfrom . import c importlab-0.8.1/LICENSE0000640000175000017500000002613514567445061014435 0ustar faunrisfaunris Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. importlab-0.8.1/bin/0000755000175000017500000000000014567445061014175 5ustar faunrisfaunrisimportlab-0.8.1/bin/importlab0000750000175000017500000000630114567445061016107 0ustar faunrisfaunris#!/usr/bin/env python3 # Copyright 2017 Google Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """main entry point.""" from __future__ import print_function import argparse import os import sys from importlib.metadata import version from importlab import environment from importlab import graph from importlab import output from importlab import utils def parse_args(): parser = argparse.ArgumentParser() parser.add_argument('inputs', metavar='input', type=str, nargs='*', help=('Input files or directories. Directories will be ' 'recursively scanned for .py files')) parser.add_argument('--tree', dest='tree', action='store_true', default=False, help='Display import tree.') parser.add_argument('--unresolved', dest='unresolved', action='store_true', default=False, help='Display unresolved dependencies.') default_python_version = '%d.%d' % sys.version_info[:2] parser.add_argument('-V', '--python_version', type=str, action='store', dest='python_version', default=default_python_version, help='Python version of target code, e.g. "2.7"') parser.add_argument('-P', '--pythonpath', type=str, action='store', dest='pythonpath', default='', help=('Directories for reading dependencies - a list ' 'of paths separated by "%s".') % os.pathsep) parser.add_argument('--trim', dest='trim', action='store_true', default=False, help=('Trim the dependencies of builtin and system ' 'files.')) parser.add_argument('-v', '--version', action='version', version=version('importlab'), help='Script version') return parser.parse_args() def main(): args = parse_args() # Exit early if we don't have any output args. if not (args.tree or args.unresolved): print('Nothing to do!') sys.exit(0) args.inputs = utils.expand_source_files(args.inputs) print('Reading %d files' % len(args.inputs)) env = environment.create_from_args(args) import_graph = graph.ImportGraph.create(env, args.inputs, args.trim) if args.tree: print('Source tree:') output.print_tree(import_graph) output.maybe_show_unreadable(import_graph) sys.exit(0) if args.unresolved: print('Unresolved dependencies:') output.print_unresolved_dependencies(import_graph) output.maybe_show_unreadable(import_graph) sys.exit(0) if __name__ == "__main__": sys.exit(main()) importlab-0.8.1/importlab/0000755000175000017500000000000014567445061015416 5ustar faunrisfaunrisimportlab-0.8.1/importlab/resolve.py0000640000175000017500000002136114567445061017446 0ustar faunrisfaunris# Copyright 2017 Google Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Logic for resolving import paths.""" import logging import os from . import import_finder from . import utils class ImportException(ImportError): def __init__(self, module_name): super(ImportException, self).__init__(module_name) self.module_name = module_name class ResolvedFile(object): def __init__(self, path, module_name): self.path = path self.module_name = module_name def is_extension(self): return self.path.endswith('.so') @property def package_name(self): f, _ = os.path.splitext(self.path) if f.endswith('__init__'): return self.module_name elif '.' in self.module_name: return self.module_name[:self.module_name.rindex('.')] else: return None @property def short_path(self): parts = self.path.split(os.path.sep) n = self.module_name.count('.') if parts[-1] == '__init__.py': n += 1 parts = parts[-(n+1):] return os.path.join(*parts) class Direct(ResolvedFile): """Files added directly as arguments.""" def __init__(self, path, module_name=''): # We do not necessarily have a module name for a directly added file. super(Direct, self).__init__(path, module_name) class Builtin(ResolvedFile): """Imports that are resolved via python's builtins.""" class System(ResolvedFile): """Imports that are resolved by python.""" pass class Local(ResolvedFile): """Imports that are found in a local pythonpath.""" def __init__(self, path, module_name, fs): super(Local, self).__init__(path, module_name) self.fs = fs def convert_to_path(name): """Converts ".module" to "./module", "..module" to "../module", etc.""" if name.startswith('.'): remainder = name.lstrip('.') dot_count = (len(name) - len(remainder)) prefix = '../'*(dot_count-1) else: remainder = name dot_count = 0 prefix = '' filename = prefix + os.path.join(*remainder.split('.')) return (filename, dot_count) def infer_module_name(filename, fspath): """Convert a python filename to a module relative to pythonpath.""" filename, _ = os.path.splitext(filename) for f in fspath: short_name = f.relative_path(filename) if short_name: # The module name for __init__.py files is the directory. if short_name.endswith(os.path.sep + "__init__"): short_name = short_name[:short_name.rfind(os.path.sep)] return short_name.replace(os.path.sep, '.') # We have not found filename relative to anywhere in pythonpath. return '' def get_absolute_name(package, relative_name): """Joins a package name and a relative name. Args: package: A dotted name, e.g. foo.bar.baz relative_name: A dotted name with possibly some leading dots, e.g. ..x.y Returns: The relative name appended to the parent's package, after going up one level for each leading dot. e.g. foo.bar.baz + ..hello.world -> foo.hello.world The unchanged relative_name if it does not start with a dot or has too many leading dots. """ path = package.split('.') if package else [] name = relative_name.lstrip('.') ndots = len(relative_name) - len(name) if ndots > len(path): return relative_name absolute_path = path[:len(path) + 1 - ndots] if name: absolute_path.append(name) return '.'.join(absolute_path) class Resolver: def __init__(self, fs_path, current_module): self.fs_path = fs_path self.current_module = current_module self.current_directory = os.path.dirname(current_module.path) def _find_file(self, fs, name): init = os.path.join(name, '__init__.py') py = name + '.py' for x in [init, py]: if fs.isfile(x): return fs.refer_to(x) return None def resolve_import(self, item): """Simulate how Python resolves imports. Returns the filename of the source file Python would load when processing a statement like 'import name' in the module we're currently under. Args: item: An instance of ImportItem Returns: A filename Raises: ImportException: If the module doesn't exist. """ name = item.name # The last part in `from a.b.c import d` might be a symbol rather than a # module, so we try a.b.c and a.b.c.d as names. short_name = None if item.is_from and not item.is_star: if '.' in name.lstrip('.'): # The name is something like `a.b.c`, so strip off `.c`. rindex = name.rfind('.') else: # The name is something like `..c`, so strip off just `c`. rindex = name.rfind('.') + 1 short_name = name[:rindex] if import_finder.is_builtin(name): filename = name + '.so' return Builtin(filename, name) filename, level = convert_to_path(name) if level: # This is a relative import; we need to resolve the filename # relative to the importing file path. filename = os.path.normpath( os.path.join(self.current_directory, filename)) if not short_name: try_filename = True try_short_filename = False elif item.source: # If the import has a source path, we can use it to eliminate # filenames that don't match. source_filename, _ = os.path.splitext(item.source) dirname, basename = os.path.split(source_filename) if basename == "__init__": source_filename = dirname try_filename = source_filename.endswith(filename) try_short_filename = not try_filename else: try_filename = try_short_filename = True files = [] if try_filename: files.append((name, filename)) if try_short_filename: short_filename = os.path.dirname(filename) files.append((short_name, short_filename)) for module_name, path in files: for fs in self.fs_path: f = self._find_file(fs, path) if not f or f == self.current_module.path: # We cannot import a file from itself. continue if item.is_relative(): package_name = self.current_module.package_name if package_name is None: # Relative import in non-package raise ImportException(name) module_name = get_absolute_name(package_name, module_name) if isinstance(self.current_module, System): return System(f, module_name) return Local(f, module_name, fs) # If the module isn't found in the explicit pythonpath, see if python # itself resolved it. if item.source: prefix, ext = os.path.splitext(item.source) mod_name = name # We need to check for importing a symbol here too. if short_name: mod = prefix.replace(os.path.sep, '.') mod = utils.strip_suffix(mod, '.__init__') if not mod.endswith(name) and mod.endswith(short_name): mod_name = short_name if ext == '.pyc': pyfile = prefix + '.py' if os.path.exists(pyfile): return System(pyfile, mod_name) elif not ext: pyfile = os.path.join(prefix, "__init__.py") if os.path.exists(pyfile): return System(pyfile, mod_name) return System(item.source, mod_name) raise ImportException(name) def resolve_all(self, import_items): """Resolves a list of imports. Yields filenames. """ for import_item in import_items: try: yield self.resolve_import(import_item) except ImportException as err: logging.info('unknown module %s', err.module_name) importlab-0.8.1/importlab/import_finder.py0000640000175000017500000001246014567445061020630 0ustar faunrisfaunris# NOTE: Do not add any dependencies to this file - it needs to be run in a # subprocess by a python version that might not have any installed packages, # including importlab itself. from __future__ import print_function import ast import json import os import sys # Pytype doesn't recognize the `major` attribute: # https://github.com/google/pytype/issues/127. if sys.version_info[0] >= 3: # Note that `import importlib` does not work: accessing `importlib.util` # will give an attribute error. This is hard to reproduce in a unit test but # can be seen by installing importlab in a Python 3 environment and running # `importlab --tree --trim` on a file that imports one of: # * jsonschema (`pip install jsonschema`) # * pytype (`pip install pytype`), # * dotenv (`pip install python-dotenv`) # * IPython (`pip install ipython`) # A correct output will look like: # Reading 1 files # Source tree: # + foo.py # :: jsonschema/__init__.py # An incorrect output will be missing the line with the import. import importlib.util else: import imp class ImportFinder(ast.NodeVisitor): """Walk an AST collecting import statements.""" def __init__(self): # tuples of (name, alias, is_from, is_star) self.imports = [] def visit_Import(self, node): for alias in node.names: self.imports.append((alias.name, alias.asname, False, False)) def visit_ImportFrom(self, node): module_name = '.'*node.level + (node.module or '') for alias in node.names: if alias.name == '*': self.imports.append((module_name, alias.asname, True, True)) else: if not module_name.endswith('.'): module_name = module_name + '.' name = module_name + alias.name asname = alias.asname or alias.name self.imports.append((name, asname, True, False)) def _find_package(parts): """Helper function for _resolve_import_versioned.""" for i in range(len(parts), 0, -1): prefix = '.'.join(parts[0:i]) if prefix in sys.modules: return i, sys.modules[prefix] return 0, None def is_builtin(name): return name in sys.builtin_module_names or name.startswith("__future__") # Pytype doesn't recognize the `major` attribute: # https://github.com/google/pytype/issues/127. if sys.version_info[0] < 3: def _resolve_import_versioned(name): """Python 2 helper function for resolve_import.""" parts = name.split('.') i, mod = _find_package(parts) if mod: if hasattr(mod, '__file__'): path = [os.path.dirname(mod.__file__)] elif hasattr(mod, '__path__'): path = mod.__path__ else: path = None else: path = None for part in parts[i:]: try: if path: spec = imp.find_module(part, [path]) else: spec = imp.find_module(part) except ImportError: return None path = spec[1] return path else: def _resolve_import_versioned(name): """Python 3 helper function for resolve_import.""" try: spec = importlib.util.find_spec(name) return spec and spec.origin except Exception: # find_spec may re-raise an arbitrary exception encountered while # inspecting a module. Since we aren't able to get the file path in # this case, we consider the import unresolved. return None def _resolve_import(name): """Helper function for resolve_import.""" if name in sys.modules: return getattr(sys.modules[name], '__file__', name + '.so') return _resolve_import_versioned(name) def resolve_import(name, is_from, is_star): """Use python to resolve an import. Args: name: The fully qualified module name. Returns: The path to the module source file or None. """ # Don't try to resolve relative imports or builtins here; they will be # handled by resolve.Resolver if name.startswith('.') or is_builtin(name): return None ret = _resolve_import(name) if ret is None and is_from and not is_star: package, _ = name.rsplit('.', 1) ret = _resolve_import(package) return ret def get_imports(filename): """Get all the imports in a file. Each import is a tuple of: (name, alias, is_from, is_star, source_file) """ with open(filename, "rb") as f: src = f.read() finder = ImportFinder() finder.visit(ast.parse(src, filename=filename)) imports = [] for i in finder.imports: name, _, is_from, is_star = i imports.append(i + (resolve_import(name, is_from, is_star),)) return imports def print_imports(filename): """Print imports in csv format to stdout.""" print(json.dumps(get_imports(filename))) def read_imports(imports_str): """Print imports in csv format to stdout.""" return json.loads(imports_str) if __name__ == "__main__": # This is used to parse a file with a different python version, launching a # subprocess and communicating with it via reading stdout. filename = sys.argv[1] print_imports(filename) importlab-0.8.1/importlab/parsepy.py0000640000175000017500000000665314567445061017461 0ustar faunrisfaunris# Copyright 2017 Google Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Logic for resolving import paths.""" import collections import logging import sys from . import import_finder from . import utils class ParseError(Exception): """Error parsing a file with python.""" pass class ImportStatement(collections.namedtuple( 'ImportStatement', ['name', 'new_name', 'is_from', 'is_star', 'source'])): """A Python import statement, such as "import foo as bar".""" def __new__(cls, name, new_name=None, is_from=False, is_star=False, source=None): """Create a new ImportStatement. Args: name: Name of the module to be imported. E.g. "sys". new_name: What the module is renamed to. The "y" in "import x as y". is_from: If the last part of the name (the "z" in "x.y.z") can be an element within a module, instead of a module itself. Happens e.g. for "from sys import argv". is_star: If this is an import of the form "from x import *". source: The path to the file as resolved by python. Returns: A new ImportStatement instance. """ return super(ImportStatement, cls).__new__( cls, name, new_name or name, is_from, is_star, source) def is_relative(self): return self.name.startswith('.') def __str__(self): if self.is_star: assert self.name == self.new_name assert self.is_from return 'from ' + self.name + ' import *' if self.is_from: try: left, right = self.name.rsplit('.', 2) except ValueError: left, right = self.name, '' module = left + '[.' + right + ']' else: module = self.name if self.new_name != self.name: return 'import ' + module + ' as ' + self.new_name else: return 'import ' + module def get_imports(filename, python_version): if python_version == sys.version_info[0:2]: # Invoke import_finder directly try: imports = import_finder.get_imports(filename) except Exception: raise ParseError(filename) else: # Call the appropriate python version in a subprocess f = sys.modules['importlab.import_finder'].__file__ if f.rsplit('.', 1)[-1] == 'pyc': # In host Python 2, importlab ships with .pyc files. f = f[:-1] ret, stdout, stderr = utils.run_py_file(python_version, f, filename) if not ret: if sys.version_info[0] == 3: stdout = stdout.decode('ascii') imports = import_finder.read_imports(stdout) else: if sys.version_info[0] == 3: stderr = stderr.decode('ascii') logging.info(stderr) raise ParseError(filename) return [ImportStatement(*imp) for imp in imports] importlab-0.8.1/importlab/graph.py0000640000175000017500000002136414567445061017073 0ustar faunrisfaunrisimport collections import os import networkx as nx from . import resolve from . import parsepy class NodeSet(object): """A strongly connected component - a set of mutually dependent files.""" def __init__(self, nodes): self.nodes = sorted(nodes) def __contains__(self, v): return v in self.nodes def pp(self): return '[' + '->'.join([str(f) for f in self.nodes]) + ']' def __str__(self): return self.pp() def __len__(self): return len(self.nodes) def __iter__(self): return self.nodes.__iter__() class DependencyGraph(object): """A set of file dependencies stored in a graph structure. The graph needs to be constructed in two phases: 1. Call add_file_recursive() for every root file to add to the graph. 2. Call build() to collapse cycles and build the final graph. Calling build() sets self.final = True and treats the graph as immutable thereafter. """ def __init__(self): self.graph = nx.DiGraph() # import statements that did not resolve to python files. self.broken_deps = collections.defaultdict(set) # files that were not syntactically valid python. self.unreadable_files = set() # once self.final is set the graph can no longer be modified. self.final = False # sources is a set of files directly added to the graph via # add_file or add_file_recursive. self.sources = set() # provenance is a map of file path (as stored in the graph) to where the # file was sourced from (see resolve.ResolvedFile) self.provenance = {} def get_file_deps(self, filename): raise NotImplementedError() def add_source_file(self, filename): self.sources.add(filename) self.provenance[filename] = self.get_source_file_provenance(filename) def get_source_file_provenance(self, filename): return resolve.Direct(filename) def add_file(self, filename): """Add a file and all its immediate dependencies to the graph.""" assert not self.final, 'Trying to mutate a final graph.' self.add_source_file(filename) resolved, unresolved = self.get_file_deps(filename) self.graph.add_node(filename) for f in resolved: self.graph.add_node(f) self.graph.add_edge(filename, f) for imp in unresolved: self.broken_deps[filename].add(imp) def follow_file(self, f, seen, trim): """Whether to recurse into a file's dependencies.""" return (f not in self.graph.nodes and f not in seen and (not trim or not isinstance(self.provenance[f], (resolve.Builtin, resolve.System)))) def add_file_recursive(self, filename, trim=False): """Add a file and all its recursive dependencies to the graph. Args: filename: The name of the file. trim: Whether to trim the dependencies of builtin and system files. """ assert not self.final, 'Trying to mutate a final graph.' self.add_source_file(filename) queue = collections.deque([filename]) seen = set() while queue: filename = queue.popleft() self.graph.add_node(filename) try: deps, broken = self.get_file_deps(filename) except parsepy.ParseError: # Python couldn't parse `filename`. If we're sure that it is a # Python file, we mark it as unreadable and keep the node in the # graph so importlab's callers can do their own syntax error # handling if desired. if os.path.splitext(filename)[1] in ('.py', '.so'): self.unreadable_files.add(filename) else: self.graph.remove_node(filename) continue for f in broken: self.broken_deps[filename].add(f) for f in deps: if self.follow_file(f, seen, trim): queue.append(f) seen.add(f) self.graph.add_node(f) if filename != f: # Prevent self edges if our dependency checker mistakenly # detects a module as its own direct dependency. self.graph.add_edge(filename, f) def shrink_to_node(self, scc): """Shrink a strongly connected component into a node.""" assert not self.final, 'Trying to mutate a final graph.' self.graph.add_node(scc) edges = list(self.graph.edges) for k, v in edges: if k not in scc and v in scc: self.graph.remove_edge(k, v) self.graph.add_edge(k, scc) elif k in scc and v not in scc: self.graph.remove_edge(k, v) self.graph.add_edge(scc, v) for node in scc.nodes: self.graph.remove_node(node) def format(self, node): if isinstance(node, NodeSet): return node.pp() else: return node def inspect_graph(self): keys = set(x[0] for x in self.graph.edges) for key in sorted(keys): k = self.format(key) for _, value in sorted(self.graph.edges([key])): v = self.format(value) print(' %s -> %s' % (k, v)) for value in sorted(self.broken_deps[key]): print(' %s -> <%s>' % (k, value)) def build(self): """Finalise the graph, after adding all input files to it.""" assert not self.final, 'Trying to mutate a final graph.' # Replace each strongly connected component with a single node `NodeSet` for scc in sorted(nx.kosaraju_strongly_connected_components(self.graph), key=len, reverse=True): if len(scc) == 1: break self.shrink_to_node(NodeSet(scc)) self.final = True def sorted_source_files(self): """Returns a list of targets in topologically sorted order.""" assert self.final, 'Call build() before using the graph.' out = [] for node in nx.topological_sort(self.graph): if isinstance(node, NodeSet): out.append(node.nodes) else: # add a one-element list for uniformity out.append([node]) return list(reversed(out)) def deps_list(self): """Returns a list of (target, dependencies).""" assert self.final, 'Call build() before using the graph.' out = [] for node in nx.topological_sort(self.graph): deps = [v for k, v in self.graph.out_edges([node])] out.append((node, deps)) return out def get_all_unresolved(self): """Returns a set of all unresolved imports.""" assert self.final, 'Call build() before using the graph.' out = set() for v in self.broken_deps.values(): out |= v return out class ImportGraph(DependencyGraph): """A dependency graph built from file imports.""" def __init__(self, env): super(ImportGraph, self).__init__() self.env = env self.path = env.path self.major_version = env.python_version[0] @classmethod def create(cls, env, filenames, trim=False): """Create and return a final graph. Args: env: An environment.Environment object filenames: A list of filenames trim: Whether to trim the dependencies of builtin and system files. Returns: An immutable ImportGraph with the recursive dependencies of all the files in filenames """ import_graph = cls(env) for filename in filenames: import_graph.add_file_recursive(os.path.abspath(filename), trim) import_graph.build() return import_graph def get_source_file_provenance(self, filename): """Infer the module name if possible.""" module_name = resolve.infer_module_name(filename, self.path) return resolve.Direct(filename, module_name) def get_file_deps(self, filename): resolved = [] unresolved = [] parent = self.provenance[filename] r = resolve.Resolver(self.path, parent) for imp in parsepy.get_imports(filename, self.env.python_version): try: f = r.resolve_import(imp) if isinstance(f, resolve.Builtin): continue full_path = os.path.abspath(f.path) resolved.append(full_path) self.provenance[full_path] = f except resolve.ImportException: unresolved.append(imp) return (resolved, unresolved) importlab-0.8.1/importlab/__init__.py0000640000175000017500000000000014567445061017511 0ustar faunrisfaunrisimportlab-0.8.1/importlab/utils.py0000640000175000017500000001100014567445061017114 0ustar faunrisfaunris"""Utility functions.""" from contextlib import contextmanager import errno import logging import os import shutil import subprocess import tempfile import textwrap def setup_logging(name, log_file, level=logging.INFO): formatter = logging.Formatter( fmt='%(asctime)s %(levelname)s %(message)s', datefmt='%Y-%m-%d %H:%M:%S') handler = logging.FileHandler(log_file) handler.setFormatter(formatter) logger = logging.getLogger(name) logger.setLevel(level) logger.addHandler(handler) return logger @contextmanager def cd(path): old = os.getcwd() os.chdir(os.path.expanduser(path)) try: yield finally: os.chdir(old) def expand_path(path, cwd=None): def expand(p): return os.path.realpath(os.path.expanduser(p)) if cwd: with cd(cwd): return expand(path) else: return expand(path) def expand_paths(paths, cwd=None): return [expand_path(x, cwd) for x in paths] def collect_files(path, extension): """Collect all the files with extension in a directory tree.""" # We should only call this on an actual directory; callers should do the # validation. assert os.path.isdir(path) out = [] # glob would be faster (see PEP471) but python glob doesn't do **/* for root, _, files in os.walk(path): out += [os.path.join(root, f) for f in files if f.endswith(extension)] return out def expand_source_files(filenames, cwd=None): """Expand a list of filenames passed in as sources. This is a helper function for handling command line arguments that specify a list of source files and directories. Any directories in filenames will be scanned recursively for .py files. Any files that do not end with ".py" will be dropped. Args: filenames: A list of filenames to process. cwd: An optional working directory to expand relative paths Returns: A list of sorted full paths to .py files """ out = [] for f in expand_paths(filenames, cwd): if os.path.isdir(f): # If we have a directory, collect all the .py files within it. out += collect_files(f, ".py") else: if f.endswith(".py"): out.append(f) return sorted(set(out)) def split_version(version): return tuple([int(v) for v in version.split('.')]) def makedirs(path): """Create a nested directory; don't fail if any of it already exists.""" try: os.makedirs(path) except OSError as e: if e.errno != errno.EEXIST: raise class Tempdir(object): """Context handler for creating temporary directories.""" def __enter__(self): self.setup() return self def setup(self): self.path = tempfile.mkdtemp() def create_directory(self, filename): """Create a subdirectory in the temporary directory.""" path = os.path.join(self.path, filename) makedirs(path) return path def create_file(self, filename, indented_data=None): """Create a file in the temporary directory. Dedents the contents. """ filedir, filename = os.path.split(filename) if filedir: self.create_directory(filedir) path = os.path.join(self.path, filedir, filename) data = indented_data if isinstance(data, bytes) and not isinstance(data, str): # This is binary data rather than text. mode = 'wb' else: mode = 'w' if data: data = textwrap.dedent(data) with open(path, mode) as fi: if data: fi.write(data) return path def delete_file(self, filename): os.unlink(os.path.join(self.path, filename)) def teardown(self): shutil.rmtree(path=self.path) def __exit__(self, error_type, value, tb): self.teardown() return False # reraise any exceptions def __getitem__(self, filename): """Get the full path for an entry in this directory.""" return os.path.join(self.path, filename) def strip_suffix(string, suffix): """Remove a suffix from a string if it exists.""" if string.endswith(suffix): return string[:-(len(suffix))] return string def run_py_file(version, path, *args): exe = 'python%d.%d' % version args = [exe, path] + list(args) p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) stdout, stderr = p.communicate() return p.returncode, stdout, stderr importlab-0.8.1/importlab/fs.py0000640000175000017500000001202414567445061016373 0ustar faunrisfaunrisimport abc import glob import os import tarfile import tempfile class FileSystemError(Exception): pass class FileSystem(abc.ABC): """Interface for file systems.""" @abc.abstractmethod def isfile(self, path): """Is this a file?""" pass @abc.abstractmethod def isdir(self, path): """Is this a directory?""" pass @abc.abstractmethod def read(self, path): """Read a file.""" pass @abc.abstractmethod def refer_to(self, path): """Get a fully qualified path for the given path.""" pass def relative_path(self, path): """Return the relative path to `path`. If this filesystem has a root directory, and path is within that directory tree, return the relative path; otherwise return None. """ return None class StoredFileSystem(FileSystem): """File system based on a file list.""" def __init__(self, files): self.files = files self.dirs = {os.path.dirname(f) for f in files} def isfile(self, path): return path in self.files def isdir(self, path): return path in self.dirs def read(self, path): return self.files[path] def refer_to(self, path): return path class OSFileSystem(FileSystem): """File system that uses an OS file system underneath.""" def __init__(self, root): assert root is not None self.root = root _, tmp_path = tempfile.mkstemp() self._is_case_insensitive = os.path.exists(tmp_path.upper()) def _join(self, path): return os.path.join(self.root, path) def _matches_path(self, path): if self._is_case_insensitive: return path in glob.glob(path+'*') return True def isfile(self, path): assert path is not None fullpath = self._join(path) return os.path.isfile(fullpath) and self._matches_path(fullpath) def isdir(self, path): assert path is not None fullpath = self._join(path) return os.path.isdir(fullpath) and self._matches_path(fullpath) def read(self, path): with open(self._join(path), 'r') as fi: return fi.read() def refer_to(self, path): return self._join(path) def relative_path(self, path): if path.startswith(self.root): return path[len(self.root) + 1:] return None class RemappingFileSystem(FileSystem, abc.ABC): """File system wrapper that transforms a path before looking it up.""" def __init__(self, underlying): self.underlying = underlying @abc.abstractmethod def map_path(self, path): pass def isfile(self, path): return self.underlying.isfile(self.map_path(path)) def isdir(self, path): return self.underlying.isdir(self.map_path(path)) def read(self, path): return self.underlying.read(self.map_path(path)) def refer_to(self, path): return self.underlying.refer_to(self.map_path(path)) class ExtensionRemappingFileSystem(RemappingFileSystem): """File system that remaps .py file extensions.""" def __init__(self, underlying, extension): super(ExtensionRemappingFileSystem, self).__init__(underlying) self.extension = extension def map_path(self, path): p, ext = os.path.splitext(path) if ext == '.py': return p + '.' + self.extension return path class PYIFileSystem(ExtensionRemappingFileSystem): """File system that remaps .py file extensions to pyi.""" def __init__(self, underlying): super(PYIFileSystem, self).__init__(underlying, 'pyi') class TarFileSystem(object): """Filesystem that serves files out of a .tar.""" def __init__(self, tar): self.tar = tar self.files = list(t.name for t in tar.getmembers() if t.isfile()) self.directories = list(t.name for t in tar.getmembers() if t.isdir()) self.top_level = {f.split(os.path.sep)[0] for f in self.files} def isfile(self, path): return any(os.path.join(top, path) in self.files for top in self.top_level) def isdir(self, path): return any(os.path.join(top, path) in self.files for top in self.top_level) def read(self, path): return self.tar.extractfile(path).read() def refer_to(self, path): return 'tar:' + path @staticmethod def read_tarfile(archive_filename): tar = tarfile.open(archive_filename) return TarFileSystem(tar) class Path(object): def __init__(self, paths=None): self.paths = paths if paths else [] def add_path(self, path, kind='os'): if kind == 'os': path = OSFileSystem(path) elif kind == 'pyi': path = PYIFileSystem(OSFileSystem(path)) else: raise FileSystemError('Unrecognized filesystem type: ', kind) self.paths.append(path) def add_fs(self, fs): assert isinstance(fs, FileSystem), 'Unrecognised filesystem: %r' % fs self.paths.append(fs) importlab-0.8.1/importlab/environment.py0000640000175000017500000000123214567445061020326 0ustar faunrisfaunrisimport os from . import utils from . import fs class Environment(object): def __init__(self, path, python_version): self.path = path.paths self.python_version = python_version def path_from_pythonpath(pythonpath): """Create an fs.Path object from a pythonpath string.""" path = fs.Path() for p in pythonpath.split(os.pathsep): path.add_path(utils.expand_path(p), 'os') return path def create_from_args(args): python_version_string = args.python_version python_version = utils.split_version(python_version_string) path = path_from_pythonpath(args.pythonpath) return Environment(path, python_version) importlab-0.8.1/importlab/output.py0000640000175000017500000000544214567445061017331 0ustar faunrisfaunrisfrom __future__ import print_function import networkx as nx from . import graph from . import resolve def inspect_graph(import_graph): keys = set(x[0] for x in import_graph.graph.edges) for key in sorted(keys): k = import_graph.format(key) for _, value in sorted(import_graph.graph.edges([key])): v = import_graph.format(value) print(' %s -> %s' % (k, v)) for value in sorted(import_graph.broken_deps[key]): print(' %s -> <%s>' % (k, value)) def format_file_node(import_graph, node, indent): """Prettyprint nodes based on their provenance.""" f = import_graph.provenance[node] if isinstance(f, resolve.Direct): out = '+ ' + f.short_path elif isinstance(f, resolve.Local): out = ' ' + f.short_path elif isinstance(f, resolve.System): out = ':: ' + f.short_path elif isinstance(f, resolve.Builtin): out = '(%s)' % f.module_name else: out = '%r' % node return ' '*indent + out def format_node(import_graph, node, indent): """Helper function for print_tree""" if isinstance(node, graph.NodeSet): ind = ' ' * indent out = [ind + 'cycle {'] + [ format_file_node(import_graph, n, indent + 1) for n in node.nodes ] + [ind + '}'] return '\n'.join(out) else: return format_file_node(import_graph, node, indent) def print_tree(import_graph): def _print_tree(root, indent=0): if root in seen: return seen.add(root) print(format_node(import_graph, root, indent)) for _, v in import_graph.graph.out_edges([root]): _print_tree(v, indent=indent+2) seen = set() for root in nx.topological_sort(import_graph.graph): if not import_graph.graph.in_edges([root]): _print_tree(root) def print_topological_sort(import_graph): for node in nx.topological_sort(import_graph.graph): print(import_graph.format(node)) def formatted_deps_list(import_graph): out = [] for node, deps in import_graph.deps_list(): out.append('source: ' + import_graph.format(node)) if deps: out.append('deps:') for dep in deps: out.append(' ' + import_graph.format(dep)) return '\n'.join(out) def print_unresolved_dependencies(import_graph): for imp in sorted(import_graph.get_all_unresolved()): print(' ', imp.name) def print_unreadable_files(import_graph): for f in sorted(import_graph.unreadable_files): print(' ', f) def maybe_show_unreadable(import_graph): """Only print an unreadable files section if nonempty.""" if import_graph.unreadable_files: print() print('Unreadable files:') print_unreadable_files(import_graph) importlab-0.8.1/CHANGELOG0000640000175000017500000000305414567445061014635 0ustar faunrisfaunrisVersion 0.8.1 (2023-10-06) * FIX: Filter out self-edges from the graph. Version 0.8 (2022-09-22) * Add --version to bin importlab. * Use ImportStatement.source to impove import resolution. Version 0.7 (2022-01-07) * Fix path resolution for case-insensitive filesystems. Version 0.6.1 (2021-01-14) * Include .so files in the dependency graph. * Drop support for running under Python 2 and modernize the code. * MANIFEST.in: Add tests to sdist Version 0.5.1 (2019-05-24) * FIX: Correct a broken import Version 0.5 (2018-12-26) * Support `from . import Symbol` style of relative imports * Set the default target Python version to the host version * Keep unreadable source files in the dependency graph * Allow non-.py files Version 0.4 (2018-11-02) * Add an option to trim the dependencies of builtin and system files Version 0.3.1 (2018-09-27) * FIX: Strip __init__ from inferred module names Version 0.3 (2018-09-07) * Allow inputs to be directories as well as files * Deal gracefully with unreadable source files (e.g. py3 files in a py2 tree) Version 0.2.1 (2018-08-16) * Add pytype and travis integration * Make all tests 2-and-3 compatible Version 0.2 (2018-06-06) * Track file provenance for all imports * Don't allow relative imports in non-packages Version 0.1.1 (2018-05-14) * Import graph with cycle detection and topological sorting * Add a broken dependency checker * Importlab works under python 2.7 and python 3.5+ * Invoke python to resolve imports from site packages * Remove dependencies on pytype and typeshed Version 0.1 (2017-11-17) * Initial import importlab-0.8.1/tests/0000755000175000017500000000000014567445061014567 5ustar faunrisfaunrisimportlab-0.8.1/tests/test_resolve.py0000640000175000017500000003373614567445061017667 0ustar faunrisfaunris"""Tests for resolve.py.""" import unittest from importlab import fs from importlab import parsepy from importlab import resolve from importlab import utils FILES = { "a.py": "contents of a", "b.py": "contents of b", "foo/c.py": "contents of c", "foo/d.py": "contents of d", "bar/e.py": "contents of e", "baz/__init__.py": "contents of init", "baz/f.py": "contents of f" } PYI_FILES = { "x.pyi": "contents of x", "y.pyi": "contents of y", } class TestResolver(unittest.TestCase): """Tests for Resolver.""" def setUp(self): self.py_fs = fs.StoredFileSystem(FILES) self.pyi_fs = fs.PYIFileSystem(fs.StoredFileSystem(PYI_FILES)) self.path = [self.pyi_fs, self.py_fs] def make_resolver(self, filename, module_name): module = resolve.Local(filename, module_name, self.py_fs) return resolve.Resolver(self.path, module) def testResolveWithFilesystem(self): imp = parsepy.ImportStatement("a") r = self.make_resolver("b.py", "b") f = r.resolve_import(imp) self.assertTrue(isinstance(f, resolve.Local)) self.assertEqual(f.fs, self.py_fs) self.assertEqual(f.path, "a.py") self.assertEqual(f.module_name, "a") def testResolveTopLevel(self): imp = parsepy.ImportStatement("a") r = self.make_resolver("b.py", "b") f = r.resolve_import(imp) self.assertEqual(f.path, "a.py") self.assertEqual(f.module_name, "a") def testResolvePackageFile(self): imp = parsepy.ImportStatement("foo.c") r = self.make_resolver("b.py", "b") f = r.resolve_import(imp) self.assertEqual(f.path, "foo/c.py") self.assertEqual(f.module_name, "foo.c") def testResolveSamePackageFile(self): imp = parsepy.ImportStatement(".c") r = self.make_resolver("foo/d.py", "foo.d") f = r.resolve_import(imp) self.assertEqual(f.path, "foo/c.py") def testResolveParentPackageFile(self): imp = parsepy.ImportStatement("..a") r = self.make_resolver("foo/d.py", "foo.d") f = r.resolve_import(imp) self.assertEqual(f.path, "a.py") self.assertTrue(isinstance(f, resolve.Local)) self.assertEqual(f.module_name, "..a") def testResolveParentPackageFileWithModule(self): imp = parsepy.ImportStatement("..a") r = self.make_resolver("foo/d.py", "bar.foo.d") f = r.resolve_import(imp) self.assertEqual(f.path, "a.py") self.assertTrue(isinstance(f, resolve.Local)) self.assertEqual(f.module_name, "bar.a") def testResolveSiblingPackageFile(self): # This is an invalid import, since we are trying to resolve a relative # import beyond the top-level package. The file resolver does not figure # out that we are moving beyond the top-level, but the module name does # not get qualified and remains relative. imp = parsepy.ImportStatement("..bar.e") r = self.make_resolver("foo/d.py", "foo.d") f = r.resolve_import(imp) self.assertEqual(f.path, "bar/e.py") self.assertEqual(f.module_name, "..bar.e") def testResolveInitFile(self): imp = parsepy.ImportStatement("baz") r = self.make_resolver("b.py", "b") f = r.resolve_import(imp) self.assertEqual(f.path, "baz/__init__.py") self.assertEqual(f.module_name, "baz") def testResolveInitFileRelative(self): imp = parsepy.ImportStatement("..baz") r = self.make_resolver("foo/d.py", "foo.d") f = r.resolve_import(imp) self.assertTrue(isinstance(f, resolve.Local)) self.assertEqual(f.path, "baz/__init__.py") self.assertEqual(f.module_name, "..baz") def testResolveRelativeFromInitFileWithModule(self): parent = resolve.Direct("baz/__init__.py", "baz") imp = parsepy.ImportStatement(".f") r = resolve.Resolver(self.path, parent) f = r.resolve_import(imp) self.assertTrue(isinstance(f, resolve.Local)) self.assertEqual(f.path, "baz/f.py") self.assertEqual(f.module_name, "baz.f") def testResolveRelativeSymbol(self): # importing the Symbol object from baz/__init__.py while in baz/f.py parent = resolve.Direct("baz/f.py", "baz.f") imp = parsepy.ImportStatement(".Symbol", is_from=True) r = resolve.Resolver(self.path, parent) f = r.resolve_import(imp) self.assertTrue(isinstance(f, resolve.Local)) self.assertEqual(f.path, "baz/__init__.py") self.assertEqual(f.module_name, "baz") def testResolveModuleFromFile(self): # from foo import c imp = parsepy.ImportStatement("foo.c", is_from=True) r = self.make_resolver("x.py", "x") f = r.resolve_import(imp) self.assertEqual(f.path, "foo/c.py") self.assertEqual(f.module_name, "foo.c") def testResolveSymbolFromFile(self): # from foo.c import X imp = parsepy.ImportStatement("foo.c.X", is_from=True) r = self.make_resolver("x.py", "x") f = r.resolve_import(imp) self.assertEqual(f.path, "foo/c.py") self.assertEqual(f.module_name, "foo.c") def testOverrideSource(self): imp = parsepy.ImportStatement("foo.c", source="/system/c.py") r = self.make_resolver("x.py", "x") f = r.resolve_import(imp) self.assertEqual(f.path, "foo/c.py") self.assertEqual(f.module_name, "foo.c") def testFallBackToSource(self): imp = parsepy.ImportStatement("f", source="/system/f.py") r = self.make_resolver("x.py", "x") f = r.resolve_import(imp) self.assertTrue(isinstance(f, resolve.System)) self.assertEqual(f.path, "/system/f.py") self.assertEqual(f.module_name, "f") def testResolveSystemSymbol(self): imp = parsepy.ImportStatement("argparse.ArgumentParser", source="/system/argparse.pyc", is_from=True) r = self.make_resolver("x.py", "x") f = r.resolve_import(imp) self.assertTrue(isinstance(f, resolve.System)) self.assertEqual(f.module_name, "argparse") def testResolveSystemSymbolNameClash(self): # from foo.foo import foo imp = parsepy.ImportStatement("foo.foo.foo", source="/system/bar/foo/foo.pyc", is_from=True) r = self.make_resolver("x.py", "x") f = r.resolve_import(imp) self.assertTrue(isinstance(f, resolve.System)) self.assertEqual(f.module_name, "foo.foo") def testResolveSystemFileNameClash(self): # `import a` in a.py should get the system a.py sys_file = "/system/a.py" imp = parsepy.ImportStatement("a", source=sys_file) r = self.make_resolver("a.py", "a") f = r.resolve_import(imp) self.assertTrue(isinstance(f, resolve.System)) self.assertEqual(f.path, sys_file) self.assertEqual(f.module_name, "a") def testResolveSystemInitFile(self): # from foo.foo import foo imp = parsepy.ImportStatement("foo.bar.X", source="/system/foo/bar/__init__.pyc", is_from=True) r = self.make_resolver("x.py", "x") f = r.resolve_import(imp) self.assertTrue(isinstance(f, resolve.System)) self.assertEqual(f.module_name, "foo.bar") def testResolveSystemPackageDir(self): with utils.Tempdir() as d: py_file = d.create_file("foo/__init__.py") imp = parsepy.ImportStatement("foo", source=d["foo"], is_from=True) r = self.make_resolver("x.py", "x") f = r.resolve_import(imp) self.assertTrue(isinstance(f, resolve.System)) self.assertEqual(f.module_name, "foo") self.assertEqual(f.path, py_file) def testGetPyFromPycSource(self): # Override a source pyc file with the corresponding py file if it exists # in the native filesystem. with utils.Tempdir() as d: py_file = d.create_file("f.py") pyc_file = d.create_file("f.pyc") imp = parsepy.ImportStatement("f", source=pyc_file) r = self.make_resolver("x.py", "x") f = r.resolve_import(imp) self.assertEqual(f.path, py_file) self.assertEqual(f.module_name, "f") def testPycSourceWithoutPy(self): imp = parsepy.ImportStatement("f", source="/system/f.pyc") r = self.make_resolver("x.py", "x") f = r.resolve_import(imp) self.assertEqual(f.path, "/system/f.pyc") self.assertEqual(f.module_name, "f") def testResolveBuiltin(self): imp = parsepy.ImportStatement("sys") r = self.make_resolver("x.py", "x") f = r.resolve_import(imp) self.assertTrue(isinstance(f, resolve.Builtin)) self.assertEqual(f.path, "sys.so") self.assertEqual(f.module_name, "sys") def testResolveStarImport(self): # from foo.c import * imp = parsepy.ImportStatement("foo.c", is_from=True, is_star=True) r = self.make_resolver("x.py", "x") f = r.resolve_import(imp) self.assertEqual(f.path, "foo/c.py") self.assertEqual(f.module_name, "foo.c") def testResolveStarImportBuiltin(self): imp = parsepy.ImportStatement("sys", is_from=True, is_star=True) r = self.make_resolver("x.py", "x") f = r.resolve_import(imp) self.assertTrue(isinstance(f, resolve.Builtin)) self.assertEqual(f.path, "sys.so") self.assertEqual(f.module_name, "sys") def testResolveStarImportSystem(self): imp = parsepy.ImportStatement("f", is_from=True, is_star=True, source="/system/f.py") r = self.make_resolver("x.py", "x") f = r.resolve_import(imp) self.assertEqual(f.path, "/system/f.py") self.assertEqual(f.module_name, "f") def testResolvePyiFile(self): imp = parsepy.ImportStatement("x") r = self.make_resolver("b.py", "b") f = r.resolve_import(imp) self.assertTrue(isinstance(f, resolve.Local)) self.assertEqual(f.fs, self.pyi_fs) self.assertEqual(f.path, "x.pyi") self.assertEqual(f.module_name, "x") def testResolveSystemRelative(self): with utils.Tempdir() as d: os_fs = fs.OSFileSystem(d.path) fspath = [os_fs] d.create_file("foo/x.py") d.create_file("foo/y.py") imp = parsepy.ImportStatement(".y") module = resolve.System(d["foo/x.py"], "foo.x") r = resolve.Resolver(fspath, module) f = r.resolve_import(imp) self.assertEqual(f.path, d["foo/y.py"]) self.assertTrue(isinstance(f, resolve.System)) self.assertEqual(f.module_name, "foo.y") def testResolveRelativeInNonPackage(self): r = self.make_resolver("a.py", "a") imp = parsepy.ImportStatement(".b", is_from=True) with self.assertRaises(resolve.ImportException): r.resolve_import(imp) def testResolveWithImportSource(self): for source, expected_resolution in [("z.py", "pyi"), ("z/__init__.py", "pyi"), ("z/zz.py", "system"), ("z/zz/__init__.py", "system")]: for pyi in ["z.pyi", "z/__init__.pyi"]: with self.subTest(source=source, pyi=pyi): pyis = {pyi: "contents of z", **PYI_FILES} pyi_fs = fs.PYIFileSystem(fs.StoredFileSystem(pyis)) self.path = [pyi_fs, self.py_fs] r = self.make_resolver("a.py", "a") imp = parsepy.ImportStatement( name="z.zz", new_name="zz", is_from=True, source=source) f = r.resolve_import(imp) if expected_resolution == "pyi": self.assertEqual(f.fs, pyi_fs) self.assertEqual(f.path, pyi) else: assert expected_resolution == "system" self.assertTrue(isinstance(f, resolve.System)) class TestResolverUtils(unittest.TestCase): """Tests for utility functions.""" def testInferModuleName(self): with utils.Tempdir() as d: os_fs = fs.OSFileSystem(d.path) fspath = [os_fs] py_file = d.create_file("foo/bar.py") self.assertEqual( resolve.infer_module_name(py_file, fspath), "foo.bar") # Standalone Python scripts often don't have extensions. self.assertEqual( resolve.infer_module_name(d["foo/baz"], fspath), "foo.baz") self.assertEqual( resolve.infer_module_name(d["random/src.py"], fspath), "random.src") self.assertEqual( resolve.infer_module_name("/some/random/file", fspath), "") def testInferInitModuleName(self): with utils.Tempdir() as d: os_fs = fs.OSFileSystem(d.path) fspath = [os_fs] py_file = d.create_file("foo/__init__.py") self.assertEqual( resolve.infer_module_name(py_file, fspath), "foo") def testGetAbsoluteName(self): test_cases = [ ("x.y", "a.b", "x.y.a.b"), ("", "a.b", "a.b"), ("x.y", ".a.b", "x.y.a.b"), ("x.y", "..a.b", "x.a.b"), ("x.y", "...a.b", "...a.b"), ] for parent, name, expected in test_cases: self.assertEqual( resolve.get_absolute_name(parent, name), expected) if __name__ == "__main__": unittest.main() importlab-0.8.1/tests/test_parsepy.py0000640000175000017500000001657014567445061017670 0ustar faunrisfaunris# Copyright 2017 Google Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Tests for parsepy.py.""" import tempfile import textwrap import unittest import sys from importlab import parsepy class TestParsePy(unittest.TestCase): """Tests for parsepy.py.""" def parse(self, src): with tempfile.NamedTemporaryFile() as f: src = textwrap.dedent(src) if sys.version_info[0] == 3: src = src.encode('utf-8') f.write(src) f.flush() return parsepy.get_imports(f.name, sys.version_info[:2]) def test_simple(self): self.assertEqual(self.parse(""" import a """), [parsepy.ImportStatement(name='a')]) def test_dotted(self): self.assertEqual(self.parse(""" import a.b """), [parsepy.ImportStatement(name='a.b')]) def test_as(self): self.assertEqual(self.parse(""" import a as b """), [parsepy.ImportStatement(name='a', new_name='b')]) def test_dotted_as(self): self.assertEqual(self.parse(""" import a.b as c """), [parsepy.ImportStatement(name='a.b', new_name='c')]) def test_dotted_comma(self): self.assertEqual(self.parse(""" import a.b, c """), [parsepy.ImportStatement(name='a.b'), parsepy.ImportStatement(name='c')]) def test_multiple_1(self): self.assertEqual(self.parse(""" import a, b, c, d """), [parsepy.ImportStatement(name='a'), parsepy.ImportStatement(name='b'), parsepy.ImportStatement(name='c'), parsepy.ImportStatement(name='d')]) def test_multiple_2(self): self.assertEqual(self.parse(""" import a, b as bb """), [parsepy.ImportStatement(name='a'), parsepy.ImportStatement(name='b', new_name='bb')]) def test_multiple_3(self): self.assertEqual(self.parse(""" import a, b as bb, a.x, a.x.y as a_x_y """), [parsepy.ImportStatement(name='a'), parsepy.ImportStatement(name='b', new_name='bb'), parsepy.ImportStatement(name='a.x', new_name='a.x'), parsepy.ImportStatement(name='a.x.y', new_name='a_x_y'), ]) def test_multiple_4(self): self.assertEqual(self.parse(""" import a import b import c import d """), [parsepy.ImportStatement(name='a'), parsepy.ImportStatement(name='b'), parsepy.ImportStatement(name='c'), parsepy.ImportStatement(name='d')]) def test_from(self): self.assertEqual(self.parse(""" from a import b """), [parsepy.ImportStatement(name='a.b', new_name='b', is_from=True)]) def test_from_with_rename(self): self.assertEqual(self.parse(""" from a import b as c """), [parsepy.ImportStatement(name='a.b', new_name='c', is_from=True)]) def test_dotted_from(self): self.assertEqual(self.parse(""" from a.b.c import d as e """), [parsepy.ImportStatement(name='a.b.c.d', new_name='e', is_from=True)]) def test_from_multiple(self): self.assertEqual(self.parse(""" from a import b, c, d as dd """), [parsepy.ImportStatement(name='a.b', new_name='b', is_from=True), parsepy.ImportStatement(name='a.c', new_name='c', is_from=True), parsepy.ImportStatement(name='a.d', new_name='dd', is_from=True)]) def test_from_parentheses(self): self.assertEqual(self.parse(""" from a import (b, c, d as dd) """), [parsepy.ImportStatement(name='a.b', new_name='b', is_from=True), parsepy.ImportStatement(name='a.c', new_name='c', is_from=True), parsepy.ImportStatement(name='a.d', new_name='dd', is_from=True)]) def test_asterisk(self): self.assertEqual(self.parse(""" from a import * from a.b import * from a . b . c import * """), [parsepy.ImportStatement(name='a', is_from=True, is_star=True), parsepy.ImportStatement(name='a.b', is_from=True, is_star=True), parsepy.ImportStatement( name='a.b.c', is_from=True, is_star=True)]) def test_dot(self): self.assertEqual(self.parse(""" from . import a from .a import b from .a.b import c from .a.b.c import * """), [parsepy.ImportStatement(name='.a', new_name='a', is_from=True), parsepy.ImportStatement(name='.a.b', new_name='b', is_from=True), parsepy.ImportStatement(name='.a.b.c', new_name='c', is_from=True), parsepy.ImportStatement( name='.a.b.c', is_from=True, is_star=True)]) def test_dotdot(self): self.assertEqual(self.parse(""" from .. import a from ..a import b from ..a.b import c from ..a.b.c import * """), [parsepy.ImportStatement(name='..a', new_name='a', is_from=True), parsepy.ImportStatement(name='..a.b', new_name='b', is_from=True), parsepy.ImportStatement(name='..a.b.c', new_name='c', is_from=True), parsepy.ImportStatement( name='..a.b.c', is_from=True, is_star=True)]) def test_dotdotdot_asterisk(self): self.assertEqual(self.parse(""" from ... import * """), [parsepy.ImportStatement(name='...', is_from=True, is_star=True)]) def test_dot_multiple(self): self.assertEqual(self.parse(""" from . import a, b, c as cc """), [parsepy.ImportStatement(name='.a', new_name='a', is_from=True), parsepy.ImportStatement(name='.b', new_name='b', is_from=True), parsepy.ImportStatement(name='.c', new_name='cc', is_from=True)]) def test_encoding_utf8(self): self.assertEqual(self.parse(""" # -*- coding: utf-8 -*- # Author: Lo\x6f\xc3\xafc Fooman import a """), [parsepy.ImportStatement(name='a')]) def test_encoding_latin1(self): self.assertEqual(self.parse(""" # -*- coding: iso-8859-1 -*- # Author: Thomas Sch\xf6n import a """), [parsepy.ImportStatement(name='a')]) def test_print_function(self): self.assertEqual(self.parse(""" from __future__ import print_function import sys print("hello world", file=sys.stdout) """), [parsepy.ImportStatement(name='__future__.print_function', new_name='print_function', is_from=True), parsepy.ImportStatement(name='sys')]) def test_non_utf8(self): """Verify that we can parse files with non-utf8 encoding.""" if sys.version_info[0] == 3: # TODO(mdemello): Get this working under python3 return src = ( b'# -*- coding: iso-8859-1 -*-\n' + b'# Copyright (C) 1984 F' + chr(0xf6) + b'man\n' ) self.assertRaises(UnicodeDecodeError, unicode, src) # noqa: F821 self.assertEqual(self.parse(src), []) def test_syntax_error(self): with self.assertRaises(parsepy.ParseError): self.parse("foo(]") if __name__ == '__main__': unittest.main() importlab-0.8.1/tests/test_utils.py0000640000175000017500000000150514567445061017335 0ustar faunrisfaunris"""Tests for utils.py.""" import sys import tempfile import unittest from importlab import utils class TestUtils(unittest.TestCase): """Tests for utils.""" def test_strip_suffix(self): a = 'foo bar bar' self.assertEqual(a, utils.strip_suffix(a, 'hello')) self.assertEqual('foo bar ', utils.strip_suffix(a, 'bar')) self.assertEqual(a, utils.strip_suffix(a, 'hbar')) def test_run_py_file(self): version = sys.version_info[:2] with tempfile.NamedTemporaryFile(mode='w') as f: f.write('print("test")') f.flush() ret, stdout, stderr = utils.run_py_file(version, f.name) self.assertFalse(ret) self.assertEqual(stdout.strip().decode(), 'test') self.assertFalse(stderr) if __name__ == "__main__": unittest.main() importlab-0.8.1/tests/test_import_finder.py0000640000175000017500000000140314567445061021033 0ustar faunrisfaunris"""Tests for import_finder.py.""" import sys import unittest from importlab import import_finder class TestImportFinder(unittest.TestCase): """Tests for import_finder.""" def test_find_submodule(self): # networkx should always be findable because importlab uses it. name = 'networkx.algorithms.cluster' self.assertIsNotNone(import_finder.resolve_import(name, True, False)) @unittest.skipIf(sys.version_info[0] == 2, 'py2 uses imp, not importlib') def test_importlib_exception(self): from unittest import mock with mock.patch('importlib.util.find_spec', side_effect=AssertionError): self.assertIsNone(import_finder.resolve_import('', False, False)) if __name__ == '__main__': unittest.main() importlab-0.8.1/tests/run_all.sh0000750000175000017500000000041114567445061016551 0ustar faunrisfaunris# This script must be run from the directory above tests. set -ev python -m tests.test_fs python -m tests.test_graph python -m tests.test_import_finder python -m tests.test_output python -m tests.test_parsepy python -m tests.test_resolve python -m tests.test_utils importlab-0.8.1/tests/__init__.py0000640000175000017500000000000014567445061016662 0ustar faunrisfaunrisimportlab-0.8.1/tests/test_output.py0000640000175000017500000000401514567445061017534 0ustar faunrisfaunris"""Tests for output.py.""" import contextlib import io import sys import unittest from importlab import environment from importlab import fs from importlab import graph from importlab import output from importlab import utils FILES = { "foo/a.py": "from . import b", "foo/b.py": "pass", "x.py": "import foo.a", "y.py": "import sys", "z.py": "import unresolved" } # For Python 2 compatibility, since contextlib.redirect_stdout is 3-only. @contextlib.contextmanager def redirect_stdout(out): old = sys.stdout sys.stdout = out try: yield finally: sys.stdout = old class TestOutput(unittest.TestCase): """Basic sanity tests for output methods.""" def setUp(self): self.tempdir = utils.Tempdir() self.tempdir.setup() filenames = [ self.tempdir.create_file(f, FILES[f]) for f in FILES] self.fs = fs.OSFileSystem(self.tempdir.path) env = environment.Environment(fs.Path([self.fs]), sys.version_info[:2]) self.graph = graph.ImportGraph.create(env, filenames) def tearDown(self): self.tempdir.teardown() def assertString(self, val): if sys.version_info[0] == 3: self.assertTrue(isinstance(val, str)) else: self.assertTrue(isinstance(val, (str, unicode))) # noqa: F821 def assertPrints(self, fn): out = io.StringIO() with redirect_stdout(out): fn(self.graph) self.assertTrue(out.getvalue()) def test_inspect_graph(self): self.assertPrints(output.inspect_graph) def test_print_tree(self): self.assertPrints(output.print_tree) def test_print_topological_sort(self): self.assertPrints(output.print_topological_sort) def test_formatted_deps_list(self): self.assertString(output.formatted_deps_list(self.graph)) def test_print_unresolved(self): self.assertPrints(output.print_unresolved_dependencies) if __name__ == "__main__": unittest.main() importlab-0.8.1/tests/test_fs.py0000640000175000017500000000731414567445061016611 0ustar faunrisfaunris"""Tests for fs.py.""" import unittest from importlab import fs from importlab import utils FILES = { "a.py": "contents of a", "b.py": "contents of b", "foo/c.py": "contents of c", "foo/d.py": "contents of d", "bar/e.py": "contents of e" } class TestStoredFileSystem(unittest.TestCase): """Tests for StoredFileSystem.""" def setUp(self): self.fs = fs.StoredFileSystem(FILES) def testIsFile(self): self.assertTrue(self.fs.isfile("a.py")) self.assertTrue(self.fs.isfile("foo/c.py")) self.assertFalse(self.fs.isfile("foo/b.py")) def testIsDir(self): self.assertTrue(self.fs.isdir("foo")) self.assertTrue(self.fs.isdir("")) self.assertFalse(self.fs.isdir("foo/c.py")) self.assertFalse(self.fs.isdir("a.py")) def testNoTrivialEmptyDir(self): f = fs.StoredFileSystem({"foo/a.py": True, "bar/b.py": True}) self.assertTrue(f.isdir("foo")) self.assertTrue(f.isdir("bar")) self.assertFalse(f.isdir("")) class TestOSFileSystem(unittest.TestCase): """Tests for OSFileSystem.""" def setUp(self): self.tempdir = utils.Tempdir() self.tempdir.setup() for f in FILES: self.tempdir.create_file(f, FILES[f]) self.fs = fs.OSFileSystem(self.tempdir.path) def tearDown(self): self.tempdir.teardown() def testIsFile(self): self.assertTrue(self.fs.isfile("a.py")) self.assertTrue(self.fs.isfile("foo/c.py")) self.assertFalse(self.fs.isfile("foo/b.py")) def testIsDir(self): self.assertTrue(self.fs.isdir("foo")) self.assertTrue(self.fs.isdir("")) self.assertFalse(self.fs.isdir("foo/c.py")) self.assertFalse(self.fs.isdir("a.py")) class LowercasingFileSystem(fs.RemappingFileSystem): """Remapping file system subclass for tests.""" def map_path(self, path): return path.lower() class TestRemappingFileSystem(unittest.TestCase): """Tests for RemappingFileSystem.""" def setUp(self): self.tempdir = utils.Tempdir() self.tempdir.setup() for f in FILES: self.tempdir.create_file(f, FILES[f]) self.fs = LowercasingFileSystem( fs.OSFileSystem(self.tempdir.path)) def tearDown(self): self.tempdir.teardown() def testIsFile(self): self.assertTrue(self.fs.isfile("A.py")) self.assertTrue(self.fs.isfile("FOO/c.py")) self.assertFalse(self.fs.isfile("foO/B.py")) def testIsDir(self): self.assertTrue(self.fs.isdir("fOO")) self.assertTrue(self.fs.isdir("")) self.assertFalse(self.fs.isdir("FOO/C.PY")) self.assertFalse(self.fs.isdir("a.PY")) class TestPYIFileSystem(unittest.TestCase): """Tests for PYIFileSystem (also tests ExtensionRemappingFileSystem).""" def setUp(self): self.tempdir = utils.Tempdir() self.tempdir.setup() for f in FILES: self.tempdir.create_file(f + "i", FILES[f]) self.fs = fs.PYIFileSystem( fs.OSFileSystem(self.tempdir.path)) def tearDown(self): self.tempdir.teardown() def testIsFile(self): self.assertTrue(self.fs.isfile("a.py")) self.assertTrue(self.fs.isfile("foo/c.py")) self.assertFalse(self.fs.isfile("foo/b.py")) def testIsDir(self): self.assertTrue(self.fs.isdir("foo")) self.assertTrue(self.fs.isdir("")) self.assertFalse(self.fs.isdir("foo/c.py")) self.assertFalse(self.fs.isdir("a.py")) def testFullPath(self): self.assertEqual(self.fs.refer_to("foo/c.py"), self.tempdir["foo/c.pyi"]) if __name__ == "__main__": unittest.main() importlab-0.8.1/tests/test_graph.py0000640000175000017500000002532114567445061017300 0ustar faunrisfaunris"""Tests for graph.py.""" import contextlib import os import sys import unittest from importlab import environment from importlab import fs from importlab import graph from importlab import parsepy from importlab import resolve from importlab import utils class FakeImportGraph(graph.DependencyGraph): """An ImportGraph with file imports stubbed out. Also adds ordered_foo() wrappers around output methods to help in testing. """ def __init__(self, deps, unreadable=None): super(FakeImportGraph, self).__init__() self.deps = deps if unreadable: self.unreadable = unreadable else: self.unreadable = set() def get_source_file_provenance(self, filename): return resolve.Direct(filename, "module.name") def get_file_deps(self, filename): if filename in self.unreadable: raise parsepy.ParseError() if filename in self.deps: resolved, unresolved, provenance = self.deps[filename] self.provenance.update(provenance) return (resolved, unresolved) return ([], []) def ordered_deps_list(self): deps = [] for k, v in self.deps_list(): deps.append((k, sorted(v))) return list(sorted(deps)) def ordered_sorted_source_files(self): return [list(sorted(x)) for x in self.sorted_source_files()] # Deps = { file : ([resolved deps], [broken deps], {dep_file:provenance}) } SIMPLE_DEPS = { "a.py": (["b.py", "c.py"], [], {"b.py": resolve.Local("b.py", "b", "fs1"), "c.py": resolve.Local("c.py", "c", "fs2") }), "b.py": (["d.py"], ["e"], {"d.py": resolve.System("d.py", "d")}) } SIMPLE_NONPY_DEPS = {"a.pyi": (["b.py"], [], {})} SIMPLE_CYCLIC_DEPS = { "a.py": (["b.py", "c.py"], ["e"], {}), "b.py": (["d.py", "a.py"], ["f"], {}), } SELF_DEPS = { "a.py": (["a.py", "b.py", "c.py"], [], {}), "b.py": (["a.py", "b.py", "d.py"], [], {}), } SIMPLE_SYSTEM_DEPS = { "a.py": (["b.py"], [], {"b.py": resolve.System("b.py", "b")}), "b.py": (["c.py"], [], {"c.py": resolve.System("c.py", "c")}), } class TestDependencyGraph(unittest.TestCase): """Tests for DependencyGraph.""" def check_order(self, xs, *args): """Checks that args form an increasing sequence within xs.""" indices = [xs.index(arg) for arg in args] for i in range(1, len(indices)): self.assertTrue(indices[i - 1] < indices[i], "%s comes before %s" % (args[i], args[i - 1])) def test_simple(self): g = FakeImportGraph(SIMPLE_DEPS) g.add_file_recursive("a.py") g.build() self.assertEqual(g.ordered_deps_list(), [ ("a.py", ["b.py", "c.py"]), ("b.py", ["d.py"]), ("c.py", []), ("d.py", [])]) self.assertEqual(g.get_all_unresolved(), set(["e"])) sources = g.ordered_sorted_source_files() self.check_order(sources, ["d.py"], ["b.py"], ["a.py"]) self.check_order(sources, ["c.py"], ["a.py"]) self.assertEqual(sorted(g.provenance.keys()), ["a.py", "b.py", "c.py", "d.py"]) # a.py is a directly added source provenance = g.provenance["a.py"] self.assertTrue(isinstance(provenance, resolve.Direct)) self.assertEqual(provenance.module_name, "module.name") # b.py came from fs1 self.assertEqual(g.provenance["b.py"].fs, "fs1") def test_simple_cycle(self): g = FakeImportGraph(SIMPLE_CYCLIC_DEPS) g.add_file_recursive("a.py") g.build() cycles = [x for x, ys in g.deps_list() if isinstance(x, graph.NodeSet)] self.assertEqual(len(cycles), 1) self.assertEqual(set(cycles[0].nodes), set(["a.py", "b.py"])) self.assertEqual(g.get_all_unresolved(), set(["e", "f"])) sources = g.ordered_sorted_source_files() self.check_order(sources, ["d.py"], ["a.py", "b.py"]) self.check_order(sources, ["c.py"], ["a.py", "b.py"]) def test_self_dep(self): g = FakeImportGraph(SELF_DEPS) g.add_file_recursive("a.py") g.build() cycles = [x for x, ys in g.deps_list() if isinstance(x, graph.NodeSet)] self.assertEqual(len(cycles), 1) self.assertEqual(set(cycles[0].nodes), set(["a.py", "b.py"])) sources = g.ordered_sorted_source_files() self.check_order(sources, ["d.py"], ["a.py", "b.py"]) self.check_order(sources, ["c.py"], ["a.py", "b.py"]) def test_trim(self): # Untrimmed g1 follows system module b to its dependency c. g1 = FakeImportGraph(SIMPLE_SYSTEM_DEPS) g1.add_file_recursive("a.py", trim=False) g1.build() self.assertEqual(g1.ordered_deps_list(), [ ("a.py", ["b.py"]), ("b.py", ["c.py"]), ("c.py", [])]) # Trimmed g2 stops at b. g2 = FakeImportGraph(SIMPLE_SYSTEM_DEPS) g2.add_file_recursive("a.py", trim=True) g2.build() self.assertEqual(g2.ordered_deps_list(), [ ("a.py", ["b.py"]), ("b.py", [])]) def test_unreadable(self): # Unreadable py files are kept in the graph to give the caller # flexibility on what to do with them. g = FakeImportGraph(SIMPLE_DEPS, unreadable={"b.py"}) g.add_file_recursive("a.py") g.build() self.assertEqual(g.ordered_deps_list(), [ ("a.py", ["b.py", "c.py"]), ("b.py", []), ("c.py", []), ]) sources = g.ordered_sorted_source_files() self.check_order(sources, ["c.py"], ["a.py"]) self.assertEqual(sorted(g.provenance), ["a.py", "b.py", "c.py"]) self.assertEqual(g.unreadable_files, set(["b.py"])) def test_unreadable_direct_source(self): # Unreadable py files are kept in the graph to give the caller # flexibility on what to do with them. g = FakeImportGraph(SIMPLE_DEPS, unreadable={"a.py"}) g.add_file_recursive("a.py") g.build() self.assertEqual(g.ordered_deps_list(), [("a.py", [])]) def test_readable_nonpy(self): g = FakeImportGraph(SIMPLE_NONPY_DEPS) g.add_file_recursive("a.pyi") g.build() self.assertEqual(g.ordered_deps_list(), [ ("a.pyi", ["b.py"]), ("b.py", []), ]) def test_unreadable_nonpy(self): g = FakeImportGraph(SIMPLE_NONPY_DEPS, unreadable={"a.pyi"}) g.add_file_recursive("a.pyi") g.build() # Original source file is unreadable, so return nothing. self.assertEqual(g.ordered_deps_list(), []) FILES = { "foo/a.py": "from . import b", "foo/b.py": "pass", "x.py": "import foo.a" } class TestImportGraph(unittest.TestCase): """Tests for ImportGraph.""" def setUp(self): self.tempdir = utils.Tempdir() self.tempdir.setup() self.filenames = [ self.tempdir.create_file(f, FILES[f]) for f in FILES] self.fs = fs.OSFileSystem(self.tempdir.path) self.env = environment.Environment( fs.Path([self.fs]), sys.version_info[:2]) def tearDown(self): self.tempdir.teardown() def test_basic(self): g = graph.ImportGraph.create(self.env, self.filenames) self.assertEqual( g.sorted_source_files(), [[self.tempdir[x]] for x in ["foo/b.py", "foo/a.py", "x.py"]]) @contextlib.contextmanager def patch_resolve_import(self, mock_resolve_file): """Patch resolve_import to always return a System file.""" resolve_import = resolve.Resolver.resolve_import def mock_resolve_import(resolver_self, item): resolved_file = resolve_import(resolver_self, item) return mock_resolve_file(resolved_file) resolve.Resolver.resolve_import = mock_resolve_import try: yield finally: resolve.Resolver.resolve_import = resolve_import def test_trim(self): sources = [self.tempdir["x.py"]] mock_resolve_file = lambda f: resolve.System(f.path, f.module_name) with self.patch_resolve_import(mock_resolve_file): # Untrimmed g1 contains foo.b, the dep of system module foo.a. g1 = graph.ImportGraph.create(self.env, sources, trim=False) self.assertEqual( g1.sorted_source_files(), [[self.tempdir[x]] for x in ["foo/b.py", "foo/a.py", "x.py"]]) # Trimmed g2 stops at foo.a. g2 = graph.ImportGraph.create(self.env, sources, trim=True) self.assertEqual( g2.sorted_source_files(), [[self.tempdir[x]] for x in ["foo/a.py", "x.py"]]) def test_system_extension(self): """Tests that system .so files are included in deps.""" sources = [self.tempdir["x.py"]] def mock_resolve_file(f): path = os.path.splitext(f.path)[0] + ".so" return resolve.System(path, f.module_name) with self.patch_resolve_import(mock_resolve_file): g = graph.ImportGraph.create(self.env, sources, trim=True) foo_a = os.path.splitext(self.tempdir["foo/a.py"])[0] + ".so" self.assertEqual(g.sorted_source_files(), [[foo_a], [self.tempdir["x.py"]]]) def test_builtin_extension(self): """Tests that builtin .so files are ignored.""" sources = [self.tempdir["x.py"]] def mock_resolve_file(f): path = os.path.splitext(f.path)[0] + ".so" return resolve.Builtin(path, f.module_name) with self.patch_resolve_import(mock_resolve_file): g = graph.ImportGraph.create(self.env, sources, trim=True) foo_a = os.path.splitext(self.tempdir["foo/a.py"])[0] + ".so" self.assertEqual(g.sorted_source_files(), [[self.tempdir["x.py"]]]) def test_system_extension_notrim(self): """Tests that failing to descend into a .so file's deps is ok.""" sources = [self.tempdir["x.py"]] def mock_resolve_file(f): path = os.path.splitext(f.path)[0] + ".so" return resolve.System(path, f.module_name) with open(self.tempdir["foo/a.py"], "w") as f: f.write("syntax_error:") # simulate an unparseable .so file with self.patch_resolve_import(mock_resolve_file): g = graph.ImportGraph.create(self.env, sources, trim=False) foo_a = os.path.splitext(self.tempdir["foo/a.py"])[0] + ".so" self.assertEqual(g.sorted_source_files(), [[foo_a], [self.tempdir["x.py"]]]) if __name__ == "__main__": unittest.main() importlab-0.8.1/MANIFEST.in0000640000175000017500000000015414567445061015157 0ustar faunrisfaunrisinclude LICENSE include CHANGELOG include *.md *.rst recursive-include tests *.py run_all.sh graft testdata importlab-0.8.1/setup.py0000640000175000017500000000317414567445061015140 0ustar faunrisfaunris#!/usr/bin/env python # -*- coding: utf-8 -*- import io import os from setuptools import find_packages, setup # pytype: disable=import-error # Package meta-data. NAME = 'importlab' DESCRIPTION = 'A library to calculate python dependency graphs.' URL = 'https://github.com/google/importlab' EMAIL = 'pytype-dev@google.com' AUTHOR = 'Google Inc.' REQUIRES_PYTHON = '>=3.6.0' VERSION = '0.8.1' REQUIRED = [ 'networkx>=2', ] here = os.path.abspath(os.path.dirname(__file__)) # Import the README and use it as the long-description. with io.open(os.path.join(here, 'README.rst'), encoding='utf-8') as f: long_description = '\n' + f.read() # Load the package's __version__.py module as a dictionary. about = {} if not VERSION: with open(os.path.join(here, NAME, '__version__.py')) as f: exec(f.read(), about) else: about['__version__'] = VERSION PACKAGES = find_packages(exclude=('tests',)) setup( name=NAME, version=about['__version__'], description=DESCRIPTION, long_description=long_description, maintainer=AUTHOR, maintainer_email=EMAIL, python_requires=REQUIRES_PYTHON, url=URL, packages=PACKAGES, scripts=['bin/importlab'], install_requires=REQUIRED, include_package_data=True, license='Apache 2.0', classifiers=[ 'Development Status :: 4 - Beta', 'Intended Audience :: Developers', 'License :: OSI Approved :: Apache Software License', 'Programming Language :: Python', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: Implementation :: CPython', 'Topic :: Software Development', ], ) importlab-0.8.1/CONTRIBUTING.md0000640000175000017500000000522214567445061015653 0ustar faunrisfaunrisWant to contribute? Great! First, read this page (including the small print at the end). ### Before you contribute Before we can use your code, you must sign the [Google Individual Contributor License Agreement](https://developers.google.com/open-source/cla/individual?csw=1) (CLA), which you can do online. The CLA is necessary mainly because you own the copyright to your changes, even after your contribution becomes part of our codebase, so we need your permission to use and distribute your code. We also need to be sure of various other things -- for instance that you'll tell us if you know that your code infringes on other people's patents. You don't have to sign the CLA until after you've submitted your code for review and a member has approved it, but you must do it before we can put your code into our codebase. Before you start working on a larger contribution, you should get in touch with us first through the issue tracker with your idea so that we can help out and possibly guide you. Coordinating up front makes it much easier to avoid frustration later on. ### Code reviews All submissions, including submissions by project members, require review. We use Github pull requests for this purpose. ### Releasing to PyPI To release to PyPI: 1. (Optional) We recommend that you release from within a virtualenv: ```console $ python3 -m venv .venv_release $ source .venv_release/bin/activate ``` 1. Make sure that `wheel` and `twine` are installed: ```console $ pip install wheel twine ``` 1. Navigate into the top-level `importlab` directory and build a source distribution and a wheel: ```console $ cd importlab $ python3 setup.py sdist bdist_wheel ``` The build command puts the distributions in a `dist` subdirectory and also creates `build` and `importlab.egginfo` subdirectories as side effects. 1. Upload the distributions to (Test)PyPI: ```console $ twine upload --repository testpypi dist/* ``` Remove the `--repository testpypi` to upload to PyPI proper. 1. (Optional) If you've uploaded to TestPyPI, you can install your new version for testing like so: ```console $ pip install -U --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple importlab ``` If you've uploaded to PyPI proper, install with `pip install -U importlab` as usual. 1. Clean up the subdirectories created by the build command: ```console $ rm -rf build/ dist/ importlab.egg-info/ ``` ### The small print Contributions made by corporations are covered by a different agreement than the one mentioned above; they're covered by the the Software Grant and Corporate Contributor License Agreement. importlab-0.8.1/setup.cfg0000640000175000017500000000037514567445061015247 0ustar faunrisfaunris[metadata] license_file = LICENSE [bdist_wheel] universal = 1 [flake8] max-line-length = 80 exclude = .[!.]* build/ dist/ testdata/ [pytype] inputs = . exclude = tests/ testdata/ disable = pyi-error [egg_info] tag_build = tag_date = 0 importlab-0.8.1/README.rst0000640000175000017500000000321714567445061015113 0ustar faunrisfaunrisimportlab --------- Importlab is a library for Python that automatically infers dependencies and calculates a dependency graph. It can perform dependency ordering of a set of files, including cycle detection. Importlab's main use case is to work with static analysis tools that process one file at a time, ensuring that a file's dependencies are analysed before it is. (This is not an official Google product.) License ------- Apache 2.0 Installation ------------ Importlab can be installed from pip :: pip install importlab To check out and install the latest source code :: git clone https://github.com/google/importlab.git cd importlab python setup.py install Usage ----- Importlab is primarily intended to be used as a library. It takes one or more python files as arguments, and generates an import graph, typically used to process files in dependency order. It is currently integrated into `pytype `__ Command-line tool ----------------- Importlab ships with a small command-line tool, also called ``importlab``, which can display some information about a project's import graph. :: usage: importlab [-h] [--tree] [--unresolved] [filename [filename ...]] positional arguments: filename input file(s) optional arguments: -h, --help show this help message and exit --tree Display import tree. --unresolved Display unresolved dependencies. Roadmap ------- - ``Makefile`` generation, to take advantage of ``make``'s incremental update and parallel execution features - Integration with other static analysis tools importlab-0.8.1/importlab.egg-info/0000755000175000017500000000000014567445061017110 5ustar faunrisfaunrisimportlab-0.8.1/importlab.egg-info/PKG-INFO0000644000175000017500000000442514567445061020212 0ustar faunrisfaunrisMetadata-Version: 2.1 Name: importlab Version: 0.8.1 Summary: A library to calculate python dependency graphs. Home-page: https://github.com/google/importlab Maintainer: Google Inc. Maintainer-email: pytype-dev@google.com License: Apache 2.0 Classifier: Development Status :: 4 - Beta Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: Apache Software License Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: Implementation :: CPython Classifier: Topic :: Software Development Requires-Python: >=3.6.0 License-File: LICENSE importlab --------- Importlab is a library for Python that automatically infers dependencies and calculates a dependency graph. It can perform dependency ordering of a set of files, including cycle detection. Importlab's main use case is to work with static analysis tools that process one file at a time, ensuring that a file's dependencies are analysed before it is. (This is not an official Google product.) License ------- Apache 2.0 Installation ------------ Importlab can be installed from pip :: pip install importlab To check out and install the latest source code :: git clone https://github.com/google/importlab.git cd importlab python setup.py install Usage ----- Importlab is primarily intended to be used as a library. It takes one or more python files as arguments, and generates an import graph, typically used to process files in dependency order. It is currently integrated into `pytype `__ Command-line tool ----------------- Importlab ships with a small command-line tool, also called ``importlab``, which can display some information about a project's import graph. :: usage: importlab [-h] [--tree] [--unresolved] [filename [filename ...]] positional arguments: filename input file(s) optional arguments: -h, --help show this help message and exit --tree Display import tree. --unresolved Display unresolved dependencies. Roadmap ------- - ``Makefile`` generation, to take advantage of ``make``'s incremental update and parallel execution features - Integration with other static analysis tools importlab-0.8.1/importlab.egg-info/SOURCES.txt0000640000175000017500000000131514567445061020770 0ustar faunrisfaunrisCHANGELOG CONTRIBUTING.md LICENSE MANIFEST.in README.rst setup.cfg setup.py bin/importlab importlab/__init__.py importlab/environment.py importlab/fs.py importlab/graph.py importlab/import_finder.py importlab/output.py importlab/parsepy.py importlab/resolve.py importlab/utils.py importlab.egg-info/PKG-INFO importlab.egg-info/SOURCES.txt importlab.egg-info/dependency_links.txt importlab.egg-info/requires.txt importlab.egg-info/top_level.txt testdata/test.py testdata/pkg/a.py testdata/pkg/b.py testdata/pkg/c.py testdata/pkg/d.py tests/__init__.py tests/run_all.sh tests/test_fs.py tests/test_graph.py tests/test_import_finder.py tests/test_output.py tests/test_parsepy.py tests/test_resolve.py tests/test_utils.pyimportlab-0.8.1/importlab.egg-info/requires.txt0000640000175000017500000000001414567445061021477 0ustar faunrisfaunrisnetworkx>=2 importlab-0.8.1/importlab.egg-info/top_level.txt0000640000175000017500000000001214567445061021627 0ustar faunrisfaunrisimportlab importlab-0.8.1/importlab.egg-info/dependency_links.txt0000640000175000017500000000000114567445061023152 0ustar faunrisfaunris