pytest-multihost-3.4/0000775004305200430520000000000013643072414015606 5ustar spoorespoore00000000000000pytest-multihost-3.4/PKG-INFO0000664004305200430520000002173313643072414016711 0ustar spoorespoore00000000000000Metadata-Version: 1.1 Name: pytest-multihost Version: 3.4 Summary: Utility for writing multi-host tests for pytest Home-page: https://pagure.io/python-pytest-multihost Author: Petr Viktorin Author-email: pviktori@redhat.com License: GPL Description: A pytest plugin for multi-host testing. Downloading ----------- Release tarballs will be made available for download from Pagure Releases: https://pagure.io/releases/python-pytest-multihost/ The goal is to include this project in Fedora repositories. Until that happens, you can use testing builds from COPR – see "Developer links" below. You can also install using pip: https://pypi.python.org/pypi/pytest-multihost Usage ----- This plugin takes a description of your infrastructure, and provides, via a fixture, Host objects that commands can be called on. It is intended as a general base for a framework; any project using it will need to extend it for its own needs. The object provided to tests is a Config object, which has (among others) these attributes:: test_dir – directory to store test-specific data in, defaults to /root/multihost_tests ipv6 – true if connecting via IPv6 domains – the list of domains Hosts to run on are arranged in domains, which have:: name – the DNS name of the domain type – a string specifying the type of the domain ('default' by default) config – the Config this domain is part of hosts – list of hosts in this domain And the hosts have:: role – type of this host; should encode the OS and installed packages hostname – fully qualified hostname, usually reachable from other hosts shortname – first component of hostname external_hostname – hostname used to connect to this host ip – IP address domain – the Domain this host is part of transport – allows operations like uploading and downloading files run_command() – runs the given command on the host For each object – Config, Domain, Host – one can provide subclasses to modify the behavior (for example, FreeIPA would add Host methods to run a LDAP query or to install an IPA server). Each object has from_dict and to_dict methods, which can add additional attributes – for example, Config.ntp_server. To use the multihost plugin in tests, create a fixture listing the domains and what number of which host role is needed:: import pytest from pytest_multihost import make_multihost_fixture @pytest.fixture(scope='class') def multihost(request): mh = make_multihost_fixture( request, descriptions=[ { 'type': 'ipa', 'hosts': { 'master': 1, 'replica': 2, }, }, ], ) return mh If not enough hosts are available, all tests that use the fixture are skipped. The object returned from ``make_multihost_fixture`` only has the "config" attribute. Users are expected to add convenience attributes. For example, FreeIPA, which typically uses a single domain with one master, several replicas and some clients, would do:: from pytest_multihost import make_multihost_fixture @pytest.fixture(scope='class') def multihost(request): mh = make_multihost_fixture(request, descriptions=[ { 'type': 'ipa', 'hosts': { 'master': 1, 'replica': 1, 'client': 1, }, }, ], ) # Set convenience attributes mh.domain = mh.config.domains[0] [mh.master] = mh.domain.hosts_by_role('master') mh.replicas = mh.domain.hosts_by_role('replica') mh.clients = mh.domain.hosts_by_role('client') # IPA-specific initialization/teardown of the hosts request.cls().install(mh) request.addfinalizer(lambda: request.cls().uninstall(mh)) # Return the fixture return mh As with any pytest fixture, this can be used by getting it as a function argument. For a simplified example, FreeIPA usage could look something like this:: class TestMultihost(object): def install(self, multihost): multihost.master.run_command(['ipa-server-install']) def uninstall(self, multihost): multihost.master.run_command(['ipa-server-install', '--uninstall']) def test_installed(self, multihost): multihost.master.run_command(['ipa', 'ping']) The description of infrastructure is provided in a JSON or YAML file, which is named on the py.test command line. For example:: ssh_key_filename: ~/.ssh/id_rsa domains: - name: adomain.test type: test-a hosts: - name: master ip: 192.0.2.1 role: master - name: replica1 ip: 192.0.2.2 role: replica - name: replica2 ip: 192.0.2.3 role: replica external_hostname: r2.adomain.test - name: client1 ip: 192.0.2.4 role: client - name: extra ip: 192.0.2.6 role: extrarole - name: bdomain.test type: test-b hosts: - name: master.bdomain.test ip='192.0.2.65 role: master $ py.test --multihost-config=/path/to/configfile.yaml To use YAML files, the PyYAML package is required. Without it only JSON files can be used. Encoding and bytes/text ----------------------- When writing files or issuing commands, bytestrings are passed through unchanged, and text strings (``unicode`` in Python 2) are encoded using a configurable encoding (``utf-8`` by default). When reading files, bytestrings are returned by default, but an encoding can be given to get a test string. For command output, separate ``stdout_bytes`` and ``stdout_text`` attributes are provided. The latter uses a configurable encoding (``utf-8`` by default). Contributing ------------ The project is happy to accept patches! Please file any patches as Pull Requests on the project's `Pagure repo`_. Any development discussion should be in Pagure Pull Requests and Issues. Developer links --------------- * Bug tracker: https://pagure.io/python-pytest-multihost/issues * Code browser: https://pagure.io/python-pytest-multihost/tree/master * git clone https://pagure.io/python-pytest-multihost.git * Unstable packages for Fedora: https://copr.fedoraproject.org/coprs/pviktori/pytest-plugins/ To release, update version in setup.py, add a Git tag like "v0.3", and run `make tarball`. Running `make upload` will put the tarball to Fedora Hosted and PyPI, and a SRPM on Fedorapeople, if you have the rights. Running `make release` will upload and fire a COPR build. .. _Pagure repo: https://pagure.io/python-pytest-multihost Platform: UNKNOWN Classifier: Development Status :: 4 - Beta Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+) Classifier: Operating System :: POSIX Classifier: Framework :: Pytest Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2.6 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Topic :: Software Development :: Quality Assurance pytest-multihost-3.4/setup.cfg0000664004305200430520000000012213643072414017422 0ustar spoorespoore00000000000000[wheel] universal = 1 [egg_info] tag_build = tag_date = 0 tag_svn_revision = 0 pytest-multihost-3.4/pytest_multihost/0000775004305200430520000000000013643072414021246 5ustar spoorespoore00000000000000pytest-multihost-3.4/pytest_multihost/util.py0000664004305200430520000000137713641370452022606 0ustar spoorespoore00000000000000# # Copyright (C) 2013 Red Hat # Copyright (C) 2014 pytest-multihost contributors # See COPYING for license # import tempfile import shutil def check_config_dict_empty(dct, name): """Ensure that no keys are left in a configuration dict""" if dct: raise ValueError('Extra keys in confuguration for %s: %s' % (name, ', '.join(dct))) def shell_quote(bytestring): """Quotes a bytestring for the Bash shell""" return b"'" + bytestring.replace(b"'", b"'\\''") + b"'" class TempDir(object): """Handle for a temporary directory that's deleted on garbage collection""" def __init__(self): self.path = tempfile.mkdtemp(prefix='multihost_tests.') def __del__(self): shutil.rmtree(self.path) pytest-multihost-3.4/pytest_multihost/host.py0000664004305200430520000002431113641370452022577 0ustar spoorespoore00000000000000# # Copyright (C) 2013 Red Hat # Copyright (C) 2014 pytest-multihost contributors # See COPYING for license # """Host class for integration testing""" import os import socket import subprocess from pytest_multihost import transport from pytest_multihost.util import check_config_dict_empty, shell_quote try: basestring except NameError: basestring = str class BaseHost(object): """Representation of a remote host See README for an overview of the core classes. """ transport_class = transport.SSHTransport command_prelude = b'' def __init__(self, domain, hostname, role, ip=None, external_hostname=None, username=None, password=None, test_dir=None, host_type=None): self.host_type = host_type self.domain = domain self.role = str(role) if username is None: self.ssh_username = self.config.ssh_username else: self.ssh_username = username if password is None: self.ssh_key_filename = self.config.ssh_key_filename self.ssh_password = self.config.ssh_password else: self.ssh_key_filename = None self.ssh_password = password if test_dir is None: self.test_dir = domain.config.test_dir else: self.test_dir = test_dir shortname, dot, ext_domain = hostname.partition('.') self.shortname = shortname self.hostname = (hostname[:-1] if hostname.endswith('.') else shortname + '.' + self.domain.name) self.external_hostname = str(external_hostname or hostname) self.netbios = self.domain.name.split('.')[0].upper() self.logger_name = '%s.%s.%s' % ( self.__module__, type(self).__name__, shortname) self.log = self.config.get_logger(self.logger_name) if ip: self.ip = str(ip) else: if self.config.ipv6: # $(dig +short $M $rrtype|tail -1) dig = subprocess.Popen( ['dig', '+short', self.external_hostname, 'AAAA']) stdout, stderr = dig.communicate() self.ip = stdout.splitlines()[-1].strip() else: try: self.ip = socket.gethostbyname(self.external_hostname) except socket.gaierror: self.ip = None if not self.ip: raise RuntimeError('Could not determine IP address of %s' % self.external_hostname) self.host_key = None self.ssh_port = 22 self.env_sh_path = os.path.join(self.test_dir, 'env.sh') self.log_collectors = [] def __str__(self): template = ('<{s.__class__.__name__} {s.hostname} ({s.role})>') return template.format(s=self) def __repr__(self): template = ('<{s.__module__}.{s.__class__.__name__} ' '{s.hostname} ({s.role})>') return template.format(s=self) def add_log_collector(self, collector): """Register a log collector for this host""" self.log_collectors.append(collector) def remove_log_collector(self, collector): """Unregister a log collector""" self.log_collectors.remove(collector) @classmethod def from_dict(cls, dct, domain): """Load this Host from a dict""" if isinstance(dct, basestring): dct = {'name': dct} try: role = dct.pop('role').lower() except KeyError: role = domain.static_roles[0] hostname = dct.pop('name') if '.' not in hostname: hostname = '.'.join((hostname, domain.name)) ip = dct.pop('ip', None) external_hostname = dct.pop('external_hostname', None) username = dct.pop('username', None) password = dct.pop('password', None) host_type = dct.pop('host_type', 'default') check_config_dict_empty(dct, 'host %s' % hostname) return cls(domain, hostname, role, ip=ip, external_hostname=external_hostname, username=username, password=password, host_type=host_type) def to_dict(self): """Export info about this Host to a dict""" result = { 'name': str(self.hostname), 'ip': self.ip, 'role': self.role, 'external_hostname': self.external_hostname, } if self.host_type != 'default': result['host_type'] = self.host_type return result @property def config(self): """The Config that this Host is a part of""" return self.domain.config @property def transport(self): """Provides means to manipulate files & run processs on the remote host Accessing this property might connect to the remote Host (usually via SSH). """ try: return self._transport except AttributeError: cls = self.transport_class if cls: # transport_class is None in the base class and must be # set in subclasses. # Pylint reports that calling None will fail self._transport = cls(self) # pylint: disable=E1102 else: raise NotImplementedError('transport class not available') return self._transport def reset_connection(self): """Reset the connection The next time a connection is needed, a new Transport object will be made. This new transport will take into account any configuration changes, such as external_hostname, ssh_username, etc., that were made on the Host. """ try: del self._transport except: pass def get_file_contents(self, filename, encoding=None): """Shortcut for transport.get_file_contents""" return self.transport.get_file_contents(filename, encoding=encoding) def put_file_contents(self, filename, contents, encoding='utf-8'): """Shortcut for transport.put_file_contents""" self.transport.put_file_contents(filename, contents, encoding=encoding) def collect_log(self, filename): """Call all registered log collectors on the given filename""" for collector in self.log_collectors: collector(self, filename) def run_command(self, argv, set_env=True, stdin_text=None, log_stdout=True, raiseonerr=True, cwd=None, bg=False, encoding='utf-8'): """Run the given command on this host Returns a Command instance. The command will have already run in the shell when this method returns, so its stdout_text, stderr_text, and returncode attributes will be available. :param argv: Command to run, as either a Popen-style list, or a string containing a shell script :param set_env: If true, env.sh exporting configuration variables will be sourced before running the command. :param stdin_text: If given, will be written to the command's stdin :param log_stdout: If false, standard output will not be logged (but will still be available as cmd.stdout_text) :param raiseonerr: If true, an exception will be raised if the command does not exit with return code 0 :param cwd: The working directory for the command :param bg: If True, runs command in background. In this case, either the result should be used in a ``with`` statement, or ``wait()`` should be called explicitly when the command is finished. :param encoding: Encoding for the resulting Command instance's ``stdout_text`` and ``stderr_text``, and for ``stdin_text``, ``argv``, etc. if they are not bytestrings already. """ def encode(string): if not isinstance(string, bytes): return string.encode(encoding) else: return string command = self.transport.start_shell(argv, log_stdout=log_stdout, encoding=encoding) # Set working directory if cwd is None: cwd = self.test_dir command.stdin.write(b'cd %s\n' % shell_quote(encode(cwd))) # Set the environment if set_env: quoted = shell_quote(encode(self.env_sh_path)) command.stdin.write(b'. %s\n' % quoted) if self.command_prelude: command.stdin.write(encode(self.command_prelude)) if stdin_text: command.stdin.write(b"echo -en ") command.stdin.write(_echo_quote(encode(stdin_text))) command.stdin.write(b" | ") if isinstance(argv, basestring): # Run a shell command given as a string command.stdin.write(b'(') command.stdin.write(encode(argv)) command.stdin.write(b')') else: # Run a command given as a popen-style list (no shell expansion) for arg in argv: command.stdin.write(shell_quote(encode(arg))) command.stdin.write(b' ') command.stdin.write(b'\nexit\n') command.stdin.flush() command.raiseonerr = raiseonerr if not bg: command.wait() return command def _echo_quote(bytestring): """Encode a bytestring for use with bash & "echo -en" """ bytestring = bytestring.replace(b"\\", br"\\") bytestring = bytestring.replace(b"\0", br"\x00") bytestring = bytestring.replace(b"'", br"'\''") return b"'" + bytestring + b"'" class Host(BaseHost): """A Unix host""" command_prelude = b'set -e\n' class WinHost(BaseHost): """ Representation of a remote Windows host. """ def __init__(self, domain, hostname, role, **kwargs): # Set test_dir to the Windows directory, if not given explicitly kwargs.setdefault('test_dir', domain.config.windows_test_dir) super(WinHost, self).__init__(domain, hostname, role, **kwargs) pytest-multihost-3.4/pytest_multihost/__init__.py0000664004305200430520000000021413641370452023355 0ustar spoorespoore00000000000000# # Copyright (C) 2014 pytest-multihost contributors. See COPYING for license # from pytest_multihost.plugin import make_multihost_fixture pytest-multihost-3.4/pytest_multihost/config.py0000664004305200430520000002240013641370452023064 0ustar spoorespoore00000000000000# # Copyright (C) 2013 Red Hat # Copyright (C) 2014 pytest-multihost contributors # See COPYING for license # """Utilities for configuration of multi-master tests""" import collections import logging from pytest_multihost.util import check_config_dict_empty class FilterError(ValueError): """Raised when domains description could not be satisfied""" init_args = [ 'test_dir', 'ssh_key_filename', 'ssh_password', 'ssh_username', 'domains', 'ipv6', ] class Config(object): """Container for global configuration and a list of Domains See README for an overview of the core classes. """ extra_init_args = () def __init__(self, **kwargs): self.log = self.get_logger('%s.%s' % (__name__, type(self).__name__)) admin_password = kwargs.get('admin_password') or 'Secret123' # This unfortunately duplicates information in _setting_infos, # but is left here for the sake of static analysis. self.test_dir = kwargs.get('test_dir', '/root/multihost_tests') self.ssh_key_filename = kwargs.get('ssh_key_filename') self.ssh_password = kwargs.get('ssh_password') self.ssh_username = kwargs.get('ssh_username', 'root') self.ipv6 = bool(kwargs.get('ipv6', False)) self.windows_test_dir = kwargs.get('windows_test_dir', '/home/Administrator') if not self.ssh_password and not self.ssh_key_filename: self.ssh_key_filename = '~/.ssh/id_rsa' self.domains = [] domain_class = self.get_domain_class() for domain_dict in kwargs.pop('domains'): self.domains.append(domain_class.from_dict(dict(domain_dict), self)) def get_domain_class(self): return Domain def get_logger(self, name): """Get a logger of the given name Override in subclasses to use a custom logging system """ return logging.getLogger(name) @classmethod def from_dict(cls, dct): """Load a Config object from a dict The dict is usually loaded from an user-supplied YAML or JSON file. In the base implementation, the dict is just passed to the constructor. If more arguments are needed, include them in the class' extra_init_args set. """ # Backwards compatibility with FreeIPA's root-only logins if 'root_ssh_key_filename' in dct: dct['ssh_key_filename'] = dct.pop('root_ssh_key_filename') if 'root_password' in dct: dct['ssh_password'] = dct.pop('root_password') if 'windows_test_dir' in dct: dct['windows_test_dir'] = dct.pop('windows_test_dir') all_init_args = set(init_args) | set(cls.extra_init_args) extra_args = set(dct) - all_init_args if extra_args: ValueError('Extra keys in confuguration for config: %s' % ', '.join(extra_args)) self = cls(**dct) return self def to_dict(self, _autosave_names=()): """Save this Config object to a dict compatible with from_dict :param _autosave_names: To be used by subclasses only. Lists names that should be included in the dict. Values are taken from attributes of the same name. Usually this is a subset of the class' extra_init_args """ dct = {'domains': [d.to_dict() for d in self.domains]} autosave = (set(init_args) | set(_autosave_names)) - set(['domains']) for argname in autosave: value = getattr(self, argname) dct[argname] = value return dct def host_by_name(self, name): """Get a host from any domain by name If multiple hosts have the same name, return the first one. Raise LookupError if no host is found. See Domain.host_by_name for details on matching. """ for domain in self.domains: try: return domain.host_by_name(name) except LookupError: pass raise LookupError(name) def filter(self, descriptions): """Destructively filters hosts and orders domains to fit description :param descriptions: List of dicts such as: [ { 'type': 'ipa', 'hosts': { 'master': 1, 'replica': 2, }, }, ] i.e. the "type" is a type of domain, and "hosts" a dict mapping host roles to the number of hosts of this role that are required. """ unique_domain_types = set(d.get('type', 'default') for d in descriptions) if len(descriptions) != len(unique_domain_types): # TODO: The greedy algorithm used to match domains may not yield # the correct result if there are several domains of the same type. raise ValueError('Duplicate domain type not supported') new_domains = [] for i, description in enumerate(descriptions): for domain in list(self.domains): if domain.fits(description): domain.filter(description['hosts']) new_domains.append(domain) self.domains.remove(domain) break else: raise FilterError( 'Domain %s not configured: %s' % (i, description)) self.domains = new_domains class Domain(object): """Configuration for a domain See README for an overview of the core classes. """ def __init__(self, config, name, domain_type): self.log = config.get_logger('%s.%s' % (__name__, type(self).__name__)) self.type = str(domain_type) self.config = config self.name = str(name) self.hosts = [] def get_host_class(self, host_dict): host_type = host_dict.get('host_type', 'default') return self.host_classes[host_type] @property def host_classes(self): from pytest_multihost.host import Host, WinHost return { 'default': Host, 'windows': WinHost, } @property def roles(self): """All the roles of the hosts in this domain""" return sorted(set(host.role for host in self.hosts)) @property def static_roles(self): """Roles typical for this domain type To be overridden in subclasses """ return ('master', ) @property def extra_roles(self): """Roles of this Domain's hosts that aren't included in static_roles """ return [role for role in self.roles if role not in self.static_roles] @classmethod def from_dict(cls, dct, config): """Load this Domain from a dict """ domain_type = dct.pop('type', 'default') domain_name = dct.pop('name') self = cls(config, domain_name, domain_type) for host_dict in dct.pop('hosts'): host_class = self.get_host_class(host_dict) host = host_class.from_dict(dict(host_dict), self) self.hosts.append(host) check_config_dict_empty(dct, 'domain %s' % domain_name) return self def to_dict(self): """Export this Domain from a dict """ return { 'type': self.type, 'name': self.name, 'hosts': [h.to_dict() for h in self.hosts], } def host_by_role(self, role): """Return the first host of the given role""" hosts = self.hosts_by_role(role) if hosts: return hosts[0] else: raise LookupError(role) def hosts_by_role(self, role): """Return all hosts of the given role""" return [h for h in self.hosts if h.role == role] def host_by_name(self, name): """Return a host with the given name Checks all of: hostname, external_hostname, shortname. If more hosts match, returns the first one. Raises LookupError if no host is found. """ for host in self.hosts: if name in (host.hostname, host.external_hostname, host.shortname): return host raise LookupError(name) def fits(self, description): """Return True if the this fits the description See Domain.filter for discussion of the description. """ if self.type != description.get('type', 'default'): return False for role, number in description['hosts'].items(): if len(self.hosts_by_role(role)) < number: return False return True def filter(self, host_counts): """Destructively filter hosts in this domain :param host_counts: Mapping of host role to number of hosts wanted for that role All extra hosts are removed from this Domain. """ new_hosts = [] for host in list(self.hosts): if host_counts.get(host.role, 0) > 0: new_hosts.append(host) host_counts[host.role] -= 1 if any(h > 0 for h in host_counts.values()): raise ValueError( 'Domain does not fit host counts, extra hosts needed: %s' % host_counts) self.hosts = new_hosts pytest-multihost-3.4/pytest_multihost/transport.py0000664004305200430520000005021313643067070023657 0ustar spoorespoore00000000000000# # Copyright (C) 2015 Red Hat # Copyright (C) 2014 pytest-multihost contributors # See COPYING for license # """Objects for communicating with remote hosts This class defines "SSHTransport" as ParamikoTransport (by default), or as OpenSSHTransport (if Paramiko is not importable, or the PYTESTMULTIHOST_SSH_TRANSPORT environment variable is set to "openssh"). """ import os import socket import threading import subprocess from contextlib import contextmanager import errno import logging import io import sys from pytest_multihost import util try: import paramiko have_paramiko = True except ImportError: have_paramiko = False DEFAULT = object() class Transport(object): """Mechanism for communicating with remote hosts The Transport can manipulate files on a remote host, and open a Command. The base class defines an interface that specific subclasses implement. """ def __init__(self, host): self.host = host self.logger_name = '%s.%s' % (host.logger_name, type(self).__name__) self.log = host.config.get_logger(self.logger_name) self._command_index = 0 def get_file_contents(self, filename, encoding=None): """Read the named remote file and return the contents The string will be decoded using the given encoding; if encoding is None (default), it will be returned as a bytestring. """ raise NotImplementedError('Transport.get_file_contents') def put_file_contents(self, filename, contents, encoding='utf-8'): """Write the given string (or bytestring) to the named remote file The contents string will be encoded using the given encoding (default: ``'utf-8'``), unless aleady a bytestring. """ raise NotImplementedError('Transport.put_file_contents') def file_exists(self, filename): """Return true if the named remote file exists""" raise NotImplementedError('Transport.file_exists') def mkdir(self, path): """Make the named directory""" raise NotImplementedError('Transport.mkdir') def start_shell(self, argv, log_stdout=True, encoding=None): """Start a Shell :param argv: The command this shell is intended to run (used for logging only) :param log_stdout: If false, the stdout will not be logged (useful when binary output is expected) :param encoding: Encoding for the resulting Command's ``stdout_text`` and ``stderr_text``. Given a `shell` from this method, the caller can then use ``shell.stdin.write()`` to input any command(s), call ``shell.wait()`` to let the command run, and then inspect ``returncode``, ``stdout_text`` or ``stderr_text``. Note that ``shell.stdin`` uses bytes I/O. """ raise NotImplementedError('Transport.start_shell') def mkdir_recursive(self, path): """`mkdir -p` on the remote host""" if not self.file_exists(path): parent_path = os.path.dirname(path) if path != parent_path: self.mkdir_recursive(parent_path) self.mkdir(path) def get_file(self, remotepath, localpath): """Copy a file from the remote host to a local file""" contents = self.get_file_contents(remotepath, encoding=None) with open(localpath, 'wb') as local_file: local_file.write(contents) def put_file(self, localpath, remotepath): """Copy a local file to the remote host""" with open(localpath, 'rb') as local_file: contents = local_file.read() self.put_file_contents(remotepath, contents, encoding=None) def get_next_command_logger_name(self): self._command_index += 1 return '%s.cmd%s' % (self.host.logger_name, self._command_index) def rmdir(self, path): """Remove directory""" raise NotImplementedError('Transport.rmdir') def rename_file(self, oldpath, newpath): """Rename file""" raise NotImplementedError('Transport.rename_file') def remove_file(self, filepath): """Removes files""" raise NotImplementedError('Transport.remove_file') class _decoded_output_property(object): """Descriptor for on-demand decoding of a Command's output stream """ def __init__(self, name): self.name = name def __set_name__(self, cls, name): # Sanity check (called only on Python 3.6+). # This property expects to handle attributes named '_text'. assert name == self.name + '_text' def __get__(self, instance, cls=None): if instance is None: return self else: bytestring = getattr(instance, self.name + '_bytes') decoded = bytestring.decode(instance.encoding) setattr(instance, self.name + '_text', decoded) return decoded class Command(object): """A Popen-style object representing a remote command Instances of this class should only be created via method of a concrete Transport, such as start_shell. The standard error and output are handled by this class. They're not available for file-like reading, and are logged by default. To make sure reading doesn't stall after one buffer fills up, they are read in parallel using threads. After calling wait(), ``stdout_bytes`` and ``stderr_bytes`` attributes will be bytestrings containing the output, and ``returncode`` will contain the exit code. The ``stdout_text`` and ``stdout_text`` will be the corresponding output decoded using the given ``encoding`` (default: ``'utf-8'``). These are decoded on-demand; do not access them if a command produces binary output. A Command may be used as a context manager (in the ``with`` statement). Exiting the context will automatically call ``wait()``. This raises an exception if the exit code is not 0, unless the ``raiseonerr`` attribute is set to false before exiting the context. """ def __init__(self, argv, logger_name=None, log_stdout=True, get_logger=None, encoding='utf-8'): self.returncode = None self.argv = argv self._done = False if logger_name: self.logger_name = logger_name else: self.logger_name = '%s.%s' % (self.__module__, type(self).__name__) if get_logger is None: get_logger = logging.getLogger self.get_logger = get_logger self.log = get_logger(self.logger_name) self.encoding = encoding self.raiseonerr = True stdout_text = _decoded_output_property('stdout') stderr_text = _decoded_output_property('stderr') def wait(self, raiseonerr=DEFAULT): """Wait for the remote process to exit Raises an exception if the exit code is not 0, unless ``raiseonerr`` is true. When ``raiseonerr`` is not specified as argument, the ``raiseonerr`` attribute is used. """ if raiseonerr is DEFAULT: raiseonerr = self.raiseonerr if self._done: return self.returncode self._end_process() self._done = True if raiseonerr and self.returncode: self.log.error('Exit code: %s', self.returncode) raise subprocess.CalledProcessError(self.returncode, self.argv) else: self.log.debug('Exit code: %s', self.returncode) return self.returncode def _end_process(self): """Wait until the process exits and output is received, close channel Called from wait() """ raise NotImplementedError() def __enter__(self): return self def __exit__(self, *exc_info): self.wait(raiseonerr=self.raiseonerr) class ParamikoTransport(Transport): """Transport that uses the Paramiko SSH2 library""" def __init__(self, host): super(ParamikoTransport, self).__init__(host) sock = socket.create_connection((host.external_hostname, host.ssh_port)) self._transport = transport = paramiko.Transport(sock) transport.connect(hostkey=host.host_key) if host.ssh_key_filename: filename = os.path.expanduser(host.ssh_key_filename) key = paramiko.RSAKey.from_private_key_file(filename) self.log.debug( 'Authenticating with private RSA key using user %s' % host.ssh_username) transport.auth_publickey(username=host.ssh_username, key=key) elif host.ssh_password: self.log.debug('Authenticating with password using user %s' % host.ssh_username) transport.auth_password(username=host.ssh_username, password=host.ssh_password) else: self.log.critical('No SSH credentials configured') raise RuntimeError('No SSH credentials configured') @contextmanager def sftp_open(self, filename, mode='r'): """Context manager that provides a file-like object over a SFTP channel This provides compatibility with older Paramiko versions. (In Paramiko 1.10+, file objects from `sftp.open` are directly usable as context managers). """ file = self.sftp.open(filename, mode) try: yield file finally: file.close() @property def sftp(self): """Paramiko SFTPClient connected to this host""" try: return self._sftp except AttributeError: transport = self._transport self._sftp = paramiko.SFTPClient.from_transport(transport) return self._sftp def get_file_contents(self, filename, encoding=None): """Read the named remote file and return the contents as a string""" self.log.debug('READ %s', filename) with self.sftp_open(filename, 'rb') as f: result = f.read() if encoding: result = result.decode(encoding) return result def put_file_contents(self, filename, contents, encoding=None): """Write the given string to the named remote file""" self.log.info('WRITE %s', filename) if encoding and not isinstance(contents, bytes): contents = contents.encode(encoding) with self.sftp_open(filename, 'wb') as f: f.write(contents) def file_exists(self, filename): """Return true if the named remote file exists""" self.log.debug('STAT %s', filename) try: self.sftp.stat(filename) except IOError as e: if e.errno == errno.ENOENT: return False else: raise return True def mkdir(self, path): self.log.info('MKDIR %s', path) self.sftp.mkdir(path) def start_shell(self, argv, log_stdout=True, encoding='utf-8'): logger_name = self.get_next_command_logger_name() ssh = self._transport.open_channel('session') self.log.info('RUN %s', argv) return SSHCommand(ssh, argv, logger_name=logger_name, log_stdout=log_stdout, get_logger=self.host.config.get_logger, encoding=encoding) def get_file(self, remotepath, localpath): self.log.debug('GET %s', remotepath) self.sftp.get(remotepath, localpath) def put_file(self, localpath, remotepath): self.log.info('PUT %s', remotepath) self.sftp.put(localpath, remotepath) def rmdir(self, path): self.log.info('RMDIR %s', path) self.sftp.rmdir(path) def remove_file(self, filepath): self.log.info('REMOVE FILE %s', filepath) self.sftp.remove(filepath) def rename_file(self, oldpath, newpath): self.log.info('RENAME %s to %s', oldpath, newpath) self.sftp.rename(oldpath, newpath) class OpenSSHTransport(Transport): """Transport that uses the `ssh` binary""" def __init__(self, host): super(OpenSSHTransport, self).__init__(host) self.control_dir = util.TempDir() self.ssh_argv = self._get_ssh_argv() # Run a "control master" process. This serves two purposes: # - Establishes a control socket; other SSHs will connect to it # and reuse the same connection. This way the slow handshake # only needs to be done once # - Writes the host to known_hosts so stderr of "real" connections # doesn't contain the "unknown host" warning # Popen closes the stdin pipe when it's garbage-collected, so # this process will exit when it's no longer needed command = ['-o', 'ControlMaster=yes', '/usr/bin/cat'] self.control_master = self._run(command, collect_output=False) def _get_ssh_argv(self): """Return the path to SSH and options needed for every call""" control_file = os.path.join(self.control_dir.path, 'control') known_hosts_file = os.path.join(self.control_dir.path, 'known_hosts') argv = ['ssh', '-l', self.host.ssh_username, '-o', 'ControlPath=%s' % control_file, '-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHostsFile=%s' % known_hosts_file] if self.host.ssh_key_filename: key_filename = os.path.expanduser(self.host.ssh_key_filename) argv.extend(['-i', key_filename]) # disable password prompt argv.extend(['-o', 'BatchMode=yes']) elif self.host.ssh_password: password_file = os.path.join(self.control_dir.path, 'password') with open(password_file, 'w') as f: os.fchmod(f.fileno(), 0o600) f.write(self.host.ssh_password) f.write('\n') argv = ['sshpass', '-f', password_file] + argv else: self.log.critical('No SSH credentials configured') raise RuntimeError('No SSH credentials configured') argv.append(self.host.external_hostname) self.log.debug('SSH invocation: %s', argv) return argv def start_shell(self, argv, log_stdout=True, encoding='utf-8'): self.log.info('RUN %s', argv) command = self._run(['bash'], argv=argv, log_stdout=log_stdout, encoding=encoding) return command def _run(self, command, log_stdout=True, argv=None, collect_output=True, encoding='utf-8'): """Run the given command on the remote host :param command: Command to run (appended to the common SSH invocation) :param log_stdout: If false, stdout will not be logged :param argv: Command to log (if different from ``command`` :param collect_output: If false, no output will be collected """ if argv is None: argv = command logger_name = self.get_next_command_logger_name() ssh = SSHCallWrapper(self.ssh_argv + list(command)) return SSHCommand(ssh, argv, logger_name, log_stdout=log_stdout, collect_output=collect_output, get_logger=self.host.config.get_logger, encoding=encoding) def file_exists(self, path): self.log.info('STAT %s', path) cmd = self._run(['ls', path], log_stdout=False) cmd.wait(raiseonerr=False) return cmd.returncode == 0 def mkdir(self, path): self.log.info('MKDIR %s', path) cmd = self._run(['mkdir', path]) cmd.wait() def put_file_contents(self, filename, contents, encoding='utf-8'): self.log.info('PUT %s', filename) if encoding and not isinstance(contents, bytes): contents = contents.encode(encoding) cmd = self._run(['tee', filename], log_stdout=False) cmd.stdin.write(contents) cmd.wait() assert cmd.stdout_bytes == contents def get_file_contents(self, filename, encoding=None): self.log.info('GET %s', filename) cmd = self._run(['cat', filename], log_stdout=False) cmd.wait(raiseonerr=False) if cmd.returncode == 0: result = cmd.stdout_bytes if encoding: result = result.decode(encoding) return result else: raise IOError('File %r could not be read' % filename) def rmdir(self, path): self.log.info('RMDIR %s', path) cmd = self._run(['rmdir', path]) cmd.wait() def remove_file(self, filepath): self.log.info('REMOVE FILE %s', filepath) cmd = self._run(['rm', filepath]) cmd.wait() if cmd.returncode != 0: raise IOError('File %r could not be deleted' % filepath) def rename_file(self, oldpath, newpath): self.log.info('RENAME %s TO %s', oldpath, newpath) cmd = self._run(['mv', oldpath, newpath]) cmd.wait() if cmd.returncode != 0: raise IOError('File %r could not be renamed to %r ' % (oldpath, newpath)) class SSHCallWrapper(object): """Adapts a /usr/bin/ssh call to the paramiko.Channel interface This only wraps what SSHCommand needs. """ def __init__(self, command): self.command = command def invoke_shell(self): self.command = subprocess.Popen( self.command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) def makefile(self, mode): return { 'wb': self.command.stdin, 'rb': self.command.stdout, }[mode] def makefile_stderr(self, mode): assert mode == 'rb' return self.command.stderr def recv_exit_status(self): return self.command.wait() def close(self): return self.command.wait() class SSHCommand(Command): """Command implementation for ParamikoTransport and OpenSSHTranspport""" def __init__(self, ssh, argv, logger_name, log_stdout=True, collect_output=True, encoding='utf-8', get_logger=None): super(SSHCommand, self).__init__(argv, logger_name, log_stdout=log_stdout, get_logger=get_logger, encoding=encoding) self._stdout_lines = [] self._stderr_lines = [] self.running_threads = set() self._ssh = ssh self.log.debug('RUN %s', argv) self._ssh.invoke_shell() self._use_bytes = (encoding is None) def wrap_file(file, encoding): if self._use_bytes: return file else: return io.TextIOWrapper(file, encoding=encoding) self.stdin = self._ssh.makefile('wb') stdout = self._ssh.makefile('rb') stderr = self._ssh.makefile_stderr('rb') if collect_output: self._start_pipe_thread(self._stdout_lines, stdout, 'out', log_stdout) self._start_pipe_thread(self._stderr_lines, stderr, 'err', True) def _end_process(self): self.stdin.close() while self.running_threads: self.running_threads.pop().join() self.stdout_bytes = b''.join(self._stdout_lines) self.stderr_bytes = b''.join(self._stderr_lines) self.returncode = self._ssh.recv_exit_status() self._ssh.close() def _start_pipe_thread(self, result_list, stream, name, do_log=True): """Start a thread that copies lines from ``stream`` to ``result_list`` If do_log is true, also logs the lines under ``name`` The thread is added to ``self.running_threads``. """ log = self.get_logger(self.logger_name) def read_stream(): for line in stream: if do_log: log.debug(line.rstrip(b'\n').decode('utf-8', errors='replace')) result_list.append(line) thread = threading.Thread(target=read_stream) self.running_threads.add(thread) thread.start() return thread if ( not have_paramiko or os.environ.get('PYTESTMULTIHOST_SSH_TRANSPORT') == 'openssh' ): SSHTransport = OpenSSHTransport else: SSHTransport = ParamikoTransport pytest-multihost-3.4/pytest_multihost/plugin.py0000664004305200430520000000655513641370452023132 0ustar spoorespoore00000000000000# # Copyright (C) 2014 pytest-multihost contributors. See COPYING for license # import json import os import traceback import pytest from pytest_multihost.config import Config, FilterError try: import yaml except ImportError: yaml = None def pytest_addoption(parser): parser.addoption( '--multihost-config', dest="multihost_config", help="Site configuration for multihost tests") @pytest.mark.tryfirst def pytest_load_initial_conftests(args, early_config, parser): ns = early_config.known_args_namespace if ns.multihost_config: try: conffile = open(ns.multihost_config) except IOError as e: raise exit('Unable to open multihost configuration file %s: %s\n' 'Please check path of configuration file and retry.' % (ns.multihost_config, e.args[1])) with conffile: if yaml: confdict = yaml.safe_load(conffile) else: try: confdict = json.load(conffile) except Exception: traceback.print_exc() raise exit( 'Could not load %s. If it is a YAML file, you need ' 'PyYAML installed.' % ns.multihost_config) plugin = MultihostPlugin(confdict) pluginmanager = early_config.pluginmanager.register( plugin, 'MultihostPlugin') class MultihostPlugin(object): """The Multihost plugin The plugin is available as pluginmanager.getplugin('MultihostPlugin'), and its presence indicates that multihost testing has been configured. """ def __init__(self, confdict): self.confdict = confdict class MultihostFixture(object): """A fixture containing the multihost testing configuration Contains the `config`; other attributes may be added to it for convenience. """ def __init__(self, config, request): self.config = config self._pytestmh_request = request def install(self): """Call install()/uninstall() for the class this fixture is used on This function is DEPRECATED. """ request = self._pytestmh_request cls = request.cls install = getattr(cls, 'install', None) if install: request.addfinalizer(lambda: cls().uninstall(self)) cls().install(self) return self def make_multihost_fixture(request, descriptions, config_class=Config, _config=None): """Create a MultihostFixture, or skip the test :param request: The Pytest request object :param descriptions: Descriptions of wanted domains (see README or Domain.filter) :param config_class: Custom Config class to use :param _config: Config to be used directly. Intended mostly for testing the plugin itself. Skips the test if there are not enough resources configured. """ if _config is None: plugin = request.config.pluginmanager.getplugin('MultihostPlugin') if not plugin: pytest.skip('Multihost tests not configured') confdict = plugin.confdict _config = config_class.from_dict(confdict) try: _config.filter(descriptions) except FilterError as e: pytest.skip('Not enough resources configured: %s' % e) return MultihostFixture(_config, request) pytest-multihost-3.4/pytest_multihost.egg-info/0000775004305200430520000000000013643072414022740 5ustar spoorespoore00000000000000pytest-multihost-3.4/pytest_multihost.egg-info/entry_points.txt0000664004305200430520000000006013643072414026232 0ustar spoorespoore00000000000000[pytest11] multihost = pytest_multihost.plugin pytest-multihost-3.4/pytest_multihost.egg-info/top_level.txt0000664004305200430520000000002113643072414025463 0ustar spoorespoore00000000000000pytest_multihost pytest-multihost-3.4/pytest_multihost.egg-info/SOURCES.txt0000664004305200430520000000066213643072414024630 0ustar spoorespoore00000000000000README.rst setup.cfg setup.py pytest_multihost/__init__.py pytest_multihost/config.py pytest_multihost/host.py pytest_multihost/plugin.py pytest_multihost/transport.py pytest_multihost/util.py pytest_multihost.egg-info/PKG-INFO pytest_multihost.egg-info/SOURCES.txt pytest_multihost.egg-info/dependency_links.txt pytest_multihost.egg-info/entry_points.txt pytest_multihost.egg-info/requires.txt pytest_multihost.egg-info/top_level.txtpytest-multihost-3.4/pytest_multihost.egg-info/PKG-INFO0000664004305200430520000002173313643072414024043 0ustar spoorespoore00000000000000Metadata-Version: 1.1 Name: pytest-multihost Version: 3.4 Summary: Utility for writing multi-host tests for pytest Home-page: https://pagure.io/python-pytest-multihost Author: Petr Viktorin Author-email: pviktori@redhat.com License: GPL Description: A pytest plugin for multi-host testing. Downloading ----------- Release tarballs will be made available for download from Pagure Releases: https://pagure.io/releases/python-pytest-multihost/ The goal is to include this project in Fedora repositories. Until that happens, you can use testing builds from COPR – see "Developer links" below. You can also install using pip: https://pypi.python.org/pypi/pytest-multihost Usage ----- This plugin takes a description of your infrastructure, and provides, via a fixture, Host objects that commands can be called on. It is intended as a general base for a framework; any project using it will need to extend it for its own needs. The object provided to tests is a Config object, which has (among others) these attributes:: test_dir – directory to store test-specific data in, defaults to /root/multihost_tests ipv6 – true if connecting via IPv6 domains – the list of domains Hosts to run on are arranged in domains, which have:: name – the DNS name of the domain type – a string specifying the type of the domain ('default' by default) config – the Config this domain is part of hosts – list of hosts in this domain And the hosts have:: role – type of this host; should encode the OS and installed packages hostname – fully qualified hostname, usually reachable from other hosts shortname – first component of hostname external_hostname – hostname used to connect to this host ip – IP address domain – the Domain this host is part of transport – allows operations like uploading and downloading files run_command() – runs the given command on the host For each object – Config, Domain, Host – one can provide subclasses to modify the behavior (for example, FreeIPA would add Host methods to run a LDAP query or to install an IPA server). Each object has from_dict and to_dict methods, which can add additional attributes – for example, Config.ntp_server. To use the multihost plugin in tests, create a fixture listing the domains and what number of which host role is needed:: import pytest from pytest_multihost import make_multihost_fixture @pytest.fixture(scope='class') def multihost(request): mh = make_multihost_fixture( request, descriptions=[ { 'type': 'ipa', 'hosts': { 'master': 1, 'replica': 2, }, }, ], ) return mh If not enough hosts are available, all tests that use the fixture are skipped. The object returned from ``make_multihost_fixture`` only has the "config" attribute. Users are expected to add convenience attributes. For example, FreeIPA, which typically uses a single domain with one master, several replicas and some clients, would do:: from pytest_multihost import make_multihost_fixture @pytest.fixture(scope='class') def multihost(request): mh = make_multihost_fixture(request, descriptions=[ { 'type': 'ipa', 'hosts': { 'master': 1, 'replica': 1, 'client': 1, }, }, ], ) # Set convenience attributes mh.domain = mh.config.domains[0] [mh.master] = mh.domain.hosts_by_role('master') mh.replicas = mh.domain.hosts_by_role('replica') mh.clients = mh.domain.hosts_by_role('client') # IPA-specific initialization/teardown of the hosts request.cls().install(mh) request.addfinalizer(lambda: request.cls().uninstall(mh)) # Return the fixture return mh As with any pytest fixture, this can be used by getting it as a function argument. For a simplified example, FreeIPA usage could look something like this:: class TestMultihost(object): def install(self, multihost): multihost.master.run_command(['ipa-server-install']) def uninstall(self, multihost): multihost.master.run_command(['ipa-server-install', '--uninstall']) def test_installed(self, multihost): multihost.master.run_command(['ipa', 'ping']) The description of infrastructure is provided in a JSON or YAML file, which is named on the py.test command line. For example:: ssh_key_filename: ~/.ssh/id_rsa domains: - name: adomain.test type: test-a hosts: - name: master ip: 192.0.2.1 role: master - name: replica1 ip: 192.0.2.2 role: replica - name: replica2 ip: 192.0.2.3 role: replica external_hostname: r2.adomain.test - name: client1 ip: 192.0.2.4 role: client - name: extra ip: 192.0.2.6 role: extrarole - name: bdomain.test type: test-b hosts: - name: master.bdomain.test ip='192.0.2.65 role: master $ py.test --multihost-config=/path/to/configfile.yaml To use YAML files, the PyYAML package is required. Without it only JSON files can be used. Encoding and bytes/text ----------------------- When writing files or issuing commands, bytestrings are passed through unchanged, and text strings (``unicode`` in Python 2) are encoded using a configurable encoding (``utf-8`` by default). When reading files, bytestrings are returned by default, but an encoding can be given to get a test string. For command output, separate ``stdout_bytes`` and ``stdout_text`` attributes are provided. The latter uses a configurable encoding (``utf-8`` by default). Contributing ------------ The project is happy to accept patches! Please file any patches as Pull Requests on the project's `Pagure repo`_. Any development discussion should be in Pagure Pull Requests and Issues. Developer links --------------- * Bug tracker: https://pagure.io/python-pytest-multihost/issues * Code browser: https://pagure.io/python-pytest-multihost/tree/master * git clone https://pagure.io/python-pytest-multihost.git * Unstable packages for Fedora: https://copr.fedoraproject.org/coprs/pviktori/pytest-plugins/ To release, update version in setup.py, add a Git tag like "v0.3", and run `make tarball`. Running `make upload` will put the tarball to Fedora Hosted and PyPI, and a SRPM on Fedorapeople, if you have the rights. Running `make release` will upload and fire a COPR build. .. _Pagure repo: https://pagure.io/python-pytest-multihost Platform: UNKNOWN Classifier: Development Status :: 4 - Beta Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+) Classifier: Operating System :: POSIX Classifier: Framework :: Pytest Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2.6 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Topic :: Software Development :: Quality Assurance pytest-multihost-3.4/pytest_multihost.egg-info/requires.txt0000664004305200430520000000001513643072414025334 0ustar spoorespoore00000000000000pytest>=2.4.0pytest-multihost-3.4/pytest_multihost.egg-info/dependency_links.txt0000664004305200430520000000000113643072414027006 0ustar spoorespoore00000000000000 pytest-multihost-3.4/setup.py0000664004305200430520000000243313643072157017326 0ustar spoorespoore00000000000000#!/usr/bin/python2 # # Copyright (C) 2014 pytest-multihost contributors. See COPYING for license # from setuptools import setup import io with io.open('README.rst', 'rt', encoding='utf-8') as f: readme_contents = f.read() setup_args = dict( name = "pytest-multihost", version = "3.4", description = "Utility for writing multi-host tests for pytest", long_description = readme_contents, url = "https://pagure.io/python-pytest-multihost", license = "GPL", author = "Petr Viktorin", author_email = "pviktori@redhat.com", packages = ["pytest_multihost"], classifiers=[ 'Development Status :: 4 - Beta', 'License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)', 'Operating System :: POSIX', 'Framework :: Pytest', 'Programming Language :: Python', 'Programming Language :: Python :: 2.6', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Topic :: Software Development :: Quality Assurance', ], install_requires=['pytest>=2.4.0'], # (paramiko & PyYAML are suggested) entry_points = { 'pytest11': [ 'multihost = pytest_multihost.plugin', ], }, ) if __name__ == '__main__': setup(**setup_args) pytest-multihost-3.4/README.rst0000664004305200430520000001514013642661213017276 0ustar spoorespoore00000000000000A pytest plugin for multi-host testing. Downloading ----------- Release tarballs will be made available for download from Pagure Releases: https://pagure.io/releases/python-pytest-multihost/ The goal is to include this project in Fedora repositories. Until that happens, you can use testing builds from COPR – see "Developer links" below. You can also install using pip: https://pypi.python.org/pypi/pytest-multihost Usage ----- This plugin takes a description of your infrastructure, and provides, via a fixture, Host objects that commands can be called on. It is intended as a general base for a framework; any project using it will need to extend it for its own needs. The object provided to tests is a Config object, which has (among others) these attributes:: test_dir – directory to store test-specific data in, defaults to /root/multihost_tests ipv6 – true if connecting via IPv6 domains – the list of domains Hosts to run on are arranged in domains, which have:: name – the DNS name of the domain type – a string specifying the type of the domain ('default' by default) config – the Config this domain is part of hosts – list of hosts in this domain And the hosts have:: role – type of this host; should encode the OS and installed packages hostname – fully qualified hostname, usually reachable from other hosts shortname – first component of hostname external_hostname – hostname used to connect to this host ip – IP address domain – the Domain this host is part of transport – allows operations like uploading and downloading files run_command() – runs the given command on the host For each object – Config, Domain, Host – one can provide subclasses to modify the behavior (for example, FreeIPA would add Host methods to run a LDAP query or to install an IPA server). Each object has from_dict and to_dict methods, which can add additional attributes – for example, Config.ntp_server. To use the multihost plugin in tests, create a fixture listing the domains and what number of which host role is needed:: import pytest from pytest_multihost import make_multihost_fixture @pytest.fixture(scope='class') def multihost(request): mh = make_multihost_fixture( request, descriptions=[ { 'type': 'ipa', 'hosts': { 'master': 1, 'replica': 2, }, }, ], ) return mh If not enough hosts are available, all tests that use the fixture are skipped. The object returned from ``make_multihost_fixture`` only has the "config" attribute. Users are expected to add convenience attributes. For example, FreeIPA, which typically uses a single domain with one master, several replicas and some clients, would do:: from pytest_multihost import make_multihost_fixture @pytest.fixture(scope='class') def multihost(request): mh = make_multihost_fixture(request, descriptions=[ { 'type': 'ipa', 'hosts': { 'master': 1, 'replica': 1, 'client': 1, }, }, ], ) # Set convenience attributes mh.domain = mh.config.domains[0] [mh.master] = mh.domain.hosts_by_role('master') mh.replicas = mh.domain.hosts_by_role('replica') mh.clients = mh.domain.hosts_by_role('client') # IPA-specific initialization/teardown of the hosts request.cls().install(mh) request.addfinalizer(lambda: request.cls().uninstall(mh)) # Return the fixture return mh As with any pytest fixture, this can be used by getting it as a function argument. For a simplified example, FreeIPA usage could look something like this:: class TestMultihost(object): def install(self, multihost): multihost.master.run_command(['ipa-server-install']) def uninstall(self, multihost): multihost.master.run_command(['ipa-server-install', '--uninstall']) def test_installed(self, multihost): multihost.master.run_command(['ipa', 'ping']) The description of infrastructure is provided in a JSON or YAML file, which is named on the py.test command line. For example:: ssh_key_filename: ~/.ssh/id_rsa domains: - name: adomain.test type: test-a hosts: - name: master ip: 192.0.2.1 role: master - name: replica1 ip: 192.0.2.2 role: replica - name: replica2 ip: 192.0.2.3 role: replica external_hostname: r2.adomain.test - name: client1 ip: 192.0.2.4 role: client - name: extra ip: 192.0.2.6 role: extrarole - name: bdomain.test type: test-b hosts: - name: master.bdomain.test ip='192.0.2.65 role: master $ py.test --multihost-config=/path/to/configfile.yaml To use YAML files, the PyYAML package is required. Without it only JSON files can be used. Encoding and bytes/text ----------------------- When writing files or issuing commands, bytestrings are passed through unchanged, and text strings (``unicode`` in Python 2) are encoded using a configurable encoding (``utf-8`` by default). When reading files, bytestrings are returned by default, but an encoding can be given to get a test string. For command output, separate ``stdout_bytes`` and ``stdout_text`` attributes are provided. The latter uses a configurable encoding (``utf-8`` by default). Contributing ------------ The project is happy to accept patches! Please file any patches as Pull Requests on the project's `Pagure repo`_. Any development discussion should be in Pagure Pull Requests and Issues. Developer links --------------- * Bug tracker: https://pagure.io/python-pytest-multihost/issues * Code browser: https://pagure.io/python-pytest-multihost/tree/master * git clone https://pagure.io/python-pytest-multihost.git * Unstable packages for Fedora: https://copr.fedoraproject.org/coprs/pviktori/pytest-plugins/ To release, update version in setup.py, add a Git tag like "v0.3", and run `make tarball`. Running `make upload` will put the tarball to Fedora Hosted and PyPI, and a SRPM on Fedorapeople, if you have the rights. Running `make release` will upload and fire a COPR build. .. _Pagure repo: https://pagure.io/python-pytest-multihost