vmdb2/0000755000175000017500000000000013577177751011132 5ustar gwolfgwolfvmdb2/NEWS0000644000175000017500000001526613577177751011643 0ustar gwolfgwolfNEWS for vmdb2, the Debian disk image builder ============================================================================= Version 0.13.2+git, not yet released ------------------------------------ * Source code for vmdb2 is now hosted on git.liw.fi, moved there from GitHub. * Fixed smoke test timeout for starting a VM to 5 min. The previous timeout (30 s) was short enough that tests often failed because of it. * The `debootstrap` action now finishes with `apt-get update`. For some reason thing had changed so that this became necessary. * Christian Schlüter added the optional `label` field to the `mkfs` step to set the filesystem label upon creation. * Changed the `grub` step to take an optional `image-dev` field to override the heuristics for finding the device where GRUB is to be installed. This was necessary to allow vmdb2 to install Debian onto real hardware. * The smoke test now tests a UEFI image as well. * The rootfs unpacking action now copies `/etc/resolv.conf` into the chroot, overwriting what the tarball has. This allows vmdb2 to work with a tarball generated in a different network location. * Add plugins for doing full-disk encryption via cryptsetup, and LVM2. * There is now some documentation. * Christian Schlüter added support for f2fs labels in the `mkfs` step. * Step runners may now implement the `run_even_if_skipped` method to have code that is run whether the step is skipped or not. The `debootstrap` step now uses this to run `apt-get update` always. This allows a rootfs tarball that is old to be used, without the Packages files being too old to be usable. * The `cache_rootfs` step now caches all the explicitly mounted filesystems, not just the root filesystem. Version 0.13.2, released 2018-05-06 ------------------------------------ * New build, after CI screwed up. Version 0.13.1, released 2018-04-30 ---------------------------------- * Fix build-dependency on pandoc. Version 0.13.1, released 2018-04-30 ---------------------------------- Version 0.12, released 2018-02-24 ---------------------------------- * Minor Debian packaging fixes. Version 0.11, released 2018-02-10 ---------------------------------- * Build fixes so that .deb doesn't contain Python 2.7 stuff, which isn't wanted or used. Version 0.10, released 2018-02-10 --------------------------------- * Build tweaks to make builds work on Debian unstable. Version 0.9, released 2017-10-11 --------------------------------- * Stuart Prescott added a `variant` parameter to the `vmdebootstrap` plugin. * Stuart and Lars documented some of the dependencies in README. If they're insufficient, report a bug please. * Stuart added a `qemu-debootstrap` plugin to build images for a foreign architecture. * Lars Wirzenius added the function `vmdb.runcm_chroot` for executing programs inside a chroot. It's a short wrapper around `vmdb.runcmd`, but is a little easier to get right than invoking **chroot**(8) via `runcmd`. Michael Stapelberg fixed the function. * Lars Wirzenius changed it so that `.vmdb` files can have structured YAML values and their constituent strings are used as Jinja2 templates. Previously vmdb2 would just crash. This opens a possibility to have things like "list of packages to install" for the `apt` step as a YAML list, instead of having the step runner parse a space delimited list. * Michael Stapelberg added the `fs-type` parameter to the `mkpart` step so that Raspberry Pi images can be created. * Lars Wirzenius changed the `apt` step. Previously the `apt` field values was the name of one package to install. Now the value MUST be `install` and a separate field `packages` is a YAML list of package names to install. These will all be installed at once. * Michael Stapelberg and Lars Wirzenius added support for a `components` list to the `qemu-debootstrap` step, to allow giving a list of components to the debootstrap run by the step. Version 0.8, released 2017-07-29 --------------------------------- * The `mount` plugin can now specify a mount point on an already mounted filesystem, such as /boot inside the root filesystem already mounted. This allows constructing a system with a separate /boot partition. Thanks to Stuart Prescott for reporting the lack of this feature. Version 0.7, released 2017-06-18 --------------------------------- * New plugin `virtuals` provides step `mount-virtual-filesystems` for mounting virtual filesystems such as `/proc` and `/dev` as well. * The `apt` step installs `eatmydata` and runs `apt` under it, to speed up package installs. Version 0.6, released 2017-06-11 --------------------------------- * setup.py now installs the plugins, making the .deb package actually be usable. Version 0.5, released 2017-06-04 --------------------------------- * The "unless:" part of vmdb spec files now actually works. Previously it was entrely unimplmented (there was code to implement the check, but it was never actually called). The apt and debootstrap plugins have been fixed to not do "has rootfs tarball been unpacked" checks themselves. Version 0.4, released 2017-06-03 --------------------------------- * Add a rudimenteary smoke test yarn for vmdb2-built images. * Add a plugin to run Ansible to configure an image at build time. Doing this via Ansible saves me from having to write equivalent functionality directly into vmdb2, which would be quite a lot of work. (If someone wants puppet, a plugin for that is probably easily doable, assuming it can run against a chroot. I don't now, I've never used puppet.) Version 0.3, released 2017-05-21 --------------------------------- * Simplify progress reporting to go to stdout, plus stop logging progress reports ar as errors. * Add a BIOS flavor for installing GRUB. Version 0.2, released 2017-05-14 --------------------------------- * Add plugin to provide steps to cache the rootfs, and to unpack the rootfs from the cache, instead of running debootstrap or installing packages. This speeds up iteration time from about 9 minutes to 40 seconds on my laptop. See small.vmdb for an example. * Add a generic "unless this condition is true" functionality to steps. If a step has a field "unless: foo" it is skipped the variable foo exists and is true. The variables are set by steps, and currently only the rootfs unpacking step sets a variable (the `rootfs_unpacked` variable). This allows debootstrap to be skipped if the rootfs has already been created by unpacking a cached tarball. Version 0.1, released 2017-05-13 ----------------------------------------------------------------------------- This is the first release. It can build a UEFI image for the amd64 architecture. It's not meant to really be useful for other people. vmdb2/README.md0000644000175000017500000001525413577177751012420 0ustar gwolfgwolfREADME for vmdb2 or vmdebootstrap 2nd generation ; -*- mode: markdown;-*- ============================================================================= [vmdb2][] is a program for producing a disk image with Debian installed. [vmdb2]: https://vmdb2.liw.fi/ Introduction ----------------------------------------------------------------------------- [vmdebootstrap][] installs Debian onto a disk image. It is like the [debootstrap][] tool, except the end result is a bootable disk image, not a directory. vmdebootstrap takes care of creating partitions, and filesystems, and allows some more customization than the older vmdebootstrap does. vmdebootstrap is also a messy pile of kludge, and rather inflexible. vmdb2 is a re-implementation from scratch, without a need for backwards compatibility. It aims to provide more flexibility than vmdebootstrap, without becoming anywhere near as complicated. Think of vmdb2 as "vmdebootstrap the second generation". The name has changed to allow the two tools to installable in parallel, which is important for a transition period. The main user-visible difference between vmdebootstrap and vmdb2 is that the older program provides extensibility via a legion of command line options and the newer program by providing a domain specific language to express what kind of Debian system is to be created. (Lars Wirzenius wrote both vmdebootstrap and vmdb2 and is entitled to sneer at his younger self. It's his way of dealing with the mountain of guilt of making something as awful as vmdebootstrap.) [vmdebootstrap]: http://liw.fi/vmdebootstrap/ [debootstrap]: https://packages.debian.org/unstable/debootstrap Getting vmdb2 ----------------------------------------------------------------------------- vmdb2 source code is available via git: * * It used to be on GitHub as well, but was withdrawn from there due to GitHub being a proprietary service. Requirements: The following tools are used by vmdb2 (Debian package names in brackets). * `kpartx` [kpartx, mkpart command] * `parted` [`parted`, mklabel command] * `qemu-img` [`qemu-utils`, mkimg command] * `qemu-user-static` [`qemu-user-static`, qemu-debootstrap command] The following Python modules are used by vmdb2 (Debian package names in brackets). * cliapp [`python3-cliapp`] * jinja2 [`python3-jinja2`] * yaml [`python3-yaml`] Dependencies for smoke.sh ----------------------------------------------------------------------------- You probably need the following installed to run the smoke test: - git - python3-coverage-test-runner - python3-cliapp - python3-jinja2 - cmdtest 0.31 or later - qemu-utils - parted - kpartx - debootstrap - expect - qemu-system Tutorial ----------------------------------------------------------------------------- To use vmdb2, git clone the source and at the root of the source tree run the following command: sudo ./vmdb2 --output simple.img simple.vmdb --log simple.log `--output simple.img` specifies that the output image is called `simple.img`, the specification is `simple.vmdb` and the log file goes to `simple.log`. Plugins and steps ----------------------------------------------------------------------------- The `vmdb2` architecture consists of a main program that reads the input file, finds a matching "step runner" for each step used in the input file, and then runs the steps in order. If there's a problem, it runs corresponding "teardown" steps in reverse order of the steps. A step might be "mount this filesystem", and the corresponding teardown is "unmount". Steps (and teardowns) are provided by plugins; see the `vmdb/plugins` directory in the source tree. Steps are intended to be very cohesive and lowly coupled. They may share some state (such as mounted filesystems) via the `State` object, but not in any other way. A plugin may provide multiple steps. See the plugin directory for which steps currently exist. A list of steps that will become incomplete as soon as development continues: * chroot (run shell snippet in chroot) * shell (run shell snippet without chroot) * debootstrap (run deboostrap) * apt (install packagers in chroot with apt) * mkimg (create disk image) * mklabel (create partition table on a disk image) * mkpart (create partition) * mkfs (create filesystem in a partition) * mount (mount filesystem, teardown unmounts it automatically) See `simple.vmdb` for examples. Note how the file uses Jinja2 templating for value fields to get value of `--output` in the right places. Also note how creating a partition or mounting a filesystem assigns a "tag" that can be referenced in steps where the partition/filesystem is needed, without having to know the actual path to the device node or mount point. Writing plugins ----------------------------------------------------------------------------- More step runners would be good, and will be added based on actual reported needs by users ("I need to have this to..."), not speculatively ("This seems like a good idea"). To write a plugin, see the existing ones for examples, and put it in `vmdb/plugins/foo_plugin.py` for some value of `foo`. Plugins are meant to be very easy to write. If not, there's probably something wrong with `vmdb2`. Please raise the issue. Hacking ----------------------------------------------------------------------------- To run automated tests: ./check This only runs the unit tests and build tests. To run a smoke test that actually builds and boots images: sudo ./smoke.sh cache.tar.gz where `cache.tar.gz` caches the debootstrap output for a future run. You'll need the yarn program (part of the [cmdtest][] package), and also [CoverageTestRunner][] for running the unit tests. [cmdtest]: http://liw.fi/cmdtest/ [CoverageTestRunner]: http://liw.fi/coverage-test-runner/ Contact ----------------------------------------------------------------------------- To contact Lars, email is best: `liw@liw.fi`. There is an IRC channel for vmdb2: irc.oftc.net network, `#vmdb2`. Legalese ----------------------------------------------------------------------------- Copyright 2017-2019 Lars Wirzenius This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . =*= License: GPL-3+ =*= vmdb2/ansible.vmdb0000644000175000017500000000147013577177751013423 0ustar gwolfgwolf# This is a sample VMDB2 input file to specify a simple # system that boots on a PC with BIOS. steps: - mkimg: "{{ output }}" size: 4G - mklabel: msdos device: "{{ output }}" - mkpart: primary device: "{{ output }}" start: 0% end: 100% part-tag: root-part - mkfs: ext4 partition: root-part - mount: root-part fs-tag: root-fs - unpack-rootfs: root-fs - debootstrap: stretch mirror: http://deb.debian.org/debian target: root-fs unless: rootfs_unpacked - apt: install packages: - linux-image-amd64 - python fs-tag: root-fs unless: rootfs_unpacked - cache-rootfs: root-fs unless: rootfs_unpacked - ansible: root-fs playbook: ansible.yml - grub: bios root-fs: root-fs root-part: root-part device: "{{ output }}" vmdb2/ansible.yml0000644000175000017500000000013713577177751013273 0ustar gwolfgwolf- hosts: image roles: - set_hostname - unset_root_password vars: hostname: pc vmdb2/check0000755000175000017500000000040513577177751012134 0ustar gwolfgwolf#!/bin/sh set -eu python3 -m CoverageTestRunner --ignore-missing-from=without-tests yarns vmdb yarn \ --shell=python3 \ --shell-arg '' \ --shell-library yarns/lib.py \ --env "PYTHONPATH=$(pwd)/yarns" \ --cd-datadir \ yarns/*.yarn "$@" vmdb2/code-of-conduct.md0000644000175000017500000000622213577177751014427 0ustar gwolfgwolf# Contributor Covenant Code of Conduct ## Our Pledge In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation. ## Our Standards Examples of behavior that contributes to creating a positive environment include: * Using welcoming and inclusive language * Being respectful of differing viewpoints and experiences * Gracefully accepting constructive criticism * Focusing on what is best for the community * Showing empathy towards other community members Examples of unacceptable behavior by participants include: * The use of sexualized language or imagery and unwelcome sexual attention or advances * Trolling, insulting/derogatory comments, and personal or political attacks * Public or private harassment * Publishing others' private information, such as a physical or electronic address, without explicit permission * Other conduct which could reasonably be considered inappropriate in a professional setting ## Our Responsibilities Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior. Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful. ## Scope This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers. ## Enforcement Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at liw@liw.fi. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately. Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership. ## Attribution This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html [homepage]: https://www.contributor-covenant.org vmdb2/debian/0000755000175000017500000000000013577177751012354 5ustar gwolfgwolfvmdb2/debian/changelog0000644000175000017500000001003013577177751014220 0ustar gwolfgwolfvmdb2 (0.13.2+git20191206-2) unstable; urgency=medium * Updated Vcs-Git and Vcs-Browser to the repositories used for Debian development (not upstream) * standards-version 4.3.0→4.4.1 (no changes needed) -- Gunnar Wolf Tue, 10 Dec 2019 17:39:34 -0600 vmdb2 (0.13.2+git20191206-1) unstable; urgency=medium * Adding Vcs-Git and Vcs-Browser fields to debian/control * Fix rootfs-unpacked to be rootfs_unpacked in vmdb2.mdwn. (Closes:#922709) * Drop build-dependency on pylint (its use has already been removed upstream) (Closs: #945480) * Patch for #922660 was incorporated upstream (Closes: #922660) -- Gunnar Wolf Fri, 06 Dec 2019 10:47:42 -0600 vmdb2 (0.13.2+git20190215-1) unstable; urgency=medium * Stop specifying xelatex as the PDF engine for building documentation, and add both texlive-fonts-recommended and lmodern as build-dependencies, as both issues caused the package to FTBFS under pbuilder (Closes: #922355) * Add build-dependency on dh-python, lintian is happier now -- Gunnar Wolf Fri, 15 Feb 2019 11:47:15 -0600 vmdb2 (0.13.2+git20190214-1) unstable; urgency=medium * New upstream version. * Add dependency and build-dependency on qemu-utils, for qemu-img. * Updated Homepage. * Build the manual and include it in the Debian package. (Closes: #907614) * Add build-dependcies for building docs. * Taking over maintainership, as upstream author retired from Debian, (Closes: #914027) * Bumping up standards-version 3.9.8 → 4.3.0, dh compat level 9 → 12 * Get the version information from debian/changelog instead of git, fixing potential (not reported) FTBFS -- Gunnar Wolf Tue, 12 Feb 2019 09:17:52 -0600 vmdb2 (0.13.2-1) unstable; urgency=medium * New upstream version. -- Lars Wirzenius Sun, 06 May 2018 09:26:24 +0300 vmdb2 (0.13.1-1) unstable; urgency=medium * New upstream version. -- Lars Wirzenius Mon, 30 Apr 2018 17:51:13 +0300 vmdb2 (0.13-1) unstable; urgency=medium * New upstream version. * Use new pandoc options (Closes: #897107) -- Lars Wirzenius Mon, 30 Apr 2018 11:23:38 +0300 vmdb2 (0.12-1) unstable; urgency=medium * New upstream version. * debian/copyright: Added Stuart. * debian/control: Updated Homepage, patch from Geert Stappers. -- Lars Wirzenius Sat, 24 Feb 2018 11:05:32 +0200 vmdb2 (0.11-1) unstable; urgency=medium * New upstream version. -- Lars Wirzenius Sat, 10 Feb 2018 17:56:41 +0200 vmdb2 (0.10-1) unstable; urgency=medium * New upstream version. -- Lars Wirzenius Sat, 10 Feb 2018 17:04:30 +0200 vmdb2 (0.9-1) unstable; urgency=medium * New upstream version. * Add build-dep on python3-coverage-test-runner, pylint3. * Recommend ansible when vmdb2 is installed. -- Lars Wirzenius Wed, 11 Oct 2017 18:46:59 +0300 vmdb2 (0.8-1) unstable; urgency=medium * New upstream version. -- Lars Wirzenius Sat, 29 Jul 2017 16:58:42 +0300 vmdb2 (0.7-1) unstable; urgency=medium * New upstream version. -- Lars Wirzenius Sun, 18 Jun 2017 16:03:28 +0300 vmdb2 (0.6-1) unstable; urgency=medium * New upstream version. -- Lars Wirzenius Sun, 11 Jun 2017 15:36:46 +0300 vmdb2 (0.5-1) unstable; urgency=medium * New upstream version. -- Lars Wirzenius Sun, 04 Jun 2017 13:49:07 +0300 vmdb2 (0.4-1) unstable; urgency=medium * New upstream version. -- Lars Wirzenius Sat, 03 Jun 2017 22:42:46 +0300 vmdb2 (0.3-1) unstable; urgency=medium * New upstream version. -- Lars Wirzenius Sun, 21 May 2017 16:27:35 +0300 vmdb2 (0.2-1) unstable; urgency=medium * New upstream version. -- Lars Wirzenius Sun, 14 May 2017 16:13:30 +0300 vmdb2 (0.1-1) unstable; urgency=low * Initial packaging. This is not intended to be uploaded to Debian, so no closing of an ITP bug. -- Lars Wirzenius Sat, 13 May 2017 23:42:59 +0300 vmdb2/debian/compat0000644000175000017500000000000313577177751013553 0ustar gwolfgwolf12 vmdb2/debian/control0000644000175000017500000000177013577177751013764 0ustar gwolfgwolfSource: vmdb2 Homepage: https://vmdb2.liw.fi/ Maintainer: Gunnar Wolf Section: admin Priority: optional Standards-Version: 4.4.1 Vcs-Browser: https://salsa.debian.org/debian/vmdb2/ Vcs-Git: https://salsa.debian.org/debian/vmdb2.git Build-Depends: debhelper (>= 12), python3-all, python3-coverage-test-runner, dh-python, pandoc (>= 2.1.2~), texlive-xetex, lmodern, fonts-freefont-ttf, texlive-fonts-recommended, python3-cliapp, python3-jinja2, cmdtest, python3-yaml, debootstrap, qemu-utils, parted, kpartx Package: vmdb2 Architecture: all Depends: ${python3:Depends}, ${misc:Depends}, python3, python3-cliapp, python3-jinja2, python3-yaml, cmdtest, debootstrap, qemu-utils, parted, kpartx Recommends: ansible Description: creator of disk images with Debian installed vmdb2 will be a successor of vmdebootstrap. It will create disk images for virtual machines and real hardware, with partitioning, and a boot loader, and a Debian installation. vmdb2/debian/copyright0000644000175000017500000000172313577177751014312 0ustar gwolfgwolfFormat: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ Upstream-Name: vmdb2 Upstream-Contact: Lars Wirzenius Source: http://git.liw.fi/vmdb2 Files: * Copyright: 2017-2018, Lars Wirzenius, Stuart Prescott License: GPL-3+ This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. . This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. . You should have received a copy of the GNU General Public License along with this program. If not, see . . On a Debian system, you can find a copy of GPL version 3 at /usr/share/common-licenses/GPL-3 . vmdb2/debian/docs0000644000175000017500000000002113577177751013220 0ustar gwolfgwolfyarns/yarns.html vmdb2/debian/patches/0000755000175000017500000000000013577177751014003 5ustar gwolfgwolfvmdb2/debian/patches/get_version_from_changelog0000644000175000017500000000125113577177751021303 0ustar gwolfgwolfAuthor: Gunnar Wolf "$tmp/prelude.mdwn" pandoc \ vmdb2/debian/patches/series0000644000175000017500000000003313577177751015214 0ustar gwolfgwolfget_version_from_changelog vmdb2/debian/rules0000755000175000017500000000027013577177751013433 0ustar gwolfgwolf#!/usr/bin/make -f export PYBUILD_NAME=vmdb2 %: dh $@ --with=python3 --buildsystem=pybuild override_dh_auto_test: ifeq (,$(findstring nocheck,$(DEB_BUILD_OPTIONS))) ./check endif vmdb2/debian/source/0000755000175000017500000000000013577177751013654 5ustar gwolfgwolfvmdb2/debian/source/format0000644000175000017500000000001413577177751015062 0ustar gwolfgwolf3.0 (quilt) vmdb2/format.sh0000755000175000017500000000126013577177751012760 0ustar gwolfgwolf#!/bin/sh set -eu cleanup() { rm -rf "$tmp" } tmp="$(mktemp -d)" trap cleanup EXIT version="$(git describe)" sed "s/^date: .*/date: $version/" vmdb2.mdwn > "$tmp/prelude.mdwn" pandoc \ --self-contained \ --standalone \ --css vmdb2.css \ --toc \ --number-sections \ -o vmdb2.html \ "$tmp/prelude.mdwn" vmdb/plugins/*.mdwn pandoc \ --toc \ --number-sections \ -Vdocumentclass=report \ -Vgeometry:a4paper \ -Vfontsize:12pt \ -Vmainfont:FreeSerif \ -Vsansfont:FreeSans \ -Vmonofont:FreeMonoBold \ '-Vgeometry:top=2cm, bottom=2.5cm, left=2cm, right=1cm' \ -o vmdb2.pdf \ "$tmp/prelude.mdwn" vmdb/plugins/*.mdwn vmdb2/lvm2.lukskey0000644000175000017500000000001013577177751013412 0ustar gwolfgwolfhunter2 vmdb2/lvm2.vmdb0000644000175000017500000000225313577177751012666 0ustar gwolfgwolf# This is a sample VMDB2 input file to specify a simple # system that boots on a PC with BIOS. Uses LVM2. steps: - mkimg: "{{ image }}" size: 4G - mklabel: msdos device: "{{ image }}" - mkpart: primary device: "{{ image }}" start: 1M end: 100M tag: /boot - mkpart: primary device: "{{ image }}" start: 100M end: 2G tag: /rootpv - cryptsetup: /rootpv tag: rootpv_crypt key-cmd: echo lvm2.lukskey - vgcreate: rootvg physical: - rootpv_crypt - lvcreate: rootvg name: rootfs size: 1G - mkfs: ext2 partition: boot - mkfs: ext4 partition: rootfs - mount: /boot - mount: /boot - mount: rootfs - unpack-rootfs: rootfs - debootstrap: stretch mirror: http://deb.debian.org/debian target: rootfs unless: rootfs_unpacked - apt: install packages: - linux-image-amd64 tag: rootfs unless: rootfs_unpacked - cache-rootfs: rootfs unless: rootfs_unpacked - chroot: rootfs shell: | sed -i '/^root:[^:]*:/s//root::/' /etc/passwd echo pc-vmdb2 > /etc/hostname - grub: bios tag: rootfs image-dev: "{{ image }}" console: serial vmdb2/pc.vmdb0000644000175000017500000000172113577177751012407 0ustar gwolfgwolf# This is a sample VMDB2 input file to specify a simple # system that boots on a PC with BIOS. steps: - mkimg: "{{ output }}" size: 4G - mklabel: msdos device: "{{ output }}" - mkpart: primary device: "{{ output }}" start: 0% end: 100% part-tag: root-part - mkfs: ext4 partition: root-part - mount: root-part fs-tag: root-fs - unpack-rootfs: root-fs - debootstrap: stretch mirror: http://deb.debian.org/debian target: root-fs unless: rootfs_unpacked - apt: install packages: - linux-image-amd64 fs-tag: root-fs unless: rootfs_unpacked - cache-rootfs: root-fs unless: rootfs_unpacked - chroot: root-fs shell: | apt -y install python - shell: | printf '[pc]\n%s hostname=pc\n' "$ROOT" > pc.inventory ansible-playbook -i pc.inventory -c chroot pc.yml root-fs: root-fs - grub: bios root-fs: root-fs root-part: root-part device: "{{ output }}" vmdb2/roles/0000755000175000017500000000000013577177751012256 5ustar gwolfgwolfvmdb2/roles/set_hostname/0000755000175000017500000000000013577177751014747 5ustar gwolfgwolfvmdb2/roles/set_hostname/tasks/0000755000175000017500000000000013577177751016074 5ustar gwolfgwolfvmdb2/roles/set_hostname/tasks/main.yml0000644000175000017500000000011713577177751017542 0ustar gwolfgwolf- name: set /etc/hostname shell: | echo "{{ hostname }}" > /etc/hostname vmdb2/roles/unset_root_password/0000755000175000017500000000000013577177751016401 5ustar gwolfgwolfvmdb2/roles/unset_root_password/tasks/0000755000175000017500000000000013577177751017526 5ustar gwolfgwolfvmdb2/roles/unset_root_password/tasks/main.yml0000644000175000017500000000013113577177751021170 0ustar gwolfgwolf- name: unset root password shell: | sed -i '/^root:[^:]*:/s//root::/' /etc/passwd vmdb2/setup.py0000755000175000017500000000477713577177751012666 0ustar gwolfgwolf#!/usr/bin/env python3 # Copyright (C) 2017 Lars Wirzenius # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. from distutils.core import setup, Extension from distutils.cmd import Command from distutils.command.build import build from distutils.command.clean import clean import os import glob import cliapp import vmdb class Build(build): def run(self): build.run(self) self.build_manpage('vmdb2', '') self.format_yarns() def build_manpage(self, program, lang): return # building manpage fails on by unstable CI worker for mysterious # reasons, will re-enable later print('building manpage for %s (lang=%s)' % (program, lang)) self.generate_troff(program, lang) self.format_man_as_txt(program) def generate_troff(self, program, lang): with open('%s.1%s' % (program, lang), 'w') as f: cliapp.runcmd( ['python3', program, '--generate-manpage=%s.1%s.in' % (program, lang), '--output=%s.1' % program], stdout=f) def format_man_as_txt(self, program): env = dict(os.environ) env['MANWIDTH'] = '80' with open('%s.1.txt' % program, 'w') as f: cliapp.runcmd( ['man', '-l', '%s.1' % program], ['col', '-b'], stdout=f, env=env) def format_yarns(self): print('building yarns') cliapp.runcmd(['make', '-C', 'yarns']) setup( name='vmdb2', version=vmdb.__version__, description='create disk image with Debian installed', author='Lars Wirzenius', author_email='liw@liw.fi', url='http://liw.fi/vmdebootstrap/', scripts=['vmdb2'], packages=['vmdb', 'vmdb.plugins'], data_files=[('share/man/man1', glob.glob('*.1'))], cmdclass={ 'build': Build, }, ) vmdb2/simple.vmdb0000644000175000017500000000076313577177751013303 0ustar gwolfgwolfsteps: - mklabel: gpt device: "{{ output }}" - mkpart: primary device: "{{ output }}" start: 0% end: 100% part-tag: root-part - mkfs: ext4 partition: root-part - mount: root-part fs-tag: root-fs - debootstrap: stretch mirror: http://deb.debian.org/debian target: root-fs - apt: install packages: - linux-image-amd64 fs-tag: root-fs - shell: | echo Disk usage of this installation: du -sh "$ROOT" root-fs: root-fs vmdb2/smoke-pc.vmdb0000644000175000017500000000152113577177751013521 0ustar gwolfgwolfsteps: - mkimg: "{{ output }}" size: 4G - mklabel: msdos device: "{{ output }}" - mkpart: primary device: "{{ output }}" start: 1M end: 10M tag: unused - mkpart: primary device: "{{ output }}" start: 10M end: 100% tag: rootfs - kpartx: "{{ output }}" - mkfs: ext4 partition: rootfs label: smoke - mount: rootfs - unpack-rootfs: rootfs - debootstrap: stretch mirror: http://deb.debian.org/debian target: rootfs unless: rootfs_unpacked - apt: install packages: - linux-image-amd64 tag: rootfs unless: rootfs_unpacked - cache-rootfs: rootfs unless: rootfs_unpacked - chroot: rootfs shell: | sed -i '/^root:[^:]*:/s//root::/' /etc/passwd echo pc-vmdb2 > /etc/hostname - grub: bios tag: rootfs console: serial vmdb2/smoke-uefi.vmdb0000644000175000017500000000160013577177751014045 0ustar gwolfgwolfsteps: - mkimg: "{{ output }}" size: 4G - mklabel: gpt device: "{{ output }}" - mkpart: primary device: "{{ output }}" start: 0% end: 1G tag: efifs - mkpart: primary device: "{{ output }}" start: 1G end: 100% tag: rootfs - kpartx: "{{ output }}" - mkfs: vfat partition: efifs - mkfs: ext4 partition: rootfs label: smoke - mount: rootfs - unpack-rootfs: rootfs - debootstrap: stretch mirror: http://deb.debian.org/debian target: rootfs unless: rootfs_unpacked - apt: install packages: - linux-image-amd64 tag: rootfs unless: rootfs_unpacked - cache-rootfs: rootfs unless: rootfs_unpacked - chroot: rootfs shell: | sed -i '/^root:[^:]*:/s//root::/' /etc/passwd echo pc-vmdb2 > /etc/hostname - grub: uefi tag: rootfs efi: efifs console: serial vmdb2/smoke.sh0000755000175000017500000000013513577177751012606 0ustar gwolfgwolf#!/bin/sh set -eu tarball="$1" shift yarn smoke.yarn --env ROOTFS_TARBALL="$tarball" "$@" vmdb2/smoke.yarn0000644000175000017500000000556713577177751013160 0ustar gwolfgwolf# Smoke test vmdb2-built images This yarn file builds a basic image, and runs it under Qemu. The image is configured to have a serial console enabled, and the scenario below will log in as root (no password), and command the virtual machine to power off. If this works, the image is at least minimally functional, so the "smoke test" passes. More elaborate testing can be added, but smoke testing is enough for now. To run this yarn file, you need to run it as root (since it needs to build an image as root), and you need to run on an amd64 system (and thus build an amd64 image). Additionally, you need to add the folloing option to yarn: EXAMPLE --env ROOTFS_TARBALL=/path/to/rootfs/tarball This means vmdb2 will use the given tarball when creating an image, and if the tarball doesn't exist yet, it will create it. This makes testing multiple time much faster. SCENARIO smoke test image WHEN user runs vmdb smoke-pc.vmdb --output smoke-pc.img THEN user can BIOS boot smoke-pc.img and power it off from root shell WHEN user runs vmdb smoke-uefi.vmdb --output smoke-uefi.img THEN user can UEFI boot smoke-uefi.img and power it off from root shell IMPLEMENTS WHEN user runs vmdb (\S+) --output (\S+) "$SRCDIR/vmdb2" --no-default-config "$SRCDIR/$MATCH_1" \ --output "$DATADIR/$MATCH_2" \ --log "$DATADIR/vmdb.log" \ --verbose \ --rootfs-tarball "$ROOTFS_TARBALL" IMPLEMENTS THEN user can BIOS boot (\S+) and power it off from root shell cd "$DATADIR" img="$MATCH_1" cat << EOF > run.sh qemu-system-x86_64 -drive file="$img",format=raw -m 1024 -nographic EOF chmod a+rx run.sh cat << EOF > expect.txt set timeout 300 proc abort {} { puts "ERROR ERROR\n" exit 1 } spawn ./run.sh expect "login: " send "root\n" expect "# " send "poweroff\r" set timeout 5 expect { "reboot: Power down" {puts poweroffing\n} eof abort timeout abort } expect eof EOF expect -d expect.txt > expect.out IMPLEMENTS THEN user can UEFI boot (\S+) and power it off from root shell cd "$DATADIR" img="$MATCH_1" cat << EOF > run.sh cp /usr/share/OVMF/OVMF_VARS.fd . qemu-system-x86_64 \ -m 1024 \ -drive if=pflash,format=raw,unit=0,file=/usr/share/ovmf/OVMF.fd,readonly=on \ -drive if=pflash,format=raw,unit=1,file=OVMF_VARS.fd \ -drive format=raw,file="$img" \ -nographic EOF chmod a+rx run.sh cat << EOF > expect.txt set timeout 300 proc abort {} { puts "ERROR ERROR\n" exit 1 } spawn ./run.sh expect "login: " send "root\n" expect "# " send "poweroff\r" set timeout 5 expect { "reboot: Power down" {puts poweroffing\n} eof abort timeout abort } expect eof EOF expect -d expect.txt > expect.out vmdb2/uefi.vmdb0000644000175000017500000000155313577177751012740 0ustar gwolfgwolf# This is a sample VMDB2 input file that specifies a simple system for # a PC that boots with UEFI. steps: - mkimg: "{{ output }}" size: 4G - mklabel: gpt device: "{{ output }}" - mkpart: primary device: "{{ output }}" start: 0% end: 1G part-tag: efi-part - mkpart: primary device: "{{ output }}" start: 1G end: 100% part-tag: root-part - mkfs: vfat partition: efi-part - mkfs: ext4 partition: root-part - mount: root-part fs-tag: root-fs - unpack-rootfs: root-fs - debootstrap: stretch mirror: http://deb.debian.org/debian target: root-fs unless: rootfs_unpacked - apt: install packages: - linux-image-amd64 fs-tag: root-fs unless: rootfs_unpacked - cache-rootfs: root-fs unless: rootfs_unpacked - grub: uefi rootfs: root-fs efi-part: efi-part vmdb2/vmdb/0000755000175000017500000000000013577177751012062 5ustar gwolfgwolfvmdb2/vmdb/__init__.py0000644000175000017500000000224013577177751014171 0ustar gwolfgwolf# Copyright 2017 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= from .version import __version__, __version_info__ from .state import State from .step_list import ( StepRunnerList, StepRunnerInterface, NoMatchingRunner, StepError, ) from .runcmd import ( runcmd, runcmd_chroot, set_verbose_progress, progress, error, ) from .tags import Tags, UnknownTag, TagInUse, AlreadyHasDev, AlreadyMounted from .unmount import unmount, NotMounted from .spec import ( Spec, expand_templates, ) from .app import Vmdb2 vmdb2/vmdb/app.py0000644000175000017500000000747113577177751013225 0ustar gwolfgwolf# Copyright 2017 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= import logging import sys import cliapp import vmdb class Vmdb2(cliapp.Application): def add_settings(self): self.settings.string( ['image'], 'create image file FILE', metavar='FILE') self.settings.boolean( ['verbose', 'v'], 'verbose output') def setup(self): self.step_runners = vmdb.StepRunnerList() def process_args(self, args): if len(args) != 1: sys.exit("No image specification was given on the command line.") vmdb.set_verbose_progress(self.settings['verbose']) spec = self.load_spec_file(args[0]) state = vmdb.State() state.tags = vmdb.Tags() params = self.create_template_vars(state) steps = spec.get_steps(params) # Check that we have step runners for each step for step in steps: self.step_runners.find(step) steps_taken, core_meltdown = self.run_steps(steps, state) if core_meltdown: vmdb.progress('Something went wrong, cleaning up!') else: vmdb.progress('All went fine, cleaning up.') self.run_teardowns(steps_taken, state) if core_meltdown: logging.error('An error occurred, exiting with non-zero exit code') sys.exit(1) def load_spec_file(self, filename): vmdb.progress('Load spec file {}'.format(filename)) spec = vmdb.Spec() spec.load_file(filename) return spec def run_steps(self, steps, state): return self.run_steps_helper( steps, state, 'Running step: %r', 'run', False) def run_teardowns(self, steps, state): return self.run_steps_helper( list(reversed(steps)), state, 'Running teardown: %r', 'teardown', True) def run_steps_helper(self, steps, state, msg, method_name, keep_going): core_meltdown = False steps_taken = [] even_if_skipped = method_name + '_even_if_skipped' for step in steps: try: logging.info(msg, step) steps_taken.append(step) runner = self.step_runners.find(step) if runner.skip(step, self.settings, state): logging.info('Skipping as requested by unless') method_names = [even_if_skipped] else: method_names = [method_name, even_if_skipped] methods = [ getattr(runner, name) for name in method_names if hasattr(runner, name) ] for method in methods: logging.info('Calling %s', method) method(step, self.settings, state) except BaseException as e: vmdb.error(str(e)) core_meltdown = True if not keep_going: break return steps_taken, core_meltdown def create_template_vars(self, state): vars = dict() for key in self.settings: vars[key] = self.settings[key] vars.update(state.as_dict()) return vars vmdb2/vmdb/plugins/0000755000175000017500000000000013577177751013543 5ustar gwolfgwolfvmdb2/vmdb/plugins/__init__.py0000644000175000017500000000014313577177751015652 0ustar gwolfgwolf# This file exists to make vmdb.plugins be a Python package, so it can # be installed by setup.py. vmdb2/vmdb/plugins/ansible.mdwn0000644000175000017500000000230613577177751016050 0ustar gwolfgwolfStep: `ansible` ----------------------------------------------------------------------------- Run Ansible using a provided playbook, to configure the image. vmdb2 sets up Ansible so that it treats the image as the host being configured (via the `chroot` connecion). The image MUST have Python installed (version 2 or 3 depending on Ansible version). Step keys: * `ansible` — REQUIRED; value is the tag of the root filesystem. * `playbook` — REQUIRED; value is the filename of the Ansible playbook, relative to the .vmdb file. Example (in the .vmdb file): - apt: install tag: root packages: [python] - ansible: root playbook: foo.yml Example (`foo.yml`): - hosts: image tasks: - name: "set /etc/hostname" shell: | echo "{{ hostname }}" > /etc/hostname - name: "unset root password" shell: | sed -i '/^root:[^:]*:/s//root::/' /etc/passwd - name: "configure networking" copy: content: | auto eth0 iface eth0 inet dhcp iface eth0 inet6 auto dest: /etc/network/interfaces.d/wired vars: hostname: discworld vmdb2/vmdb/plugins/ansible_plugin.py0000644000175000017500000000364013577177751017113 0ustar gwolfgwolf# Copyright 2017 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= import os import tempfile import cliapp import vmdb class AnsiblePlugin(cliapp.Plugin): def enable(self): self.app.step_runners.add(AnsibleStepRunner()) class AnsibleStepRunner(vmdb.StepRunnerInterface): def get_required_keys(self): return ['ansible', 'playbook'] def run(self, step, settings, state): tag = step['ansible'] playbook = step['playbook'] mount_point = state.tags.get_mount_point(tag) state.ansible_inventory = self.create_inventory(mount_point) vmdb.progress( 'Created {} for Ansible inventory'.format(state.ansible_inventory)) env = dict(os.environ) env['ANSIBLE_NOCOWS'] = '1' vmdb.runcmd( ['ansible-playbook', '-c', 'chroot', '-i', state.ansible_inventory, playbook], env=env) def teardown(self, step, settings, state): if hasattr(state, 'ansible_inventory'): vmdb.progress('Removing {}'.format(state.ansible_inventory)) os.remove(state.ansible_inventory) def create_inventory(self, chroot): fd, filename = tempfile.mkstemp() os.write(fd, '[image]\n{}\n'.format(chroot).encode()) os.close(fd) return filename vmdb2/vmdb/plugins/apt.mdwn0000644000175000017500000000075313577177751015223 0ustar gwolfgwolfStep: `apt` ----------------------------------------------------------------------------- Install packages using apt, which needs to already have been installed. Step keys: * `apt` — REQUIRED; value MUST be `install`. * `tag` — REQUIRED; value is the tag for the root filesystem. * `packages` — REQUIRED; value is a list of packages to install. Example (in the .vmdb file): - apt: install tag: root packages: - python - linux-image-amd64 vmdb2/vmdb/plugins/apt_plugin.py0000644000175000017500000000460013577177751016257 0ustar gwolfgwolf# Copyright 2017 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= import os import cliapp import vmdb class AptPlugin(cliapp.Plugin): def enable(self): self.app.step_runners.add(AptStepRunner()) class AptStepRunner(vmdb.StepRunnerInterface): def get_required_keys(self): return ['apt', 'packages'] def run(self, step, settings, state): operation = step['apt'] if operation != 'install': raise Exception('"apt" must always have value "install"') packages = step['packages'] tag = step.get('tag') if tag is None: tag = step['fs-tag'] mount_point = state.tags.get_mount_point(tag) if not self.got_eatmydata(state): self.install_packages(mount_point, [], ['eatmydata']) state.got_eatmydata = True self.install_packages(mount_point, ['eatmydata'], packages) if step.get('clean', True): self.clean_cache(mount_point) def got_eatmydata(self, state): return hasattr(state, 'got_eatmydata') and getattr(state, 'got_eatmydata') def install_packages(self, mount_point, argv_prefix, packages): env = os.environ.copy() env['DEBIAN_FRONTEND'] = 'noninteractive' vmdb.runcmd_chroot( mount_point, argv_prefix + ['apt-get', 'update'], env=env) vmdb.runcmd_chroot( mount_point, argv_prefix + ['apt-get', '-y', '--no-show-progress', 'install'] + packages, env=env) def clean_cache(self, mount_point): env = os.environ.copy() env['DEBIAN_FRONTEND'] = 'noninteractive' vmdb.runcmd_chroot( mount_point, ['apt-get', 'clean'], env=env) vmdb2/vmdb/plugins/chroot.mdwn0000644000175000017500000000145113577177751015731 0ustar gwolfgwolfStep: `chroot` ----------------------------------------------------------------------------- Run a shell snippet in a chroot inside the image. Step keys: * `chroot` — REQUIRED; value is the tag for the root filesystem. * `shell` — REQUIRED; the shell snippet to run Example (in the .vmdb file): - chroot: root shell: | echo I am in chroot Step: `shell` ----------------------------------------------------------------------------- Run a shell snippet on the host. This is not run in a chroot, and can access the host system. Step keys: * `root-fs` — REQUIRED; value is the tag for the root filesystem. * `shell` — REQUIRED; the shell snippet to run Example (in the .vmdb file): - root-fs: root shell: | echo I am in NOT in chroot. vmdb2/vmdb/plugins/chroot_plugin.py0000644000175000017500000000312013577177751016765 0ustar gwolfgwolf# Copyright 2017 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= import os import cliapp import vmdb class ChrootPlugin(cliapp.Plugin): def enable(self): self.app.step_runners.add(ChrootStepRunner()) self.app.step_runners.add(ShellStepRunner()) class ChrootStepRunner(vmdb.StepRunnerInterface): def get_required_keys(self): return ['chroot', 'shell'] def run(self, step, settings, state): fs_tag = step['chroot'] shell = step['shell'] mount_point = state.tags.get_mount_point(fs_tag) vmdb.runcmd_chroot(mount_point, ['sh', '-c', shell]) class ShellStepRunner(vmdb.StepRunnerInterface): def get_required_keys(self): return ['shell', 'root-fs'] def run(self, step, settings, state): shell = step['shell'] fs_tag = step['root-fs'] env = dict(os.environ) env['ROOT'] = state.tags.get_mount_point(fs_tag) vmdb.runcmd(['sh', '-c', shell], env=env) vmdb2/vmdb/plugins/create_file.mdwn0000644000175000017500000000255313577177751016701 0ustar gwolfgwolfStep: create-dir ----------------------------------------------------------------------------- Create a directory in the target filesystem Step keys: * `create-dir` — REQUIRED; the full (starting from the new filesystem root) path name of the directory to create. It will work as a `mkdir -p` — Any intermediate directories that do not yet exist will be created. * `perm` — OPTIONAL; the numeric (octal) representation of the directory's permissions. Defaults to 0755. * `uid` — OPTIONAL; the numeric user ID of the directory's owner. Defaults to 0 (root). * `gid` — OPTIONAL; the numeric user ID of the directory's group. Defaults to 0 (root). Step: create-file ----------------------------------------------------------------------------- Step keys: * `create-file` — REQUIRED; the full (starting from the new filesystem root) path name of the file to create. It will *not* create any directories; if they need to be created, please use `create-dir` first. * `contents` — REQUIRED; the contents to be written to the generated file. * `perm` — OPTIONAL; the numeric (octal) representation of the file's permissions. Defaults to 0644. * `uid` — OPTIONAL; the numeric user ID of the file's owner. Defaults to 0 (root). * `gid` — OPTIONAL; the numeric user ID of the file's group. Defaults to 0 (root). vmdb2/vmdb/plugins/create_file_plugin.py0000644000175000017500000000510213577177751017733 0ustar gwolfgwolf# Copyright 2019 Gunnar Wolf # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= import cliapp import vmdb import os import logging class CreateFilePlugin(cliapp.Plugin): def enable(self): self.app.step_runners.add(CreateFileStepRunner()) self.app.step_runners.add(CreateDirStepRunner()) class CreateFileStepRunner(vmdb.StepRunnerInterface): def get_required_keys(self): return ['create-file', 'contents'] def run(self, step, settings, state): root = state.tags.get_mount_point('/') newfile = step['create-file'] contents = step['contents'] perm = step.get('perm') uid = step.get('uid') gid = step.get('gid') filename = '/'.join([root,newfile]) if perm: perm = int(perm, 8) else: perm = 0o0644 if uid: uid = int(uid) else: uid = 0 if gid: gid = int(gid) else: gid = 0 logging.info('Creating file %s, uid %d, gid %d, perms %o' % (filename, uid, gid, perm)) fd = open(filename, 'w') fd.write(contents) fd.close os.chown(filename, uid, gid) os.chmod(filename, perm) class CreateDirStepRunner(vmdb.StepRunnerInterface): def get_required_keys(self): return ['create-dir'] def run(self, step, settings, state): root = state.tags.get_mount_point('/') newdir = step['create-dir'] path = '/'.join([root, newdir]) perm = step.get('perm') uid = step.get('uid') gid = step.get('gid') if perm: perm = int(perm, 8) else: perm = 0o0755 if uid: uid = int(uid) else: uid = 0 if gid: gid = int(gid) else: gid = 0 logging.info('Creating directory %s, uid %d, gid %d, perms %o' % (path, uid, gid, perm)) os.makedirs(path, perm) os.chown(path, uid, gid) vmdb2/vmdb/plugins/debootstrap.mdwn0000644000175000017500000000105313577177751016757 0ustar gwolfgwolfStep: `debootstrap` ----------------------------------------------------------------------------- Install packages using apt, which needs to already have been installed. Step keys: * `debootstrap` — REQUIRED; value is the codename of the Debian release to install: `stretch`, `buster`, etc. * `target` — REQUIRED; value is the tag for the root filesystem. * `mirror` — OPTIONAL; which Debian mirror to use Example (in the .vmdb file): - debootstrap: stretch target: root mirror: http://mirror.example.com/debian vmdb2/vmdb/plugins/debootstrap_plugin.py0000644000175000017500000000312413577177751020021 0ustar gwolfgwolf# Copyright 2017 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= import cliapp import vmdb class DebootstrapPlugin(cliapp.Plugin): def enable(self): self.app.step_runners.add(DebootstrapStepRunner()) class DebootstrapStepRunner(vmdb.StepRunnerInterface): def get_required_keys(self): return ['debootstrap', 'target', 'mirror'] def run(self, step, settings, state): suite = step['debootstrap'] tag = step['target'] target = state.tags.get_mount_point(tag) mirror = step['mirror'] variant = step.get('variant', '-') if not (suite and tag and target and mirror): raise Exception('missing arg for debootstrap step') vmdb.runcmd(['debootstrap', '--variant', variant, suite, target, mirror]) def run_even_if_skipped(self, step, settings, state): tag = step['target'] target = state.tags.get_mount_point(tag) vmdb.runcmd_chroot(target, ['apt-get', 'update']) vmdb2/vmdb/plugins/echo_plugin.py0000644000175000017500000000236613577177751016420 0ustar gwolfgwolf# Copyright 2017 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= import logging import cliapp import vmdb class EchoPlugin(cliapp.Plugin): def enable(self): self.app.step_runners.add(EchoStepRunner()) class EchoStepRunner(vmdb.StepRunnerInterface): def get_required_keys(self): return ['echo'] def run(self, step, settings, state): text = step['echo'] vmdb.progress('{}'.format(text)) def teardown(self, step, settings, state): if 'teardown' in step: text = step['teardown'] vmdb.progress('{}'.format(text)) logging.info('%s', text) vmdb2/vmdb/plugins/error_plugin.py0000644000175000017500000000264513577177751016633 0ustar gwolfgwolf# Copyright 2017 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= import cliapp import vmdb class ErrorPlugin(cliapp.Plugin): def enable(self): self.app.step_runners.add(ErrorStepRunner()) class ErrorStepRunner(vmdb.StepRunnerInterface): def get_required_keys(self): return ['error', 'teardown'] def run(self, step, settings, state): # We use vmdb.progress here to get output to go to stdout, # instead of stderr. We want that for tests. vmdb.progress('{}'.format(step['error'])) raise vmdb.StepError('an error occurred') def teardown(self, step, settings, state): # We use vmdb.progress here to get output to go to stdout, # instead of stderr. We want that for tests. vmdb.progress('{}'.format(step['teardown'])) vmdb2/vmdb/plugins/grub.mdwn0000644000175000017500000000225413577177751015374 0ustar gwolfgwolfStep: `grub` ----------------------------------------------------------------------------- Install the GRUB bootloader to the image. Works on a PC, for traditional BIOS booting or modern UEFI booting. Does not (yet?) support Secure Boot. Warning: This is the least robust part of vmdb2. Step keys: * `grub` — REQUIRED; value MUST be one of `uefi` and `bios`, for a UEFI or a BIOS boot, respectively. (FIXME: these are valid for a PC; not sure what other archs require, if grub even works there.) * `tag` — REQUIRED; value is the tag for the root filesystem. * `efi` — REQUIRED for UEFI; value is the tag for the EFI filesystem. * `console` — OPTIONAL; set to `serial` to configure the image to use a serial console. * `device` — OPTIONAL; which device to install GRUB onto; this is needed when installing to a real hard drive, instead of an image. Example (in the .vmdb file): - grub: bios tag: root Same, but for UEFI: - grub: uefi tag: root efi: efi console: serial Install to a real hard disk (named with the `--image` option): - grub: uefi tag: root efi: efi image-dev: "{{ image }}" vmdb2/vmdb/plugins/grub_plugin.py0000644000175000017500000002200413577177751016430 0ustar gwolfgwolf# Copyright 2017 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= # Installing GRUB onto a disk image is a bit of a black art. I haven't # found any good documentation for it. This plugin is written based on # de-ciphering the build_openstack_image script. Here is an explanation # of what I _THINK_ is happening. # # The crucial command is grub-install. It needs a ton of options to # work correctly: see below in the code for the list, and the manpage # for an explanation of what each of them means. We will be running # grub-install in a chroot so that we use the version in the Debian # version we're installing, rather than the host system, which might # be any Debian version. # # To run grub-install in a chroot, we need to set up the chroot in # various ways. Firstly, we need to tell grub-install which device # file the image has. We can't just give it the image file itself, # since it isn't inside the chroot, so instead we arrange to have a # loop block device that covers the whole image file, and we bind # mount /dev into the chroot so the device is available. # # grub-install seems to also require /proc and /sys so we bind mount # those into the chroot as well. # # We install the UEFI version of GRUB, and for that we additionally # bind mount the EFI partition in the image. Oh yeah, you MUST have # one. # # We also make sure the right GRUB package is installed in the chroot, # before we run grub-install. # # Further, there's some configuration tweaking we need to do. See the # code. Don't ask me why they're necessary. # # For cleanliness, we also undo any bind mounts into the chroot. Don't # want to leave them in case they cause trouble. # # Note that this is currently rather strongly assuming that UEFI and # the amd64 (a.k.a. x86_64) architecture are being used. These should # probably not be hardcoded. Patch welcome. # To use this plugin: write steps to create a root filesystem, and an # VFAT filesystem to be mounted as /boot/efi. Install Debian onto the # root filesystem. Then install grub with a step like this: # # - grub: uefi # tag: root-part # efi: efi-part # # Here: "tag" is the tag for the root filesystem (and corresponding # partition), and efi is tag for the EFI partition. # # The grub step will take of the rest. import logging import os import re import cliapp import vmdb class GrubPlugin(cliapp.Plugin): def enable(self): self.app.step_runners.add(GrubStepRunner()) class GrubStepRunner(vmdb.StepRunnerInterface): def get_required_keys(self): return ['grub', 'tag'] def run(self, step, settings, state): state.grub_mounts = [] flavor = step['grub'] if flavor == 'uefi': self.install_uefi(step, settings, state) elif flavor == 'bios': self.install_bios(step, settings, state) else: raise Exception('Unknown GRUB flavor {}'.format(flavor)) def install_uefi(self, step, settings, state): if not 'efi' in step and 'efi-part' not in step: raise Exception('"efi" or "efi-part" required in UEFI GRUB installtion') vmdb.progress('Installing GRUB for UEFI') grub_package = 'grub-efi-amd64' grub_target = 'x86_64-efi' self.install_grub(step, settings, state, grub_package, grub_target) def install_bios(self, step, settings, state): vmdb.progress('Installing GRUB for BIOS') grub_package = 'grub-pc' grub_target = 'i386-pc' self.install_grub(step, settings, state, grub_package, grub_target) def install_grub(self, step, settings, state, grub_package, grub_target): console = step.get('console', None) tag = step.get('tag') if tag is None: tag = step['root-fs'] root_dev = state.tags.get_dev(tag) chroot = state.tags.get_mount_point(tag) image_dev = step.get('image-dev') if image_dev is None: image_dev = self.get_image_loop_device(root_dev) if 'efi' in step: efi = step['efi'] efi_dev = state.tags.get_dev(efi) elif 'efi-part' in step: efi = step['efi-part'] efi_dev = state.tags.get_dev(efi) else: efi_dev = None self.bind_mount_many(chroot, ['/dev', '/proc', '/sys'], state) if efi_dev: self.mount(chroot, efi_dev, '/boot/efi', state) self.install_package(chroot, grub_package) kernel_params = [ 'biosdevname=0', 'net.ifnames=0', 'consoleblank=0', 'systemd.show_status=true', 'rw', 'quiet', 'systemd.show_status=false', ] if console == 'serial': kernel_params.extend([ 'quiet', 'loglevel=3', 'rd.systemd.show_status=false', 'systemd.show_status=false', 'console=tty0', 'console=ttyS0,115200n8', ]) self.set_grub_cmdline_config(chroot, kernel_params) self.add_grub_crypto_disk(chroot) if console == 'serial': self.add_grub_serial_console(chroot) vmdb.runcmd_chroot(chroot, ['grub-mkconfig', '-o', '/boot/grub/grub.cfg']) vmdb.runcmd_chroot( chroot, [ 'grub-install', '--target=' + grub_target, '--no-nvram', '--force-extra-removable', '--no-floppy', '--modules=part_msdos part_gpt', '--grub-mkdevicemap=/boot/grub/device.map', image_dev, ] ) # self.unmount(state) def teardown(self, step, settings, state): self.unmount(state) def unmount(self, state): mounts = getattr(state, 'grub_mounts', []) mounts.reverse() while mounts: mount_point = mounts.pop() try: vmdb.unmount(mount_point) except vmdb.NotMounted as e: logging.warning(str(e)) def get_image_loop_device(self, partition_device): # We get /dev/mappers/loopXpY and return /dev/loopX # assert partition_device.startswith('/dev/mapper/loop') m = re.match(r'^/dev/mapper/(?P.*)p\d+$', partition_device) if m is None: raise Exception('Do not understand partition device name {}'.format( partition_device)) assert m is not None loop = m.group('loop') return '/dev/{}'.format(loop) def bind_mount_many(self, chroot, paths, state): for path in paths: self.mount(chroot, path, path, state, mount_opts=['--bind']) def mount(self, chroot, path, mount_point, state, mount_opts=None): chroot_path = self.chroot_path(chroot, mount_point) if not os.path.exists(chroot_path): os.makedirs(chroot_path) if mount_opts is None: mount_opts = [] vmdb.runcmd(['mount'] + mount_opts + [path, chroot_path]) state.grub_mounts.append(chroot_path) def chroot_path(self, chroot, path): return os.path.normpath(os.path.join(chroot, '.' + path)) def install_package(self, chroot, package): env = os.environ.copy() env['DEBIAN_FRONTEND'] = 'noninteractive' vmdb.runcmd_chroot( chroot, ['apt-get', '-y', '--no-show-progress', 'install', package], env=env) def set_grub_cmdline_config(self, chroot, kernel_params): param_string = ' '.join(kernel_params) filename = self.chroot_path(chroot, '/etc/default/grub') with open(filename) as f: text = f.read() lines = text.splitlines() lines = [line for line in lines if not line.startswith('GRUB_CMDLINE_LINUX_DEFAULT')] lines.append('GRUB_CMDLINE_LINUX_DEFAULT="{}"'.format(param_string)) with open(filename, 'w') as f: f.write('\n'.join(lines) + '\n') def add_grub_serial_console(self, chroot): filename = self.chroot_path(chroot, '/etc/default/grub') with open(filename, 'a') as f: f.write('GRUB_TERMINAL=serial\n') f.write('GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 ' '--word=8 --parity=no --stop=1"\n') def add_grub_crypto_disk(self, chroot): filename = self.chroot_path(chroot, '/etc/default/grub') with open(filename, 'a') as f: f.write('GRUB_ENABLE_CRYPTODISK=y\n') vmdb2/vmdb/plugins/luks.mdwn0000644000175000017500000000204313577177751015407 0ustar gwolfgwolfStep: `luks` ----------------------------------------------------------------------------- Set up disk encryption using LUKS with the `cryptsetup` utility. The encryption passphrase is read from a file or from the output of a command. The encrypted disk gets opened and can be mounted using a separate tag for the cleartext view. Step keys: * `cryptsetup` — REQUIRED; value is the tag for the encrypted block device. This is not directly useable by users, or mountable. * `tag` — REQUIRED; the tag for the de-crypted block device. This is what gets mounted and visible to users. * `key-file` — OPTIONAL; file from where passphrase is read. * `key-cmd` — OPTIONAL; command to run, passphrase is the first line of its standard output. Example (in the .vmdb file): - cryptsetup: root tag: root_crypt key-file: disk.pass Same, except run a command to get passphrase (in this case [pass](https://www.passwordstore.org/)): - cryptsetup: root tag: root_crypt key-cmd: pass show disk-encryption vmdb2/vmdb/plugins/luks_plugin.py0000644000175000017500000000516713577177751016462 0ustar gwolfgwolf# Copyright 2018 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= import logging import os import tempfile import cliapp import vmdb class LuksPlugin(cliapp.Plugin): def enable(self): self.app.step_runners.add(CryptsetupStepRunner()) class CryptsetupStepRunner(vmdb.StepRunnerInterface): def get_required_keys(self): return ['cryptsetup'] def run(self, step, settings, state): underlying = step['cryptsetup'] crypt_name = step['tag'] state.tmp_key_file = None key_file = step.get('key-file') key_cmd = step.get('key-cmd') if key_file is None and key_cmd is None: raise Exception( 'cryptsetup step MUST define one of key-file or key-cmd') if key_file is None: output = vmdb.runcmd(['sh', '-c', key_cmd]) output = output.decode('UTF-8') key = output.splitlines()[0] fd, key_file = tempfile.mkstemp() state.tmp_key_file = key_file os.close(fd) open(key_file, 'w').write(key) dev = state.tags.get_dev(underlying) if dev is None: for t in state.tags.get_tags(): logging.debug( 'tag %r dev %r mp %r', t, state.tags.get_dev(t), state.tags.get_mount_point(t)) assert 0 vmdb.runcmd(['cryptsetup', 'luksFormat', dev, key_file]) vmdb.runcmd( ['cryptsetup', 'open', '--type', 'luks', '--key-file', key_file, dev, crypt_name]) crypt_dev = '/dev/mapper/{}'.format(crypt_name) assert os.path.exists(crypt_dev) state.tags.append(crypt_name) state.tags.set_dev(crypt_name, crypt_dev) def teardown(self, step, settings, state): x = state.tmp_key_file if x is not None and os.path.exists(x): os.remove(x) crypt_name = step['tag'] crypt_dev = '/dev/mapper/{}'.format(crypt_name) vmdb.runcmd(['cryptsetup', 'close', crypt_dev]) vmdb2/vmdb/plugins/lvm2.mdwn0000644000175000017500000000176013577177751015316 0ustar gwolfgwolfStep: `vgcreate` ----------------------------------------------------------------------------- Create an LVM2 volume group (VG), and also initialise the physical volumes for it. Step keys: * `vgcreate` — REQUIRED; value is the tag for the volume group. This gets initialised with `vgcreate`. * `physical` — REQUIRED; list of tags for block devices (partitions) to use as physical volumes. These get initialised with `pvcreate`. Example (in the .vmdb file): - vgcreate: rootvg physical: - my_partition - other_partition Step: `lvcreate` ----------------------------------------------------------------------------- Create an LVM2 logical volume (LV) in an existing volume group. Step keys: * `lvcreate` — REQUIRED; value is the tag for the volume group. * `name` — REQUIRED; tag for the new LV block device. * `size` — REQUIRED; size of the new LV. Example (in the .vmdb file): - lvcreate: rootvg name: rootfs size: 1G vmdb2/vmdb/plugins/lvm2_plugin.py0000644000175000017500000000412013577177751016350 0ustar gwolfgwolf# Copyright 2018 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= import os import cliapp import vmdb class Lvm2Plugin(cliapp.Plugin): def enable(self): self.app.step_runners.add(VgcreateStepRunner()) self.app.step_runners.add(LvcreateStepRunner()) class VgcreateStepRunner(vmdb.StepRunnerInterface): def get_required_keys(self): return ['vgcreate', 'physical'] def run(self, step, settings, state): vgname = self.get_vg(step) physical = self.get_pv(step, state) for phys in physical: vmdb.runcmd(['pvcreate', '-ff', '--yes', phys]) vmdb.runcmd(['vgcreate', vgname] + physical) def teardown(self, step, settings, state): vgname = self.get_vg(step) vmdb.runcmd(['vgchange', '-an', vgname]) def get_vg(self, step): return step['vgcreate'] def get_pv(self, step, state): return [ state.tags.get_dev(tag) for tag in step['physical'] ] class LvcreateStepRunner(vmdb.StepRunnerInterface): def get_required_keys(self): return ['lvcreate'] def run(self, step, settings, state): vgname = step['lvcreate'] lvname = step['name'] size = step['size'] vmdb.runcmd(['lvcreate', '--name', lvname, '--size', size, vgname]) lvdev = '/dev/{}/{}'.format(vgname, lvname) assert os.path.exists(lvdev) state.tags.append(lvname) state.tags.set_dev(lvname, lvdev) vmdb2/vmdb/plugins/mkfs.mdwn0000644000175000017500000000051413577177751015372 0ustar gwolfgwolfStep: `mkfs` ----------------------------------------------------------------------------- Create a filesystem. Step keys: * `mkfs` — REQUIRED; filesystem type, such as `ext4` or `vfat`. * `partition` — REQUIRED; tag for the block device to use. Example (in the .vmdb file): - mkfs: ext4 partition: root vmdb2/vmdb/plugins/mkfs_plugin.py0000644000175000017500000000265313577177751016441 0ustar gwolfgwolf# Copyright 2017 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= import cliapp import vmdb class MkfsPlugin(cliapp.Plugin): def enable(self): self.app.step_runners.add(MkfsStepRunner()) class MkfsStepRunner(vmdb.StepRunnerInterface): def get_required_keys(self): return ['mkfs', 'partition'] def run(self, step, settings, state): fstype = step['mkfs'] tag = step['partition'] device = state.tags.get_dev(tag) cmd = ['/sbin/mkfs', '-t', fstype] if 'label' in step: if fstype == 'vfat': cmd.append('-n') elif fstype == 'f2fs': cmd.append('-l') else: cmd.append('-L') cmd.append(step['label']) cmd.append(device) vmdb.runcmd(cmd) vmdb2/vmdb/plugins/mkimg.mdwn0000644000175000017500000000050413577177751015535 0ustar gwolfgwolfStep: `mkimg` ----------------------------------------------------------------------------- Create a new image file of a desired size. Step keys: * `mkimage` — REQUIRED; name of file to create. * `size` — REQUIRED; size of the image. Example (in the .vmdb file): - mkimg: "{{ output }}" size: 4G vmdb2/vmdb/plugins/mkimg_plugin.py0000644000175000017500000000234013577177751016576 0ustar gwolfgwolf# Copyright 2017 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= import cliapp import vmdb class MkimgPlugin(cliapp.Plugin): def enable(self): self.app.step_runners.add(MkimgStepRunner()) self.app.settings.bytesize( ['size'], 'size of output image', default='1GiB') class MkimgStepRunner(vmdb.StepRunnerInterface): def get_required_keys(self): return ['mkimg'] def run(self, step, settings, state): filename = step['mkimg'] size = step['size'] vmdb.runcmd(['qemu-img', 'create', '-f', 'raw', filename, size]) vmdb2/vmdb/plugins/mount.mdwn0000644000175000017500000000060113577177751015571 0ustar gwolfgwolfStep: `mount` ----------------------------------------------------------------------------- Mount a filesystem. Step keys: * `mount` — REQUIRED; tag of filesystem to mount. * `dirname` — OPTIONAL; the mount point. * `mount-on` — OPTIONAL; tag of already mounted filesystem in image. (FIXME: this may be wrong?) Example (in the .vmdb file): - mount: root vmdb2/vmdb/plugins/mount_plugin.py0000644000175000017500000000454113577177751016641 0ustar gwolfgwolf# Copyright 2017 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= import logging import os import tempfile import cliapp import vmdb class MountPlugin(cliapp.Plugin): def enable(self): self.app.step_runners.add(MountStepRunner()) class MountStepRunner(vmdb.StepRunnerInterface): def get_required_keys(self): return ['mount'] def run(self, step, settings, state): self.mount_rootfs(step, settings, state) def teardown(self, step, settings, state): self.unmount_rootfs(step, settings, state) def mount_rootfs(self, step, settings, state): tag = step['mount'] dirname = step.get('dirname') mount_on = step.get('mount-on') device = state.tags.get_dev(tag) if dirname: if not mount_on: raise Exception('no mount-on tag given') if not state.tags.has_tag(mount_on): raise Exception('cannot find tag {}'.format(mount_on)) mount_point = os.path.join( state.tags.get_mount_point(mount_on), './' + dirname) if not os.path.exists(mount_point): os.makedirs(mount_point) else: mount_point = tempfile.mkdtemp() vmdb.runcmd(['mount', device, mount_point]) state.tags.set_mount_point(tag, mount_point, cached=True) return mount_point def unmount_rootfs(self, step, settings, state): tag = step['mount'] mount_point = state.tags.get_mount_point(tag) if mount_point is None: return try: vmdb.unmount(mount_point) except vmdb.NotMounted as e: logging.warning(str(e)) if not step.get('mount-on'): os.rmdir(mount_point) vmdb2/vmdb/plugins/partition.mdwn0000644000175000017500000000254413577177751016450 0ustar gwolfgwolfStep: `mklabel` ----------------------------------------------------------------------------- Create a partition table on a block device. Step keys: * `mklabel` — REQUIRED; type of partition table, MUST be one of `msdos` and `gpt`. * `device` — REQUIRED; tag for the block device. Example (in the .vmdb file): - mklabel: msdos device: "{{ output }}" Step: `mkpart` ----------------------------------------------------------------------------- Create a partition. Step keys: * `mkpart` — REQUIRED; type of partition to create: use `primary` (but any value acceped by `parted` is OK). * `device` — REQUIRED; filename of block device where to create partition. * `start` — REQUIRED; where does the partition start? * `end` — REQUIRED; where does the partition end? * `tag` — REQUIRED; tag for the new partition. Example (in the .vmdb file): - mkpart: primary device: "{{ output }}" start: 0% end: 100% tag: root Step: `kpartx` ----------------------------------------------------------------------------- Create loop devices for partitions in an image file. Not needed when installing to a real block device, instead of an image file. Step keys: * `kpartx` — REQUIRED; filename of block device with partitions. Example (in the .vmdb file): - kpartx: "{{ output }}" vmdb2/vmdb/plugins/partition_plugin.py0000644000175000017500000000717413577177751017515 0ustar gwolfgwolf# Copyright 2017 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= import os import stat import cliapp import vmdb class PartitionPlugin(cliapp.Plugin): def enable(self): self.app.step_runners.add(MklabelStepRunner()) self.app.step_runners.add(MkpartStepRunner()) self.app.step_runners.add(KpartxStepRunner()) class MklabelStepRunner(vmdb.StepRunnerInterface): def get_required_keys(self): return ['mklabel', 'device'] def run(self, step, settings, state): label_type = step['mklabel'] device = step['device'] vmdb.runcmd(['parted', '-s', device, 'mklabel', label_type]) class MkpartStepRunner(vmdb.StepRunnerInterface): def get_required_keys(self): return ['mkpart', 'device', 'start', 'end'] def run(self, step, settings, state): part_type = step['mkpart'] device = step['device'] start = step['start'] end = step['end'] tag = step.get('tag') or step.get('part-tag') if tag is None: tag = step['part-tag'] fs_type = step.get('fs-type', 'ext2') orig = self.list_partitions(device) vmdb.runcmd(['parted', '-s', device, 'mkpart', part_type, fs_type, start, end]) state.tags.append(tag) if self.is_block_dev(device): new = self.list_partitions(device) diff = self.diff_partitions(orig, new) assert len(diff) == 1 vmdb.progress('remembering partition {} as {}'.format(diff[0], tag)) state.tags.set_dev(tag, diff[0]) def is_block_dev(self, filename): st = os.lstat(filename) return stat.S_ISBLK(st.st_mode) def list_partitions(self, device): output = vmdb.runcmd(['parted', '-m', device, 'print']) output = output.decode('UTF-8') partitions = [ line.split(':')[0] for line in output.splitlines() if ':' in line ] return [ word if word.startswith('/') else '{}{}'.format(device, word) for word in partitions ] def diff_partitions(self, old, new): return [ line for line in new if line not in old ] class KpartxStepRunner(vmdb.StepRunnerInterface): def get_required_keys(self): return ['kpartx'] def run(self, step, settings, state): device = step['kpartx'] tags = state.tags.get_tags() devs = self.kpartx(device) for tag, dev in zip(tags, devs): vmdb.progress('remembering {} as {}'.format(dev, tag)) state.tags.set_dev(tag, dev) def kpartx(self, device): output = vmdb.runcmd(['kpartx', '-asv', device]).decode('UTF-8') for line in output.splitlines(): words = line.split() if words[0] == 'add': name = words[2] yield '/dev/mapper/{}'.format(name) def teardown(self, step, settings, state): device = step['kpartx'] vmdb.runcmd(['kpartx', '-dsv', device]) vmdb2/vmdb/plugins/qemudebootstrap.mdwn0000644000175000017500000000154613577177751017656 0ustar gwolfgwolfStep: `qemu-debootstrap` ----------------------------------------------------------------------------- Install packages using apt, which needs to already have been installed, for a different architecture than the host where vmdb2 is being run. For example, for building an image for a Raspberry Pi on an Intel PC. Step keys: * `qemu-debootstrap` — REQUIRED; value is the codename of the Debian release to install: `stretch`, `buster`, etc. * `target` — REQUIRED; value is the tag for the root filesystem. * `mirror` — OPTIONAL; which Debian mirror to use. * `arch` — REQUIRED; the foreign architecture touse. * `variant` — OPTIONAL; the variant for debootstrap. Example (in the .vmdb file): - qemu-debootstrap: stretch target: root mirror: http://mirror.example.com/debian arch: arm64 variant: buildd vmdb2/vmdb/plugins/qemudebootstrap_plugin.py0000644000175000017500000000365113577177751020716 0ustar gwolfgwolf# Copyright 2017 Lars Wirzenius and Stuart Prescott # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= import cliapp import vmdb class QemuDebootstrapPlugin(cliapp.Plugin): def enable(self): self.app.step_runners.add(QemuDebootstrapStepRunner()) class QemuDebootstrapStepRunner(vmdb.StepRunnerInterface): def get_required_keys(self): return ['qemu-debootstrap', 'target', 'mirror', 'arch'] def run(self, step, settings, state): suite = step['qemu-debootstrap'] tag = step['target'] target = state.tags.get_mount_point(tag) mirror = step['mirror'] variant = step.get('variant', '-') arch = step['arch'] components = step.get('components', ['main']) if not (suite and tag and target and mirror and arch): raise Exception('missing arg for qemu-debootstrap step') vmdb.runcmd( ['qemu-debootstrap', '--arch', arch, '--variant', variant, '--components', ','.join(components), suite, target, mirror]) vmdb.runcmd_chroot(target, ['apt-get', 'update']) def run_even_if_skipped(self, step, settings, state): tag = step['target'] target = state.tags.get_mount_point(tag) vmdb.runcmd_chroot(target, ['apt-get', 'update']) vmdb2/vmdb/plugins/rootfs_cache.mdwn0000644000175000017500000000136313577177751017074 0ustar gwolfgwolfStep: `cache-rootfs` ----------------------------------------------------------------------------- Create a tarball of the root filesystem in the image. Step keys: * `cache-rootfs` — REQUIRED; tag of root filesystem on image. Example (in the .vmdb file): - cache-rootfs: root unless: rootfs_unpacked Step: `unpack-rootfs` ----------------------------------------------------------------------------- Unpack a tarball of the root filesystem to the image, and set the `rootfs_unpacked` condition to true. If the tarball doesn't exist, do nothing and leave the `rootfs_unpacked` condition to false. Step keys: * `unpack-rootfs` — REQUIRED; tag for the root filesystem. Example (in the .vmdb file): - unpack-rootfs: root vmdb2/vmdb/plugins/rootfs_cache_plugin.py0000644000175000017500000000634313577177751020140 0ustar gwolfgwolf# Copyright 2017 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= import os import cliapp import vmdb class RootFSCachePlugin(cliapp.Plugin): def enable(self): self.app.settings.string( ['rootfs-tarball'], 'store rootfs cache tar archives in FILE', metavar='FILE') self.app.step_runners.add(MakeCacheStepRunner()) self.app.step_runners.add(UnpackCacheStepRunner()) class MakeCacheStepRunner(vmdb.StepRunnerInterface): def get_required_keys(self): return ['cache-rootfs'] def run(self, step, settings, state): fs_tag = step['cache-rootfs'] rootdir = state.tags.get_mount_point(fs_tag) tar_path = settings['rootfs-tarball'] opts = step.get('options', '--one-file-system').split() if not tar_path: raise Exception('--rootfs-tarball MUST be set') dirs = self._find_cacheable_mount_points(state.tags, rootdir) tags = state.tags for tag in tags.get_tags(): vmdb.progress( 'tag {} mounted {} cached {}'.format( tag, tags.get_mount_point(tag), tags.is_cached(tag))) vmdb.progress('caching rootdir {}'.format(rootdir)) vmdb.progress('caching relative {}'.format(dirs)) if not os.path.exists(tar_path): vmdb.runcmd( ['tar'] + opts + ['-C', rootdir, '-caf', tar_path] + dirs) def _find_cacheable_mount_points(self, tags, rootdir): return list(sorted( self._make_relative(rootdir, tags.get_mount_point(tag)) for tag in tags.get_tags() if tags.is_cached(tag) )) def _make_relative(self, rootdir, dirname): assert dirname == rootdir or dirname.startswith(rootdir + '/') if dirname == rootdir: return '.' return dirname[len(rootdir) + 1:] class UnpackCacheStepRunner(vmdb.StepRunnerInterface): def get_required_keys(self): return ['unpack-rootfs'] def run(self, step, settings, state): fs_tag = step['unpack-rootfs'] rootdir = state.tags.get_mount_point(fs_tag) tar_path = settings['rootfs-tarball'] if not tar_path: raise Exception('--rootfs-tarball MUST be set') if os.path.exists(tar_path): vmdb.runcmd( ['tar', '-C', rootdir, '-xf', tar_path, '--numeric-owner']) self.copy_resolv_conf(rootdir) state.rootfs_unpacked = True def copy_resolv_conf(self, rootdir): filename = os.path.join(rootdir, 'etc', 'resolv.conf') vmdb.runcmd(['cp', '/etc/resolv.conf', filename]) vmdb2/vmdb/plugins/virtuals_plugin.py0000644000175000017500000000471613577177751017354 0ustar gwolfgwolf# Copyright 2017 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= import logging import os import cliapp import vmdb class VirtualFilesystemMountPlugin(cliapp.Plugin): def enable(self): self.app.step_runners.add(VirtualFilesystemMountStepRunner()) class VirtualFilesystemMountStepRunner(vmdb.StepRunnerInterface): virtuals = [ ['none', '/proc', 'proc'], ['none', '/dev', 'devtmpfs'], ['none', '/dev/pts', 'devpts'], ['none', '/dev/shm', 'tmpfs'], ['none', '/run', 'tmpfs'], ['none', '/run/lock', 'tmpfs'], ['none', '/sys', 'sysfs'], ] def get_required_keys(self): return ['mount-virtual-filesystems'] def run(self, step, settings, state): fstag = step['mount-virtual-filesystems'] mount_point = state.tags.get_mount_point(fstag) self.mount_virtuals(mount_point, state) def teardown(self, step, settings, state): self.unmount_virtuals(state) def mount_virtuals(self, rootfs, state): if not hasattr(state, 'virtuals'): state.virtuals = [] for device, mount_point, fstype in self.virtuals: path = os.path.join(rootfs, './' + mount_point) if not os.path.exists(path): os.mkdir(path) vmdb.runcmd(['mount', '-t', fstype, device, path]) state.virtuals.append(path) logging.debug('mounted virtuals: %r', state.virtuals) def unmount_virtuals(self, state): logging.debug('unmounting virtuals: %r', state.virtuals) for mount_point in reversed(state.virtuals): try: vmdb.unmount(mount_point) except vmdb.NotMounted as e: logging.warning(str(e)) except cliapp.AppException: vmdb.warning('Something went wrong while unmounting. Ignoring.') vmdb2/vmdb/runcmd.py0000644000175000017500000000322413577177751013725 0ustar gwolfgwolf# Copyright 2017 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= import logging import os import sys import cliapp _verbose = False def set_verbose_progress(verbose): global _verbose _verbose = verbose def error(msg): logging.error(msg, exc_info=True) if _verbose: sys.stderr.write('ERROR: {}\n'.format(msg)) def progress(msg): logging.info(msg) if _verbose: sys.stdout.write('{}\n'.format(msg)) def runcmd(argv, *argvs, **kwargs): progress('Exec: %r' % (argv,)) kwargs['stdout_callback'] = _log_stdout kwargs['stderr_callback'] = _log_stderr env = kwargs.get('env', os.environ.copy()) env['LC_ALL'] = 'C' kwargs['env'] = env return cliapp.runcmd(argv, *argvs, **kwargs) def runcmd_chroot(chroot, argv, *argvs, **kwargs): full_argv = ['chroot', chroot] + argv return runcmd(full_argv, *argvs, **kwargs) def _log_stdout(data): logging.debug('STDOUT: %r', data) return data def _log_stderr(data): logging.debug('STDERR: %r', data) return data vmdb2/vmdb/spec.py0000644000175000017500000000275213577177751013374 0ustar gwolfgwolf# Copyright 2017 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= import jinja2 import yaml class Spec: def __init__(self): self._dict = None def load_file(self, filename): with open(filename) as f: self._dict = yaml.safe_load(f) def as_dict(self): return dict(self._dict) def get_steps(self, params): return expand_templates(self._dict['steps'], params) def expand_templates(value, params): if isinstance(value, str): template = jinja2.Template(value) return template.render(**params) elif isinstance(value, list): return [expand_templates(x, params) for x in value] elif isinstance(value, dict): return { key: expand_templates(value[key], params) for key in value } else: assert 0, 'Unknown value type: {!r}'.format(value) vmdb2/vmdb/spec_tests.py0000644000175000017500000000552213577177751014614 0ustar gwolfgwolf# Copyright 2017 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= import io import os import tempfile import unittest import yaml import vmdb class SpecTests(unittest.TestCase): spec_yaml = ''' steps: - step: foo arg: "{{ var1 }}" - step: bar ''' def setUp(self): self.filename = write_temp_file(bytes(self.spec_yaml, 'ascii')) self.spec = vmdb.Spec() def tearDown(self): if os.path.exists(self.filename): os.remove(self.filename) def test_loads_spec(self): self.spec.load_file(self.filename) self.assertEqual(self.spec.as_dict(), as_dict(self.spec_yaml)) def test_expands_templates(self): self.spec.load_file(self.filename) params = { 'var1': 'value1', } steps = self.spec.get_steps(params) self.assertEqual( steps, [ { 'step': 'foo', 'arg': 'value1', }, { 'step': 'bar', }, ] ) class ExpandTemplatesTests(unittest.TestCase): def test_raises_assert_if_given_incomprehensible_value(self): with self.assertRaises(AssertionError): vmdb.expand_templates(None, {}) def test_returns_same_given_string_without_template(self): self.assertEqual(vmdb.expand_templates('foo', {}), 'foo') def test_expands_simple_string_template(self): params = { 'foo': 'bar', } self.assertEqual(vmdb.expand_templates('{{ foo }}', params), 'bar') def test_expands_list_of_templates(self): params = { 'foo': 'bar', } self.assertEqual(vmdb.expand_templates(['{{ foo }}'], params), ['bar']) def test_expands_dict_of_templates(self): params = { 'foo': 'bar', } self.assertEqual( vmdb.expand_templates({'key': '{{ foo }}'}, params), {'key': 'bar'} ) def write_temp_file(data): fd, filename = tempfile.mkstemp() os.write(fd, data) os.close(fd) return filename def as_dict(yaml_text): with io.StringIO(yaml_text) as f: return yaml.safe_load(f) vmdb2/vmdb/state.py0000644000175000017500000000174013577177751013556 0ustar gwolfgwolf# Copyright 2017 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= class State: def __init__(self): self._attrs = {} # make sure this attribute exists self._attrs = self.as_dict() def as_dict(self): return { key: getattr(self, key) for key in dir(self) if not key in self._attrs } vmdb2/vmdb/step_list.py0000644000175000017500000000503113577177751014441 0ustar gwolfgwolf# Copyright 2017 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= import cliapp class StepRunnerInterface: # pragma: no cover def get_required_keys(self): raise NotImplementedError() def run(self, step_spec, settings, state): raise NotImplementedError() def run_even_if_skipped(self, step_spec, settings, state): pass def teardown(self, step_spec, settings, state): # Default implementation does nop, so that sub-classes don't # need to have a nop teardown. pass def skip(self, step_spec, settings, state): # Return true if step should be skipped and not run. Does not # apply to teardowns. # Skipping is indicated by the step having a field 'unless', # which is either the name of a variable (field in state), or # a list of such names. If all variables have a value that # evaluates as truth, the step is skipped. value = step_spec.get('unless', None) if value is None: return False if isinstance(value, list): return all(getattr(state, field, False) for field in value) return getattr(state, value, False) class StepRunnerList: def __init__(self): self._runners = [] def __len__(self): return len(self._runners) def add(self, runner): self._runners.append(runner) def find(self, step_spec): actual = set(step_spec.keys()) for runner in self._runners: required = set(runner.get_required_keys()) if actual.intersection(required) == required: return runner raise NoMatchingRunner(actual) class StepError(cliapp.AppException): pass class NoMatchingRunner(cliapp.AppException): def __init__(self, keys): super(NoMatchingRunner, self).__init__( 'No runner implements step with keys {}'.format( ', '.join(keys))) vmdb2/vmdb/step_list_tests.py0000644000175000017500000000325113577177751015665 0ustar gwolfgwolf# Copyright 2017 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= import unittest import vmdb class StepRunnerListTests(unittest.TestCase): def test_is_empty_initially(self): steps = vmdb.StepRunnerList() self.assertEqual(len(steps), 0) def test_adds_a_runner(self): steps = vmdb.StepRunnerList() runner = DummyStepRunner() steps.add(runner) self.assertEqual(len(steps), 1) def test_finds_correct_runner(self): steps = vmdb.StepRunnerList() runner = DummyStepRunner() steps.add(runner) found = steps.find({'foo': None, 'bar': None}) self.assertEqual(runner, found) def test_raises_error_if_runner_not_found(self): steps = vmdb.StepRunnerList() runner = DummyStepRunner() steps.add(runner) with self.assertRaises(vmdb.NoMatchingRunner): steps.find({'foo': None}) class DummyStepRunner(vmdb.StepRunnerInterface): def run(self, *args): pass def get_required_keys(self): return ['foo', 'bar'] vmdb2/vmdb/tags.py0000644000175000017500000000520313577177751013372 0ustar gwolfgwolf# Copyright 2018 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= # Unmount a directory, including any mount points under that # directory. If /mnt/foo is given, and /mnt/foo/bar is also mounted, # unmount /mnt/foo/bar first, and /mnt/foo then. Look for sub-mounts # in /proc/mounts. class Tags: def __init__(self): self._tags = {} self._tagnames = [] def get_tags(self): return list(self._tags.keys()) def has_tag(self, tag): return tag in self._tags def get_dev(self, tag): item = self._get(tag) return item['dev'] def get_mount_point(self, tag): item = self._get(tag) return item['mount_point'] def is_cached(self, tag): item = self._get(tag) return item.get('cached', False) def append(self, tag): if tag in self._tags: raise TagInUse(tag) self._tagnames.append(tag) self._tags[tag] = { 'dev': None, 'mount_point': None, } def set_dev(self, tag, dev): item = self._get(tag) if item['dev'] is not None: raise AlreadyHasDev(tag) item['dev'] = dev def set_mount_point(self, tag, mount_point, cached=False): item = self._get(tag) if item['mount_point'] is not None: raise AlreadyMounted(tag) item['mount_point'] = mount_point item['cached'] = cached def _get(self, tag): item = self._tags.get(tag) if item is None: raise UnknownTag(tag) return item class TagInUse(Exception): def __init__(self, tag): super().__init__('Tag already used: {}'.format(tag)) class UnknownTag(Exception): def __init__(self, tag): super().__init__('Unknown tag: {}'.format(tag)) class AlreadyHasDev(Exception): def __init__(self, tag): super().__init__('Already has device: {}'.format(tag)) class AlreadyMounted(Exception): def __init__(self, tag): super().__init__('Already mounted tag: {}'.format(tag)) vmdb2/vmdb/tags_tests.py0000644000175000017500000000734313577177751014623 0ustar gwolfgwolf# Copyright 2018 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= import unittest import vmdb class TagsTests(unittest.TestCase): def test_lists_not_tags_initally(self): tags = vmdb.Tags() self.assertEqual(tags.get_tags(), []) def test_tells_if_tag_exists(self): tags = vmdb.Tags() self.assertFalse(tags.has_tag('foo')) tags.append('foo') self.assertTrue(tags.has_tag('foo')) self.assertEqual(tags.get_tags(), ['foo']) def test_remembers_order(self): tags = vmdb.Tags() tags.append('foo') tags.append('bar') self.assertTrue(tags.get_tags(), ['foo', 'bar']) def test_get_dev_raises_error_for_unknown_tag(self): tags = vmdb.Tags() with self.assertRaises(vmdb.UnknownTag): tags.get_dev('does-not-exist') def test_get_mount_point_raises_error_for_unknown_tag(self): tags = vmdb.Tags() with self.assertRaises(vmdb.UnknownTag): tags.get_mount_point('does-not-exist') def test_raises_error_for_reused_tag(self): tags = vmdb.Tags() tags.append('tag') with self.assertRaises(vmdb.TagInUse): tags.append('tag') def test_sets_dev(self): tags = vmdb.Tags() tags.append('first') tags.set_dev('first', '/dev/foo') self.assertEqual(tags.get_tags(), ['first']) self.assertEqual(tags.get_dev('first'), '/dev/foo') self.assertEqual(tags.get_mount_point('first'), None) def test_adds_mount_point(self): tags = vmdb.Tags() tags.append('first') tags.set_mount_point('first', '/mnt/foo') self.assertEqual(tags.get_tags(), ['first']) self.assertEqual(tags.get_dev('first'), None) self.assertEqual(tags.get_mount_point('first'), '/mnt/foo') def test_mount_point_is_uncached_by_default(self): tags = vmdb.Tags() tags.append('first') tags.set_mount_point('first', '/mnt/foo') self.assertFalse(tags.is_cached('first')) def test_mount_point_can_be_made_cached(self): tags = vmdb.Tags() tags.append('first') tags.set_mount_point('first', '/mnt/foo', cached=True) self.assertTrue(tags.is_cached('first')) def test_set_dev_raises_error_for_unknown_tag(self): tags = vmdb.Tags() with self.assertRaises(vmdb.UnknownTag): tags.set_dev('first', '/mnt/foo') def test_set_mount_point_raises_error_for_unknown_tag(self): tags = vmdb.Tags() with self.assertRaises(vmdb.UnknownTag): tags.set_mount_point('first', '/mnt/foo') def test_set_mount_point_raises_error_for_double_mount(self): tags = vmdb.Tags() tags.append('first') tags.set_mount_point('first', '/mnt/foo') with self.assertRaises(vmdb.AlreadyMounted): tags.set_mount_point('first', '/mnt/foo') def test_set_dev_raises_error_for_double_dev(self): tags = vmdb.Tags() tags.append('first') tags.set_dev('first', '/dev/foo') with self.assertRaises(vmdb.AlreadyHasDev): tags.set_dev('first', '/dev/foo') vmdb2/vmdb/unmount.py0000644000175000017500000000420213577177751014137 0ustar gwolfgwolf# Copyright 2018 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= # Unmount a directory, including any mount points under that # directory. If /mnt/foo is given, and /mnt/foo/bar is also mounted, # unmount /mnt/foo/bar first, and /mnt/foo then. Look for sub-mounts # in /proc/mounts. import vmdb def unmount(what, mounts=None, real_unmount=None): if mounts is None: # pragma: no cover mounts = _read_proc_mounts() if real_unmount is None: # pragma: no cover real_unmount = _real_unmount mounts = _parse_proc_mounts(mounts) dirnames = _find_what_to_unmount(mounts, what) for dirname in dirnames: real_unmount(dirname) def _read_proc_mounts(): # pragma: no cover with open('/proc/mounts') as f: return f.read() def _real_unmount(what): # pragma: no cover vmdb.runcmd(['umount', what]) def _parse_proc_mounts(text): return [ line.split()[:2] for line in text.splitlines() ] def _find_what_to_unmount(mounts, what): dirname = _find_mount_point(mounts, what) dirnameslash = dirname + '/' to_unmount = [ point for dev, point in mounts if point == dirname or point.startswith(dirnameslash) ] return list(reversed(sorted(to_unmount))) def _find_mount_point(mounts, what): for dev, point in mounts: if what in (dev, point): return point raise NotMounted(what) class NotMounted(Exception): def __init__(self, what): super().__init__('Not mounted: {}'.format(what)) vmdb2/vmdb/unmount_tests.py0000644000175000017500000000426413577177751015371 0ustar gwolfgwolf# Copyright 2018 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= import unittest import vmdb class UnmountTests(unittest.TestCase): def setUp(self): self.mounts = ProcMounts() def unmount(self, what): vmdb.unmount( what, mounts=str(self.mounts), real_unmount=self.mounts.unmount) def test_raises_error_if_not_mounted(self): with self.assertRaises(vmdb.NotMounted): self.unmount('/foo') def test_unmounts_mounted_dir(self): self.mounts.mount('/dev/foo', '/foo') self.unmount('/foo') self.assertFalse(self.mounts.is_mounted('/foo')) def test_unmounts_mounted_dir_with_submounts(self): self.mounts.mount('/dev/foo', '/foo') self.mounts.mount('/dev/bar', '/foo/bar') self.unmount('/foo') self.assertFalse(self.mounts.is_mounted('/foo')) self.assertFalse(self.mounts.is_mounted('/foo/bar')) class ProcMounts: def __init__(self): self.mounts = [] def is_mounted(self, what): return any(what in mount for mount in self.mounts) def mount(self, device, point): self.mounts.append((device, point)) def unmount(self, what): self.mounts = [ mount for mount in self.mounts if what not in mount ] def __str__(self): return ''.join( '{}\n'.format(self.mount_line(mount)) for mount in self.mounts ) def mount_line(self, mount): return '{} {} fstype options 0 0'.format(mount[0], mount[1]) vmdb2/vmdb/version.py0000644000175000017500000000010113577177751014111 0ustar gwolfgwolf__version__ = "0.13.2+git" __version_info__ = (0, 13, 2, '+git') vmdb2/vmdb20000755000175000017500000000011413577177751012066 0ustar gwolfgwolf#!/usr/bin/python3 import vmdb vmdb.Vmdb2(version=vmdb.__version__).run() vmdb2/vmdb2.1.in0000644000175000017500000000255213577177751012637 0ustar gwolfgwolf.\" Copyright 2017 Lars Wirzenius .\" .\" This program is free software: you can redistribute it and/or modify .\" it under the terms of the GNU General Public License as published by .\" the Free Software Foundation, either version 3 of the License, or .\" (at your option) any later version. .\" .\" This program is distributed in the hope that it will be useful, .\" but WITHOUT ANY WARRANTY; without even the implied warranty of .\" MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the .\" GNU General Public License for more details. .\" .\" You should have received a copy of the GNU General Public License .\" along with this program. If not, see . .\" .TH VMDB2 1 .SH NAME vmdb2 \- create a disk image with Debian installed .SH SYNOPSIS .B vmdb .RI [ options ] .IR SPECFILE ... .SH DESCRIPTION .B vmdb2 reads a .I "specification file" that describes a disk image, and builds the corresponding image. The specification file uses YAML syntax, and contains a list of steps. Each step tells vmdb2 what to do next. .SH OPTIONS .SH BUGS This is unfinished at this time. .SH EXAMPLE To build a disk image: .PP .nf .RS vmdb2 foo.vmdb .RE .fi .SH "SEE ALSO" .BR vmdebootstrap (1). .PP See the .B vmdb2 manual and test suite for a full explanation of specification files. This may be installed as .IR /usr/share/doc/vmdb2/manual.html .vmdb2/vmdb2.css0000644000175000017500000000077013577177751012662 0ustar gwolfgwolfhtml { font-family: "FreeSerif", serif; } h1, h2, h3, h4, h5, h6 { font-family: "FreeSans", sans; } header h1.title { display: block; text-align: center; font-family: "FreeSans", sans; font-size: 200%; } header h2.author { display: block; text-align: center; font-family: "FreeSans", sans; font-size: 150%; } header h3.date { display: block; text-align: center; font-family: "FreeSans", sans; font-size: 150%; } pre { margin-left: 4em; } vmdb2/vmdb2.mdwn0000644000175000017500000001343313577177751013037 0ustar gwolfgwolf--- title: Building Debian system images with vmdb2 author: Lars Wirzenius date: work-in-progress ... Introduction ============================================================================= vmdb2 builds disk images with Debian installed. The images can be used for virtual machines, or can be written to USB flash memory devices, and hardware computers can be booted off them. It is a successor of the vmdebootstrap program, written by the same author, to fix a number of architectural problems with the old program. The new program is not compatible with the old one; that would've required keeping the problems, as well. This manual is published as HTML at and as a PDF at . Why vmdb2 given vmdebootstrap already existed ----------------------------------------------------------------------------- `vmdebootstrap` was the first attempt by Lars Wirzenius to write a tool to build system images. It turned out to not be well designed. Specifically, it was not easily extensible to be as flexible as a tool of this sort should be. Why vmdb2 given other tools already exist ----------------------------------------------------------------------------- Lars likes to write tools for himself and had some free time. He sometimes prefers to write his own tools rather than spend time and energy evaluating and improving existing tools. He admits this is a character flaw. Also, he felt ashamed of how messy `vmdebootstrap` turned out to be. If nobody else likes `vmdb2`, that just means Lars had some fun on his own. Installation ============================================================================= You can get vmdb2 by getting the source code from git: git clone git://git.liw.fi/vmdb2 You can then run it from the source tree: sudo /path/to/vmdb2/vmdb2 ... In Debian 10 ("buster") and its derivatives, you can also install the vmdb2 package: apt install vmdb2 For any other systems, we have no instructions. If you figure it out, please tell us how. Getting started ============================================================================= vmdb2 works by reading specification file with instructions for how an image should be built, using YAML syntax, and following those instructions. A minimal specification file example: steps: - mkimg: "{{ output }}" size: 4G - mklabel: gpt device: "{{ output }}" - mkpart: primary device: "{{ output }}" start: 0% end: 100% tag: root - kpartx: "{{ output }}" - mkfs: ext4 partition: root - mount: root - debootstrap: stretch mirror: http://deb.debian.org/debian target: root - apt: install packages: - linux-image-amd64 tag: root - grub: bios tag: root console: serial The above creates a four gigabyte file, creates a GPT partition table, a single partition, with an ext4 filesystem, and installs Debian release stretch onto it. It also installs a kernel, and a boot loader. To use this, save the specification into `test.vmdb`, and run the following command: sudo vmdb2 test.vmdb --output test.img --verbose This will take a long time, mostly at the `debootstrap` step. Tags ----------------------------------------------------------------------------- Instead of device filenames, vmdb2 steps refer to block devices inside the image, and their mount points, by symbolic names called tags. Tags are any names that the user likes, and vmdb2 does not assign meaning to them. They're just strings. Jinja2 expansion ----------------------------------------------------------------------------- To refer to the filename specified with the `--output` or `--image` command line options, you can use [Jinja2](http://jinja.pocoo.org/) templating. The variables `output` and `image` can be used. - mkimg: "{{ output }}" - mklabel: "{{ image }}" The difference is that `--output` creates a new file, or truncates an existing file, whereas `--images` requires the file to already exist. The former is better for image file, the latter for real block devices. Speed up image creation by caching the root filesystem ----------------------------------------------------------------------------- Building an image can take several minutes, and that's with fast access to a Debian mirror and an SSD. The slowest part is typically running debootstrap, and that always results in the same output, for a given Debian release. This means its easy to cache. vmdb2 has the two actions `cache-roots` and `unpack-rootfs` and the command line option `--rootfs-tarball` to allow user to cache. The user uses the option to name a file. `cache-rootfs` takes the root filesystem and stores it into the file as a compress tar archive ("tarball"). `unpack-rootfs` unpacks the tarball. This allows vmdb2 to skip running debootstrap needlessly. The specify which steps should be skipped, the `unless` field can be used: `unpack-rootfs` sets the `rootfs_unpacked` flag if it actually unpacks a tarball, and `unless` allows checking for that flag. If the tarball doesn't exist, the flag is not set. - unpack-rootfs: root - debootstrap: stretch target: root unless: rootfs_unpacked - cache-rootfs: root unless: rootfs_unpacked If the tarball exists, it is unpacked, and the `debootstrap` and `cache-rootfs` steps are skipped. It's possible to have any number of steps between the unpack and the cache steps. However, note that if you change anything within those steps, or time passes and you want to include the new packages that have made it into Debian, you need to delete the tarball so it is run again. TODO: unless, caching, tags, jinja2 Step reference manual ============================================================================= vmdb2/without-tests0000644000175000017500000000123313577177751013717 0ustar gwolfgwolfyarns/lib.py vmdb/__init__.py vmdb/app.py vmdb/runcmd.py vmdb/state.py vmdb/version.py vmdb/plugins/__init__.py vmdb/plugins/ansible_plugin.py vmdb/plugins/apt_plugin.py vmdb/plugins/chroot_plugin.py vmdb/plugins/debootstrap_plugin.py vmdb/plugins/echo_plugin.py vmdb/plugins/error_plugin.py vmdb/plugins/grub_plugin.py vmdb/plugins/kernel_plugin.py vmdb/plugins/lvm2_plugin.py vmdb/plugins/luks_plugin.py vmdb/plugins/mkfs_plugin.py vmdb/plugins/mkimg_plugin.py vmdb/plugins/mount_plugin.py vmdb/plugins/qemudebootstrap_plugin.py vmdb/plugins/partition_plugin.py vmdb/plugins/rootfs_cache_plugin.py vmdb/plugins/virtuals_plugin.py vmdb/plugins/create_file_plugin.py vmdb2/yarns/0000755000175000017500000000000013577177751012266 5ustar gwolfgwolfvmdb2/yarns/100-mvp.yarn0000644000175000017500000001415713577177751014271 0ustar gwolfgwolf--- title: vmdb2 MVP with echo and error author: Lars Wirzenius date: work in progress ... Introduction ============================================================================= vmdb2 is a program for producing a disk image with Debian installed. This document is a manual of sorts, and an automated test suite for it. [vmdebootstrap][] installs Debian onto a disk, or disk image. It is like the [debootstrap][] tool, except the end result is a disk or disk image, not a directory tree. vmdebootstrap takes care of creating partitions, and filesystems, and allows some more customization than vmdebootstrap does. vmdebootstrap is also a messy pile of kludge, and not flexible enough. vmdb2 is a re-implementation from scratch, without a need for backwards compatibility. It aims to provide more flexibility than vmdeboostrap, without becoming anywhere near as complicated. Think of vmdb2 as "vmdebootstrap the second generation". The name has changed to allow the two tools to installable in paralllel. The main user-visible difference between vmdebootstrap and vmdb2 is that the older program provides extensibitlity via a legion of command line options and the newer program by having the user user a domain specific language to express what kind of Debian system they want to create. (Lars Wirzenius wrote both vmdebootstrap and vmdb2 and is entitled to sneer at his younger self.) [vmdebootstrap]: http://liw.fi/vmdebootstrap/ [debootstrap]: https://packages.debian.org/unstable/debootstrap Contact ----------------------------------------------------------------------------- To make contact with the vmdb2 project you can email Lars directly (`liw@liw.fi?`) or use the `#vmdb2` IRC channel on the irc.oftc.net network. Specification files ============================================================================= A vmdb2 specification file is a YAML file that looks like this (this is an imaginary example that doens't actually work right now): EXAMPLE steps: - mkimg: raw size: 4G - mkfs: ext4 label: vmdb2rootfs - debootstrap: jessie - shell: | rm -rf /usr/share/man - adduser: jimbo gecos: James Bond shell: /bin/zsh - sudo: jimbo - ifupdown: eth0 auto: yes dhcp: yes - apt: openssh-server - convert: qcow2 - compress: xz The list of steps produces the kind of image that the user wants (or else an unholy mess). The specification file can easily be shared, and put under version control. Every action in a step is provided by a plugin to vmdb2. Each action (technically, "step runner") is a well-defined task, which may be parameterised by some of the key/value pairs in the step. For example, `mkimg` would create a disk image file. In the above example it is a raw disk image file, as opposed to some other format. The image is 4 gigabytes in size. `mkfs` creates an ext4 filesystem in the image file; in thie example there are no partitions. And so on. Steps may need to clean up after themselves. For example, a step that mounts a filesystem will need to unmount it at the end of the image creation. Also, if a later step fails, then the unmount needs to happen as well. This is called a "teardown". Some steps are provided by a plugin that handles the teardown automatically, others may need to provide instructions for the teardown in the specification file. By providing well-defined steps that the user may combine as they wish, vmdb2 gives great flexibility without much complexity, but at the cost of forcing the user to write a longer specification file than a single vmdeboostrap command line. A happy path ============================================================================= The first case we look at is one for the happy path: a specification with two echo steps, and nothing else. It's very simple, and nothing goes wrong when executing it. In addition to the actual thing to do, each step may also define a "teardown" thing to do. For example, if the step mounts a filesystem, the teardown would unmount it. SCENARIO happy path GIVEN a specification file called happy.vmdb containing ... { ... steps: [ ... { echo: "foo", teardown: "foo_teardown" }, ... { echo: "bar", teardown: "bar_teardown" } ... ] ... } WHEN user runs vmdb2 -v happy.vmdb THEN exit code is 0 AND stdout contains "foo" followed by "bar" AND stdout contains "bar" followed by "bar_teardown" AND stdout contains "bar_teardown" followed by "foo_teardown" Jinja2 templating in specification file values ============================================================================= Vmdb2 allows values in specification files to be processed by the Jinja2 templating engine. This allows users to do thing such as write specifications that use configuration values to determine what happens. For our simple echo/error steps, we will write a rule that outputs the image file name given by the user. A more realistic specification file would instead do thing like create the file. SCENARIO jinja2 templating GIVEN a specification file called j2.vmdb containing ... { ... steps: [ ... { echo: "image is {{ output }}" }, ... { echo: "bar" }, ... ] ... } WHEN user runs vmdb2 -v --output=foo.img j2.vmdb THEN exit code is 0 AND stdout contains "image is foo.img" followed by "bar" Error handling ============================================================================= Sometimes things do not quite go as they should. What does vmdb2 do then? SCENARIO error handling GIVEN a specification file called unhappy.vmdb containing ... { ... steps: [ ... { echo: "foo", teardown: "foo_teardown" }, ... { error: "yikes", teardown: "WAT?!" }, ... { echo: "bar_step", teardown: "bar_teardown" } ... ] ... } WHEN user runs vmdb2 -v unhappy.vmdb THEN exit code is 1 AND stdout contains "foo" followed by "yikes" AND stdout contains "yikes" followed by "WAT?!" AND stdout contains "WAT?!" followed by "foo_teardown" AND stdout does NOT contain "bar_step" AND stdout does NOT contain "bar_teardown" vmdb2/yarns/900-implements.yarn0000644000175000017500000000247113577177751015650 0ustar gwolfgwolfIMPLEMENTS for all scenario steps ============================================================================= This chapter contains the implementations for all scenario steps. IMPLEMENTS GIVEN a specification file called (\S+) containing (.+) filename = get_next_match() spec = get_next_match() open(filename, 'w').write(spec) IMPLEMENTS WHEN user runs vmdb2 (.*) args = get_next_match() vmdb2 = os.path.join(srcdir, 'vmdb2') exit, out, err = cliapp.runcmd_unchecked([vmdb2] + args.split()) vars['exit'] = exit vars['stdout'] = out.decode() vars['stderr'] = err.decode() IMPLEMENTS THEN exit code is (\d+) wanted = int(get_next_match()) exit = vars['exit'] print('exit code', exit) print('stdout:', vars['stdout']) print('stderr:', vars['stderr']) assertEqual(exit, wanted) IMPLEMENTS THEN stdout contains "(.+)" followed by "(.+)" first = get_next_match() second = get_next_match() stdout = vars['stdout'] first_i = stdout.find(first) assertGreaterThan(first_i, 0) rest = stdout[first_i + len(first):] second_i = rest.find(second) assertGreaterThan(second_i, -1) IMPLEMENTS THEN stdout does NOT contain "(\S+)" what = get_next_match() stdout = vars['stdout'] i = stdout.find(what) assertEqual(i, -1) vmdb2/yarns/Makefile0000644000175000017500000000177513577177751013740 0ustar gwolfgwolf# Copyright 2017 Lars Wirzenius # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # =*= License: GPL-3+ =*= all: yarns.html yarns.pdf: $(yarns) Makefile pandoc --chapters --toc -o yarns.pdf *.yarn yarns.html: $(yarns) Makefile ugly.css pandoc -H ugly.css -f markdown+smart --toc \ --top-level-division=chapter --number-sections \ -V geometry:lettersize \ --standalone --self-contained -o yarns.html *.yarn vmdb2/yarns/lib.py0000644000175000017500000000022613577177751013406 0ustar gwolfgwolfimport os import sys import cliapp from yarnutils import * srcdir = os.environ['SRCDIR'] datadir = os.environ['DATADIR'] vars = Variables(datadir) vmdb2/yarns/ugly.css0000644000175000017500000000173113577177751013762 0ustar gwolfgwolf